PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
Authors: Tianchen Zhao
Ke Hong
Xinhao Yang
Xuefeng Xiao
Huixia Li
Feng Ling
Ruiqi Xie
SiQi Chen
Hongyu Zhu
Zhang Yichong
Yu Wang
Venue: NeurIPS 2025 poster ()
Keywords: Video Generation
Low-bit Quantization
Sparse Attention
Diffusion model
Acceleration
Primary area: applications
TL;DR: We unify the diverse attention pattern for visual generative models, and benefit the sparse and quantization

Abstract: In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolution image or multi-frame video generation. To address this, prior research has explored techniques such as sparsification and quantization.
However, these techniques face significant challenges under low density and reduced bitwidths. Through systematic analysis, we identify that the core difficulty stems from the dispersed and irregular characteristics of visual attention patterns. Therefore, instead of introducing specialized sparsification and quantization design to accommodate such patterns, we propose an alternative strategy: "reorganizing" the attention pattern to alleviate the challenges.
Inspired by the local aggregatin nature of visual feature extraction, we design a novel **P**attern-**A**ware token **R**e**O**rdering (**PARO**) technique, which unifies the diverse attention patterns into a hardware-friendly block-wise pattern. This unification substantially simplifies and enhances both sparsification and quantization.
We evaluate the performance-efficiency trade-offs of various design choices and finalize a methodology tailored for the unified pattern.
Our approach, **PAROAttention**, achieves video and image generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density (**20%-30%**) and bitwidth (**INT8/INT4**), achieving a **1.9 - 2.7x** end-to-end latency speedup.

PAROAttentton: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
Tianchen " 1.2, Ke Hong" Xinhao Yang"), Xuefeng Xiao", Huixia Li,, Feng Ling", Ruiqi Xie Siqi Chen', Hongyu Zhu', Zhang Yichong', Yu Wang
Tsinghua University, ByteDance Seed
Challenges:
Compression Techníques
CogVideoX (720P Bs)
Flux (1024x1024)
Existing Approach: Accommodate Pottenns With Specialized Sparse Desigo
Face challenge under low derse rale
Sparse Attention
Block wiso
Original Attention (FP16 Denas)
Our Approach: Reorganize Attention Potterns
Benefits
Token
Reorder
PAROAMt
PAROAti
(INT4 75% Denos)
Visual
Unity hardware-fnenddl block-wise
Blook-wise Quantization
5.7x Speedup
2. 2.5x Atin Speedup
Attontion Patterns
2.72x E2E Speedup 1.94x E2E Speedup
Figure I: PAROAttention unifies the diverse attention patterns through token reorder, which benefits both the sparsification and quantization. It achieves nearly identical generation result from full- precision baseline without metrics degradation, under lower density (20%-30 %) and bitwidth (INT8/INT4), achieving 1.9~2.7x end-to-end latency speedup.
Abstract
In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolution image or multi-frame video generation. To address this, prior research has explored techniques such as sparsification and quantization. However, these techniques face significant challenges under low density and re- duced bitwidths. Through systematic analysis, we identify that the core difficulty stems from the dispersed and irregular characteristics of visual attention patterns. Therefore, instead of introducing specialized sparsification and quantization design accommodate such patterns, propose alternative strategy: "reorganizing" the attention pattern to alleviate the challenges. Inspired by the local aggregatin nature of visual feature extraction, design novel Pattern-Aware token Re- Ordering (PARO) technique, which unifies the diverse attention patterns into hardware-friendly block-wise pattern. This unification substantially simplifies and enhances both sparsification and quantization. We evaluate the performance- efficiency trade-offs of various design choices and finalize methodology tailored for the unified pattern. Our approach, PA PAROAttennion. achieves video and image generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density (20%-30) % and bitwidth (INT8/INT4), achieving 1.9~2.7 end-to-end latency speedup.
'Equal contribution. work done when Tianchen Zhao intern Bytedance Corresponding author: Wang (yu-wang @isinghua.edu.cm)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CKV Linear
(MUFHg
H consecutive
Matmul QK*
W
Reorga nize
Softmax
aparee design
Simple sparae design
spaza
F-comsseuzve
Selected
Non-sparae rogion large
Fully sparsie local region,
Matmul PV
sparse emrrr
Outllers c9t0t, Large quant eiror
sparse eiror
Loso variation, Small cuant orro"
Tokara in secAARTIC order. [FFFWW
FUWW
O Linear
Hard reorganizes unity attention maps acroes patterns
Cuant @
(b)
[c] Unity Block-wise patterns by
Figure 2: The Motivation of PAROAttention. (a) The computational flow of transformer. (b) The challenge for sparsification and quantization caused by visual attention pattern, and how PAROAtten- tion addresses it. (c) The illustration of 3D feature, and ID token sequence with different orders.
1 Introduction
Diffusion transformers (DiT's [35]) have garnered significant research interest in visual generation tasks. However, their excessive resource cost poses challenges for broader applications. The adoption of "3D full attention" in models like CogVideo) VideoX [56] further increases token length. The quadratic complexity of attention mechanisms in token length results in substantial memory consumption and computational overhead when processing such long sequences. For instance, generating a 49-frame 6-second 720P video involves 17K token length". Attention computation contributes to the majority of the total latency, making it the primary bottleneck and requiring urgent optimization. As shown in Fig. 2(a), the two matrix multiplications QK" and PV (P is the attention map after softmax) has a computation cost quadratic to token length, making them the primary bottleneck in attention.
Previous research has explored sparse attention mechanisms [58, 55] and quantization [61, 59, 60] to accelerate attention. While these techniques achieve notable success in language models [3, 8, 20, 18). they cannot be directly applied to visual generative models due to distinet attention patterns. As shown in Fig. 2-(b), different attention heads exhibit diverse patterns. These patterns vary not only type (e.g., blockwise, multi-diagonal) but also in their structural characteristics- the number, width, and spacing of the diagonals. Recent sparse attention methods [57, 64, 45] explore designing specialized sparse masks for visual models, but still face significant challenges in maintaining quality at lower density rates (< 50%). Quantization techniques tailored to visual generation remain underexplored. Existing methods struggle to efficiently quantize PV computations to lower-bit integers (e.g., INTS/INT4), and often remain at FP16/FPS.
as
We conduct systematic analysis of the underlying reason for subpotimal performance of existing techniques under low density and bitwidths (as discussed in Sec. 3). Our findings reveal that the core challenge of sparsification and quantization stem from distinct characteristics of visual attention patterns. For sparsification, the dispersed and dynamically changing attention patterns lead to the absence of locally sparse regions, resulting in sparsification error. For quantization, the presence of multiple "diagonal" values act outliers within data group, thereby increasing quantization error. Different from existing methods that design specialized sparsification and quantization techniques to accommodate the diverse patterns, we propose alternative direction: To "reorganize" " the attention patterns to ease the difficulty the design of both sparsification and quantization methodologies.
To design proper technique to "reorganize" the attention pattern, we further analyze the underlying causes of the diverse visual attention patterns. Prior literature [33] reveals the local aggregation nature of visual attention, which suggest that attention tends to capture relationships between neighboring pixels. In vision transformers, the 3D physical space are flattened into ID token sequences, which disrupts the data adjacency. For example, as shown in Fig. 2-(c), the neighboring tokens along F-dimension in 3D space are not consecutive, and have the same interval H x W in ID token sequence of default order [F, H, W). The aggregation of these tokens with equal intervals forms the "multi-diagonal" pattern. Therefore, the multi-diagonal and block-wise patterns are intrinsically the same, representing local aggregation in different dimensions. By applying token reordering, we rearrange the tensor layout (e.g.. permute from [F, H, W] to [H, W, F()) for each head to keep
1 videos the figure are provided the supplementary.
We use this setting as example for most of the description below, the image token length N = 17550 F * W H = 13* 30 45, where F. H, W stands for the frame number, height, and width in the latent space.

---

elements of the local aggregation dimension contiguous. The "Pattern-Aware token ReOrder (PARO)" could transform diverse patterns into a unified, hardware-friendly block-wise pattern.
Furthermore, we analyze the trade-off's among design choices in terms of accuracy, and efficiency, and design specialized sparsification and quantization techniques tailored the unified block-wise pattern, constructing the PAROAttention method. We summarize our contributions as follows:
We analyze and identify the key challenges of attention sparsification and quantization as unique attention pattern characteristic, and propose to address it from novel direction of token reorder to reorganize and unify the attention pattern.
We compare the strength and weakness of design choices, and develop specialized method- ologies tailored for the unified pattern, along with CUDA kernels for practical acceleration.
PAROAttention, achieves generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating notably lower density (20 --00% %) and bitwidth (INT8/INT4), achieving a 1.9~2.77 x end-to-end latency speedup.
2
Related Works
2.1 Visual Generative Models
Diffusion transformers [35], which leverage transformer architectures [40, 9], have achieved out- standing performance and widely adopted by recent image generation models such as PixArt [6] and Flux [19]. For video generation, earlier approaches, such as OpenSORA [15], apply "spatial- temporal" attention, which performs attention separately along the spatial and temporal dimensions. Other recent models like VogVideoX [56]. Wan [41] adopt "3D full attention" instead, processing all spatial tokens across all frames simultaneously. The increased model size and token length pose sigmificant challenges for efficient deployment.
2.2 Sparse Attention for Generative Models
Existing sparse attention research [3, 8, 5, 48, 29, 65] primarily focuses designing sparse masks aligned with specific attention patterns, such as sliding window patterns [67, 49, 50] and attention sink patterns [51, 11] commonly observed in language models. However, visual generation involves unique attention patterns that require new forms of specialized sparse mask design. Recent studies have explored various mask designs tailored for visual generation, including window-based approaches (e.g., DiTFastAttn [57], STA [64]). spatial temporal patterns (c.g., VideoGen [46, 54]). and hybrid mask combinations (e.g., MInference [18]. In contrast, Sparge [62] does not rely on predefined sparse masks but instead generates them online based on QK embeddings. Despite these advancements, the dispersed and diverse distribution of attention values in visual tasks prevents these mask patterns from maintaining quality under low density. In this work, we address this challenge by reorganizing attention patterns through token reordering.
2.3 Quantization for Generative Models
Quantization [37, 66, 22, 31, 23) has proven to be highly effective across a wide range of applications. For visual generation, prior research [69, 36, 24, 70, 21, 16, 26] has identified unique challenges associated with quantizing DiTs, they primarily focused on quantizing the linear layers in transformers. However, the increasing token length has shifted the bottleneck to the attention mechanism. Recent advancements [61, 59]. have explored quantizing QKT to INT4/8 while employing FP8 (8 bit floating boaating-point for PV. In this work, we address the unique challenges of attention quantization in visual generation, analyzing the underlying reasons behind the difficulty of quantizing the P matrix. We further extend our investigation to explore INT4/8 quantization for the PV computation.
3 Preliminary Analysis
Key Challenge for Sparse Attention: The goals of sparse attention design are two-folded: (1) preserving algorithmic performance by avoiding the removal of important attention values, and (2) enabling practical hardware acceleration. Since arbitrary sparsity patterns could not achieve practical
3

---

acceleration [7], proper structured sparsity is necessary. However, as illustrated in Fig. 2, the dispersed and diverse nature of visual attention pattemss presents significant challenges for designing structured sparsity. Firstly, diverse attention patterns vary in type (block-wise multi-diagonal, and diagonal-in- block); as well as in structural characteristics (the number, width, and spacing of diagonals). These patterns also change dynamically across different timesteps and prompts. Designing proper structured sparsity pattern that accommodates all these variations and generalizes across different scenarios is extremely difficult. Secondly, the dispersed attention distribution makes it challenging to form fully sparse regions, thereby inevitably introducing errors in structured sparsity. Given these challenges, improving sparse mask design to accommodate diverse patterns may have limited effectiveness. Instead, we propose an alternative direction:
reorganizing the attention pattern.
Key Challenge for Attention Quantization: The goal of quantization design is to minimize quantization error, and we begin by analyzing its sources. As shown in prior work [69], a major source of quantization error arises from large variations within a data group. In such cases, the computed scaling factor becomes excessively large, compressing the majority of values toward zero and leading to significant quantization error. Upon analysis, we find that the distributions of Q, K, and do not exhibit substantial variation. The primary challenge lies in properly quantizing the attention matrix P. As illustrated in Fig. 2, in the "diagonal-like" patterns, the larger values along the diagonals act as "outliers" within each local region (i.e., quantization group), leading to substantial rounding errors. Addressing this issue is crucial for reducing quantization error. This also points to the need for reorganizing the attention distribution to reduce outliers.
Methodology
4.1 Pattern-aware Token Reordering (PARO)
As concluded in Sec. 3, it is crucial to introduce a technique that reorganizes the attention distribution to mitigate challenging characteristics and promote structure more favorable to sparsification and quantization. To explore this, we investigate the root causes of the diverse attention patterns. Inspired by the local aggregation nature discussed in Sec. 1, we observe that diverse patterns actually represents local aggregation along different dimensions. They could potentially be transformed into a localized block-wise pattern by gathering the locally aggregated tokens together through token reordering.
However, determining the optimal reorder strategy for each attention head a non-trivial challenge. First, the large number of tokens (Nuken) leads to vast search space of possible reorders, making optimization difficult. Second, the reorder strategy must be carefully designed to minimize hardware overhead. Third, it should simultaneously satisfy the distinct requirements of sparsification and quantization: sparsification benefits from fully sparse local regions, whereas quantization requires balanced distributions within local regions. To address these challenges, we focus on specific subset of reorder- "permutations". inspired by the observation that certain attention heads tend to aggregate information locally along a particular dimension. In the case of 3D video generation, where tokens are structured along three dimensions [F, H, W). we limit the reordering space to the six possible permutations (P3 6). We verify that the optimal permutation is sufficient to produce unified, block-wise patterns through empirical analysis as seen in Fig. 3. We further elaborate on how we address the above mentioned challenges as follows:
We first verify that the optimal permutation remains consistent across different timesteps and prompts. Based on this observation, we determine the permutation order offline, eliminating the need for runtime overhead associated with generating the reordering strategy. Consequently, the only remaining overhead is the online application of the permutation itself, which primarily involves data movement. This cost can be significantly reduced by fusing the permutation operation with preceding kernels. After fusion, explicit data movement from global memory on the GPU is avoided-one only the output write-back addresses need to be adjusted. The resulting overhead is less than 1% of the preceding kernel (e.g., LayerNorm), and thus negligible compared to the cost of attention computation.
Minimize Hardware Overhead:
As discussed above, sparsification and quantization prefer different distributions. We design separate metrics and combine them to to determine the permutation order for each head. Given the post-softmax attention map P E RNN N = k x b, where i is the block size. P is tiled into k x sub-matrices P.₁ E b For sparsification, we choose a relatively small value (c.g., Ie-3), and classify the block as sparse if the vast majority of values (over threshold a, e.g..
Metric for Permutation Order:

---

Quant
Reorder
Q
Quant
Sparse Mask
Matmul QK"
Softmax
Sparse Mask
Quant
Matmul PV
Inv- Reorder
K
Quant
Permute Order
Online permute with negligibls Permuee onders are
Sperse Plan
Sparse attention for QX' and Offine sparse no runtime mask generation
Inooharence
mantle 22
Guantiation
483
93
22
penutc_order [s,w,)] reshapec(Tr (k,M))
10000-----------
Mut
K.reshape
for dlar talf
Permutation with Neglibble Overhead
Fused CUDA Karnl for Online
Spara readt
Timastep-koore
Pat-Row
Blockwwian
64]
Block-Abie Recrdar
[6464]
Pattern-aware Reordering
Block-wise Sparse Attention
Quantized (Sec
Offiine determined permute order
Metric for
For 18k, r.bik In cnmerstte(_bikk)
Objoctive: arguiin
sparne_ece (P aba
lntt.antt)...nttnnn Ont (a to r_bik,,bbs .eax buall_spaaree [1ec] [sparse Lact bu signa)
Migerr
Poit Marl) DTE NONT WITH
Choose permutation from (N) possible cosea 9) E (8 B.) for each head
Order
KLaparsa 18.81k sparse.sva s.a / K.blocks
En Maver
En, Masas
PLquaRt enceeeeeeeee)
Figure 3:
The overall framework of PAROAttention. The pattern- aware token reordering (PARO) applied to unify the attention pattern into hardware-frienddy block pattern. Sparse attention and quantization techniques are designed tailored for this pattern.
90%)) within the block are smaller than 6. The percentage of such sparse blocks adopted as the sparse metric M sparses it could be calculated as follows, where I stands for the indicator function.
1 << = -. En ((P::(mm (m) < e). M Maparse- EXT
Σ
(1)
bxb
For quantization, we follow the previous methods [69] and adopt "incoherence" (maximum value divided by the mean absolute value) as an indicator of quantization difficulty within data group 129 The quantization metric Mquent is defined follows:
(x) = max( Σ Mquant = RXR E (P.)))
(2)
Finally, the M Maparse and Mguant are normalized across all possible permutations Q (01. Oa) and combined as the final metric M. The weighting coefficient controls the relative importance of two aspects, and the permutation with the lowest M: is chosen.
M = C * Ea, Mararse
M
+ (1 - a) * Σ 3g
M
quant
(3)
M
quant
4.2 Block-wise Sparse Attention
After applying permutation, we obtain attention maps with unified and regular block-wise patterns. We further elaborate comparing the strengths and limitations of different design choices to conclude the final PAROAttention sparsification design.
Static vs. Dynamic Sparse Attention: There are two major schemes for sparse attention: the dynamic approach, which predicts sparse masks online, and the static approach, which calibrates sparse masks offline. Since the PARO attention reorganization is compatible with both schemes, we summarize their respective strengths and limitations below and justify our final selection:
(1) In terms of preserving algorithmic performance, the dynamic approach bases on QK embeddings predict the sparse mask. However, as shown in Fig. the pre-softmax attention map (QK") con- tains relatively uniform values and lacks distinguishable sparse patterns, making accurate prediction of the attention pattern difficult. Furthermore, the QK embeddings often need be downsampled to reduce computational overhead, which further compromises prediction accuracy. The static approach, on the other hand, benefits from access the more informative post-softmax attention patterns. Nevertheless, it still faces challenges in designing sparse masks that accommodate the highly diverse, and dynamically changing attention distributions. Fortunately, with our token reordering strategy that transforms attention into unified block- wise structure, these challenges significantly alleviated.
5

---

(2) In terms of hardware efficiency, the dynamic approach introduces runtime overhead for online sparse mask prediction. Reducing this cost often comes at the expense of prediction accuracy. The static approach, while avoiding this cost, incurs memory overhead for storing sparse masks and offline calibration overhead. To address this, we develop techniques to minimize these costs.
Minimize Hardware Overhead:
We ensure that the sparsification design of PAROAttention intro- duces minimal overhead by incorporating the following techniques:
(I) Timestep-Aware Sparse Mask Sharing: As discussed above, offline determined static sparse mask face challenges when generalizing to dynamically changing sparse patterns. To address this, we systematically analyze the similarity of attention maps across multiple dimensions. Since sparsification is applied only to image tokens (account for 99% of all tokens), observe extremely high similarity across different prompts (cosine similarity 2 0.99). However, lower similarity is witnessed across the timestep dimension and adopt timestep-wise sparse masks. Despite improving accuracy, timestep-wise sparse masks increase memory usage. We observe that attention patterns change most during the early timesteps. Thus, we apply distinct timestep-wise masks only for the first half of timesteps and reuse a common mask for the remainer. We further reduce runtime memory cost by prefetching sparse masks for each head during inference, eliminating the need to store all masks simultaneously. When stored as binary bitmasks, each head's sparse mask requires only 9.2 KB, making the overall storage and data movement overhead negligible.
(2) Block-Aligned Sparse Granularity for Efficient CUDA Execution: FlashAttention processes attention in block-wise manner, we align our sparsification granularity with the Attentit block size. This allows for an extremely simple CUDA implementation, where entire blocks can be skipped without additional branching logic overhead.
(3) Efficient Offline Mask Generation: Although static methods allow sophisticated sparse metric design, we find that simple block sum thresholding is sufficient. It also enables explicit control over density. Due to strong generalization across prompts, only 1-2 prompts are needed to determine the permutation order and generate the sparse masks, which only involves minute-level cost.
4.3 Block-wise Quantized Attention
As discussed in Sec. 3, the major source of quantization error comes from the data variation within quantization group. Prior literature [4, 69] introduces a metric a metric "incoherence" y as shown in eq. (2), to measure the relative "sharpness" of the data distribution within group. Data groups with higher incoherence are more challenging to quantize. We conclude and justify our design choice for PAROAttention quantization design to reduce incoherence and ensure hardware efficiency as follows: (1) Block-Aligned Quantization Granularity (Grouping): We emphasize the importance of aligning the quantization grouping with the FlashAttention block size, considering both algorithmic performance and hardware efficiency. As illustrated in Fig. 3, adopting naive per-row quantization scheme (analogous to per-token grouping for Q and K') not only incompatible with the block- wise processing paradigm of FlashAttention but also introduces substantial incoherence due to the inherently "diagonal" structure of visual attention patterns. This highlights the necessity of block-wise quantization grouping. However, even within local blocks, the diagonal distribution persists, leading to high incoherence (e.g.,
93), which necessitates further optimization to reduce quantization error. (2) Token Reorder for Incoherence Reduction: Prior work has proposed distribution--allanee
quantization techniques for linear layers, such as scaling [27] and rotation [4]. However, these approaches are not applicable to the PV computation in FlashAttention due to the iterative update mechanism of P, which is not explicitly materialized in order to save memory. Instead, explore novel direction: tuning the attention distribution via token reordering, which groups similar attention values together. As shown in Fig. 3, this reordering significantly reduces incoherence, thereby effectively mitigating quantization error.
5 Experiments
5.1 Experimental Setup
Video and Image Generation: For video generation, we apply PAROAttn to the Cog VogV [56] and Wan [41] (see Appendix 1ec. model for 720P 6/10-second video with 30 sampling steps. To verify generalization, we collect calibration data with the first 2 prompts of the Video example

---

Table I: Performance of PAROAttention CogVideoX text-to-video generation on VBench prompts. Baselines are evaluated using their official codebases. For fair comparison, we configure VideoGen without skipping sparsification during the first 30% of timesteps. The "SpargeAttn (0.5 + PARO)" denotes the SpargeAttention method augmented with token reordering (PARO).
Efficiency
Quality
Type
Method
Video Quality Metrics
FP Diff Metrics
Donse Rate Biwiddth
CLIPSIMI
VQAT
AFSore]]
FVD-FPI6)
PSNR1
SSIMI
CoSSmi
FP16 Full Alm.
100.0%
0,203
92.53
0.000
0,000
1.000
1.000
AiTFastAAtn (0.5)
50.0%
0.740
0.904
0.603
MInference (0.5)
50.0%:
0.197
86.02
2,250
0,368
16.54
0.696
0.945
SpargeAItt (0.5)
1.154
0.347
0.683
SpargeAtin (0.5 + DARO)
50.0%
0.198
89.26
0.671
0,259
17.32
0.693
0.948
Sparse
SparseVideoGen (0.5)
50.0%
0.198
90.14
0.568
0.186
18:50
0.755
0.960
PAROAum (0.5)
50.0%
0.203
92.56
0.103
0.068
29.14
0.936
0.997
SpargeAttt (0.3)
30.0%
0.197
86.74
1.231
0.375
15.22
0.642
0.912
SpargeAtin (0.3 PARO)
30.0%
0.197
89,96
1.142
0,339
16.89
0.683
0.946
SparseViccoGen (0.3)
30.0%
0.197
89.54
0.589
0.241
17.73
0.725
0.954
PAROAtt (0.3)
30.0%
0.204
92.66
0.101
0.153
22.89
0.984
PAROAttn (0.2)
20.0%
0.203
92.42
0.151
0.151
19.39
0.744
0.962
PAROAR (0.125)
12.5%
0.201
15.93
RTN (IN88)
OK (NNT8, PV (NNS8)
0.190
92.09
0.571
0,480
18.88
0.750
0.956
KTN (NT44)
OK (NNT4) PV (INT4)
0.184
69.25
3,360
1.446
11.99
0.500
0.905
SageAttn
QK (INTS), PV (FP16)
0,203
92.24
0.131
0.047
29.58
0.927
0.997
Quant
SageArtnV2
QK (INT4). PV (FP8)
0,200
88.79
2.460
1,750
0.824
0.979
PAROARI (INTS)
OK (INTS), PV (INTS)
0.203
92.57
0.096
0,026
29.01
0,935
0.996
PAROAtta (INT4)
QK (INTA), PV (INT4)
0.200
89.24
0.876
1.382
24.16
0.822
0.985
Sparse
PAROAttn (0. (+13+))T))
QK, PV
0.201
91.68
0.884
0.533
21.49
0.779
0.976
+ QNannt
PAROAIt (NT5+
S0% QK, PV (INT4)
0.200
90.42
0.967
1.431
24.34
0.827
0.986
FP16
DiTFastAttn (00%))
Minference (50%)
SparseVideoGen (50%)
SpargeAttn (1015)
SageAttn JOK
RTN (OX INT4 (N1)
SageAttnV2
(100%)
PV FP18)
IKK4 + PV FFS1
PAROAttn (847))
PAROAttn (X2%)
PAROAttn CL0%
PAROAttn (12.5)
PAROAttn (l0% INLS)
PAROAttn JCK INTS - INTE
PAROAttn OK INT4 PVINTA
PAROAttn (SONS (NT4)
Figure 4: Qualitative Results of CogVideoX generated videos for PAROAttention and baselines.
dataset [38] and evaluate on a subset of prompts collected from VBench [17], covering all subtasks. For image generation, we apply PAROAttn the Flux.1.Dev [19] model for 1024 resolution, with 30 sampling steps. The calibration prompts are the same as video generation and the first 1024 prompts from the COCO [28] dataset are used for evaluation. The block size for sparsification and quantization are chosen as 64 to align with the Attentnten Unlike prior work [45, 64), which avoids compressing the initial 25% of timesteps, our techniques are applicd to all timesteps.
We employ two types of metrics: (1) Quality Metrics: They measure the abso- lute quality of videos or images. For videos, we adopt CLIPSIM [43], VQA [44], and FlowScore [32] to measure text-video alignment, quality, and temporal consistency respectively. For images, we adopt CLIPScore [13] and ImageReward [53] to measure text-image alignment, and human preference. (2) Relative Difference Metrics: They quantify the difference between FP16 generation. For both video and image generation, PSNR and cosine similarity are used to measure low-level pixel-space differences. SSIM [42] evaluates structural similarity, while FVD-FP16 [39], and FID-FP16 [14] assess feature-space differences. practice, we find that relative difference metrics are more sensitive and better reflect the quality of compression techniques (discussed in Appendix Sec.2).
Evaluation Metrics:
7

---

Table 2: Performance of Flux text-to-image generation on COCO prompt set.
Efliciency
Quality
Type
Method
Deuse Rate Bitwidth
Image Quality Metrics
FP Dif, Metrics
CLIPScoNT
ImageReward
FID-FP16,
PSNRT
SSIM
CosSimt
FP16 Full Atn.
100.0%
0.258
1.02
0.00
00
1,000
1.000
DiTFastAtt (0.75)
75.0%
0.258
0.96
28.31
16.73
0.687
0.956
MInference (0.75)
75.0
0.260
34.88
14.68
0.602
0.933
Sparge/tt (0.75)
75.0%
0,260
0.97
29.04
16.76
0.680
0.951
Sparse
PAROAIt (0.75)
0.259
19.47
20.95
0.812
DiTFastAttn (0.5)
50.0%
0,255
0.81
53.95
12.49
0.537
0.890
Minference
50.0%
0.255
0.89
42.58
13.42
0.583
0.908
SpargeAitn (0.5)
50.0%:
0,255
0.91
55.47
14.62
0.602
0.924
PAROAit (0.5)
50.0%
0.259
1.04
30.39
16.20
0.683
0,944
Sage-Att SageAttnV2
QK (INTS), PV (FP16)
0,258
1:00
14.77
23.47
0.863
0.986
Quant
QK (INT4), PV (PP8)
0.257
1.00
20.11
20.95
0.814
0.979
PAROAUO (INTS)
QK (NNTS), PV (NTS)
0.258
1.90
15.34
23.04
0.856
0.986
PAROAmI (INT4)
QK (INT4), PV (INT4)
0,258
1.00
19.65
20.16
0,793
0.975
Sparse
PAROAIn (0.5+INT8)
50% OK, PV (NTS)
0.259
1.04
29.56
16.28
0.680
0.947
+Quard
PAROAII (0.75HINT()
75% QK. PV (NTT))
0,259
1.01
22,45
19.26
0.770
0.971
aitFattAttn
Minffrence
SpargsAtttn
PAROAttn
SageAttn
SageAtinV2
PAROAttn
PAROAtin
PARGABI
(100%)
RO
MOTEL
HOTEL
WOTEL
MOTEL
Figure 5: Qualitative Results of Flux generated images for PAROAttention and baseline methods.
For sparsification, we select baselines of different schemes, including: DiT- FastAttn [57] (dynamic window mask), SpargeAAtn [62] (dynamic block-wise mask), and MInfer- ence [18], Sparse VideoGen [45] (multiple static mask, dynamic selection). PAROAttn adopts static mask, and achieve superior performance under lower dense rate. For quantization, compare with naive RTN [34], SageAttn [61] (INT8 QK, FP16 PV), and SageAttm V2 (INT4 QKT, FP8 PV). PAROAttn quantizes PV to lower bitwidth (QK, PV INT8/INT4) with comparable performance.
Baseline Methods:
CUDA Kernel Implementation: We implement PAROAttn based on the SageAttn [59] kernel, incorporating customized designs for sparsity and quantization. Sparsification comparison are conducted on an NVIDIA A100 with CUDA 11.8, due to support limitations of baseline methods. The quantization comparison is conducted on NVIDIA RTX 4090 for FP8 and INT4 support.
5.2 Main Results
We present the evaluation metrics in Tab. 1 and ualitative comparisons in Fig. 4, using a challenging prompt that features a complex scene with multiple objects (e.g.. artworks and people). We conclude our findings as follows: (1) Baseline sparsification methods exhibit notable performance degradation for multiple metrics, even at relatively high dense rate of 50%, resulting in visible content distortion or blurred outputs. (2) In contrast, the PAROAtt sparsification method can generate images nearly identical the FP16 full attention baseline, even at 20% dense rate, and achieves metric scores that surpass those of the 50% baseline. (3) The PARO token reordering compatible with dynamic sparsity approaches. Simply combining PARO with SpargeAtttn at 30% density (PSNR: 16.89) achieves performance comparable to SpargeAAttn at 50% (PSNR: 16.8), yielding speedup improvement from 1.67x 2.22x, as shown in Fig. (4) The PAROAtt quantization method maintains comparable performance while further quantizing PV to lower-bit integers (e.g., PAROAttn (INT8) vs. SageAttn, and PAROAttn (INT4) vs. Vt2n (5) The PAROAttn sparsification and quantization techniques could be combined together for improved speedup. With aligned metric scores, the most aggresive plan PAROAtn (0.5+INT4) achieves nearly 10x speedup compared with baseline methods with 1.5-2x speedup.
Text-to-video generation:
8

---

Overhead
12
35
12
30
Nomaltzdd Lateney
26
20
0.8
24116
0.8
8.50x
8.50
17.55
0.G
PSAR
Nomalized
0.6
0.4
0.30x
6.3kc
9,40
772
0.35
0.55
0.77
0.28
9.1k
(4)
Ama.
A2.
INT3
ABTV2
12.3
Fullats
(6.)
10.51
(0:5)
10.11
(6:30
133HPARDI
(5.3)
(0.2]
Figure 6: Normalized latency speedup (bar plot) and PSNR (line plot) trade-off under different (a) quantization and (b) sparse configurations.
Block Uniform
Table Ablation studies. "(- Token- "eokeerr denotes PAROAtt without the token-reorder technique.
Sperss
(F.HWN]
[WHF]
[HFFA]
PP Diff Metrics
Selected
Type
Method
Block Uniform
PSNR1
SSIM1
LPIPS)
CosSim"
Block Spaam
PAROAtIn (0.5)
29.14
0.936
0.045
0.997
Sparse
26.25
0.907
0.069
0.992
[FAKH
[WFM]
[[W.F]]
Timestep Share)
PAROAtt (PV INTS)
30.17
0.940
0.039
0.995
Figure 7: Attention patterns under dif-
Quant
Roordar)
0.995
ferent permute orders.
Block Group)
27.50
0.906
0.063
0.994
We present the evaluation metrics in Tab. 2 and qualitative results in Fig. 5. The conclusions observed previous sections hold consistently, Due to the shorter token lengths, sparsification becomes more challenging, and most baseline methods introduce noticeable artifacts and content distortion at a 50% density rate. In contrast, PAROAttn effectively preserves both visual quality and content, even at low density rates and when combined with quantization.
Text-to-image generation:
Hardware Resource Savings: We compare the latency speedup and performance-efficiene off with baseline method's CUDA kernel implementation in Fig. 6. We conclude the key findings as follows: (1) The PAROAttn kernel achieves both superior speedup and algorithmic performance with aligned sparsity. For instance, at 50% density rate, PAROAttn achieves a 1.73x speedup, outperforming SpargeAttn (1.67x) and Sparse and reduced overhead. (2) PAROAttn's 's sparsification achieves speedups approaching the theoretical upper bound for computation reduction (e.g., I. 1.73x at 50% density, 2.71x at 30%), demonstrating its hardware- farinnff nature. (3) PAROAttn introduces minimal runtime overhead (<1%), compared with SpargeAAttn (6-9%) and VideoGen (10-15%). (5) PAROAttn supports quantization of PV to bit formats (c.g.. INT8/INT4), achieving similar performance to SageAttn while notably improving speedup (c.g., from 1.72x to 1.83x, and from 2.56x to 2.97x).
trade-
(1.42x), attributed to its simplified design
Analysis
We conduct extensive analyses to demonstrate the effectiveness of PAROAItn, we highlight key results here and provide additional details in the Appendix.
Ablation Studies: We present ablation studies of PAROAttn's 's techniques in Tab. 3. Removing token reorder leads to significant metric degradation for both sparsity and quantization. Similarly, eliminating timestep sharing and storing sparse masks for all timesteps does not improve performance, demonstrating that later timesteps can effectively share sparse masks. Additionally. replacing the block-wise quantization group with row-wise approach for PV quantization results in notable degradation, highlighting the importance of the block-wise quantization group.
Overhead Analysis: The additional cost of PAROAttn is two-fold: The runtime overhead is mini- mized, as shown Fig. 6. The offline mask generation incurs only minute-level cost, which faster than the hyperparameter tuning required for SpargeAttn the mask generation in Sparse VideoGen.
Effectiveness of PARO permute metrie: We visualize the attention map of the 5th head in the 1st transformer block under six different token permutation orders in Fig. 7. The permutation successfully transforms the "multi-diagonal" patterns into block-wise patterns. Notably, for the permutation [H, F, W), although the values are uniformly distributed within the block, insufficient sparsity is observed. In contrast, the selected permutation [H. W, F] exhibits both sparse and uniform blocks, demonstrating the effectiveness of the metries for permutation order.

---

Acknowledgement
This research was supported by National Natural Science Foundation of China 2N..62203257, 62325405, (00000556221111110 (62406119), 1220611,,,,,,,46111599 Tsinghua University Initiative Scientific Research Program, Tsinghua- Efort Joint Research Center for EAI Computation and Perception, Beijing National Re- search Center for Information Science, Technology (BNRist), Beijing Innovation Center for Future Chips, and State Key laboratory of Space Network and Communications.
References
[1] Tensor core. https: http:/eressurcs...iii..aaaa cww/cm///////o wet--ussteetorcoooe 26
[2] Xilinx dsp. https::/ //wcs///ccc..o// amd com/:/2021 DSP48E,. 26
guide/
[3] Moonshot AL Moba: Mixture of block attention. Technical report, MoonshotAl, 2025. 3,3
[4] Salch Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference rotated llms. arXiv preprint axXiv:240.000456 2024. 6
[5] Aiyue Chen, Bin Dong, Jingru Li, Jing Lin, Yiwu Yao, and Gongyi Wang. Rainfusion: Adaptive vidco generation accelcration via multi-dimensional visual redundancy. arXiv preprint 205::2500.., 2025.
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of di diffusion transformer for photorealistic text-to-imaae synthesis, 2023.
[7] Shail Dave, Riyadh Baghdadi, Tony Nowatzki, Sasikanth Avancha, Aviral Shrivastava, and Baoxin Li. Hardware acceleration of sparse and irregular tensor computations of ml models: survey and insights. Proceedings of the IEEE, 109(10):170-11552 2021.
[8] DeepSeek. Nested sparse attention. arXiv preprint urXiv:2502. 11089, 2025. 2.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3
[10] Tianyu Fu, Yi Ge, Yichen You, Enshu Liu, Zhihang Yuan, Guohao Dai, Shengen Yan. Huazhong Yang. and Yu Wang. R2r: Efficiently navigating divergent reasoning paths with small-large model token routing. arXiv preprint arXiv:2505.22 26600, 2025. 28
[11] Tianyu Fu, Haofeng Huang. Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan. Guohao Dai, Huazhong Yang, and Yu Wang. Moa: Mixture of sparse attention for automatic large language model compression, 2024. 3
[12] Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction large visual language models. arXiv preprint arXiv:2501.01990 2024. 28
[13] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Bras, and Choi Yejin. Clipscore: A reference-free evaluation metric for image captioning. pages 7514-7528, 2021.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, 17, page 6629-6640,, Red Hook. NY, USA, 2017. Curran Associates Inc. 7
[15] HPC-AI. Open-Sora. attpa://gitae httpaa///github..c//aapaaaee www.//pcaatecchhpenneaaaooaaaaee
2024. 3
[16] Yushi Huang, Ruihao Gong, Jing Liu, Yifu Ding, Chengtao Lv, Haotong Qin, and Jun Zhang. Qvgen: Pushing the limit of quantized vidco generative models. arXiv preprint arXiv:2005. / /497, 2025. 3
[17] Ziqi Huang. Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982. 2023. 7
10

---

[18] Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han. Amir H. Abdi, Dongsheng Li, Chin- Yew Lin, Yuqing Yang. and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context Ilms via dynamic sparse attention, 2024. 2, 3, 8,
[19] Black Forest Labs. Flux.1: A high-quality text-to-image model. 2024. Accessed [current date]. 3,
https://github.
[20] Xunhao Lai, Jianqiao Lu, Yao Luo. Yiyuan Ma, and Xun Zhou. Flexprefill: context-aware sparse attention mechanism for efficient long -equeeceence inference, 2025, 2
[21] Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdqunat: Absorbing outliers by low-rank components for 4-hit diffusion models, arXiv preprint arXiv:241 I. 1.0007,, 2024. 3
[22] Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, et al. Mbq: Modality balanced quantization for large vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4167 4177, 2025. 3
[23] Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang, Evaluating quantized large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 28480-28524, 2024. 3
[24] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535-17545, 2023. 3
[25] Haokun Lin, Teng Wang. Yixiao Ge. Yuying Ge. Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to clip for multimodal comprehension and generation. arXiv preprint (tt::5555005422 2025. 28
[26] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. Advances in Neural Information Processing Systems, 37:87766 87800, 2024. 3
[27] Ji Lin, Jiaming Tang, Haotian Tang. Shang Yang. Xingyu Dang, and Song Han. Awq: Activation-awe aware weight quantization for IIm compression and acceleration. arXiv preprint arXiv: 00978, 2023. 6
[28] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. 7
[29] Akide Liu, Zeyu Zhang, Zhexin Li, Xuchai Bai, Yizeng Han, Jiasheng Tang, Yuanjie Xing. Jichao Wu, Mingyang Yang, Weihua Chen, al. Fpsattention: Training-aware fp8 and sparsity co-design for fast video diffusion. arXiv preprint 250i:::2000044448 2025.
[30] Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to forecasting: Accelerating diffusion models with taylorseers. arXiv preprini arXiv:2503.06923 2025. 28
[31] Tengxuan Liu, Shiyao Li, Jiayi Yang. Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. Pm-kvq: Progressive mixed-precision kv cache quantization for long-cot lms.. arXiv preprint arXiv:2505. 18610, 2025. 3
[32] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalerafter: Benchmarking and evaluating large video generation models. arXiv preprint arXiv:2310.1
2023.7 7
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021. 202C. 2
[34] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint 1rXiv:2006002995 2021. 8
[35] William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. 2,
[36] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1972 1981, 2023.
11

---

[37] Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1.x-bit kv cache quantization for video large language models. arXiv preprint urXiv:2503.16257, 2025.
[38] THUDM. Cogvideox-Sb. https: httpss//haagaaigaaaae..c////////
2025. 7
[39] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly, Fvd: A new metric for video generation. 2019. 7
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. Advances neural information processing systems, 30, 2017.
[41] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Xic, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiay Wang. Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang. Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang. Tianxing Wang, Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang. Wei Wang. Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu, Xianzhong Shi. Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang. Yitong Huang. Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong. Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Ham, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, arXiv preprint ar 2rxiv:2003.00::
2025. 3, 6, 21
[42] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE transactions on image processing, 13(4)6600-612, 2004. 7
[43] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2 100.2220444466 14806, 2021. 7
[44] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang. Weaxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment user generated contents from aesthetic and technical perspectives. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20144-201555 2023.7 7
[45] Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang. Dacheng Li. Jianfei Chen, lon Stoica, Kurt Keutzer, and Song Han. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025. 2, 7,8
[46] Hancheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xinyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, et al. Sparse videogen: Accelcrating video diffusion transformers with spatial-tempore sparsity. arXiv preprint arXiv:2502.0177: 2025. 3, 24
[47] Yifei Xia, Fangeheng Fu, Wentao Zhang, Jiawei Jiang, and Bin Cui. Efficient multi-task IIm quantization and serving for multiple lora adapters. Advances in Neural Information Processing Syxtems, 37:63686- 637 14, 2024. 28
[48] Yifei Xia, Suhan Ling, Fangcheng Fu, Yujie Wang. Huixia Li, Xuefeng Xiao, and Bin Cui. Training- free and adaptive sparse attention for efficient long video generation. 2025. 3
[49] Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training free cong-context extrapolation for llms with an efficient context memory, 2024. 3
[50] Guang Xuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:2410. 10819, 2024. 3
[51] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. 3
[52] Xingrun Xing, Zheng Liu, Shitao Xiao, Boyan Gao, Yiming Liang, Wanpeng Zhang, Haokun Lin, Guoqi Li, and Jiajun Zhang. Efficientllm: Scalable pruning-aware pretraining for schittctuur-agnosste edge language models. arXiv preprint arXiv:2502.0666 2025. 28
[53] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
12

---

[54] Shuo Yang, Haocheng Xi, Yilong Zhao, Muyang Li, Jintao Zhang, Han Cai, Yujun Lin, Xiuyu Li. Chenfeng Xu, Kelly Peng, et al. Sparse videogen2: Accelerate video generation with sparse attention via semantic-aware permutation. arXiv preprint arXiv:2505. 2025.
[55] Zhuoyi Yang et al. Seerattention: Learning intrinsic sparse attention in your Ilms. arXiv preprint arXiv:2410.13276. 2024.
[56] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408. 06072, 2024. 2, 3, 6
[57] Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning. Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. DiTFastattn; Attention compression for diffusion transformer models. In The Tiihty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2, 3, 8, 24
[58] Manzil Zaheer, Guru Guruganesh, Avinava Dubey, et Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007 14062, 2020. 2
[59] Jintao Zhang. Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization, 2024. 2, 3,
[60] Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang. Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling Ip4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505. 2025. 2
[61] Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfci Chen. Sageattention: Accurate 8-bit attention for plug-and-pha -lay inference acceleration. In International Conference on Learning Representations (ICLR), 2025. 3.8
[62] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. arXiv preprint 2rXiv:2502.18137, 2025. 3,8 8
[63] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi. Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference, 2025. 21, 24,
[64] Peiyuan Zhang, Yongqi Chen, Runlong Su, Hangliang Ding, lon Stoica, Zhenghong Liu, and Hao Zhang. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04500 045007 2025. 2, 3.7
[65] Peiyuan Zhang, Haofeng Huang, Yonggi Chen, Will Lin, Zhengzhong Liu, Ion Stoica, Eric P Xing, and Hao Zhang. Faster video diffusson with trainable sparse attention. arXiv preprint arXiv:2505.13335 2025.
[66] Pengle Zhang, Jia Wei, Jintao Zhang, Jun Zhu and Jianfei Chen. Accurate int8 training through dynamic Block-level fallback. arXiv preprint arXiv:2503.0880 2025. 3
[67] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. Hęo: Heavy-hitter oracle for efficient generative inference of large language models, 2023. 3
[68] Lin Zhao, Tianchen Zhao, Zinan Lin, Xuefei Ning. Guohao Dai. Huazhong Yang, and Yu Wang. Flasheval: Towards fast and accurate evaluation of text-to-image diffusion generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16122-16131, 2024. 28
[69] Tianchen Zhao, Tongcheng Fang, Haofeng Huang. Enshu Liu, Rui Wan, Widyadewi Soedarmadji, Shiyao Li. Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang. Xuefei Ning, and Yu Wang. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation, 2025. 3, 5,
[70] Tianchen Zhao, Xuefei Ning, Tongeheng Fang, Enshu Liu, Guyue Huang. Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficient few-step text- text-to-image diffusson models with metric-decoupled mixed precision quantization. In European Conference on Computer Vision, pages 285-302. Springer, 2024. 3
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claim-that we improve attention sparsification and quantization in visual generation models through a novel approach based on attention pattern reorganiza- tion tion-accurately reflects the paper's core contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provide detailed implementation details in Sec. 5.1 and the Ap- pendix, the code for reproducing the result will also be provided in the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code will be provided in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. wttpccccppppccceeeceoc..ccccccct heabblicggiidess. /Code/ebdeeeieeeeeeeee / ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The paper provide detailed implementation details in Sec. 5.1 and the Ap- pendix, the code for reproducing the result will also be provided in the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Despite no error bar is given, we control the seed for all generation, and construct sufficiently large evaluation set to ensure statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The methodology is an effieient post-training compression method, the cost is discussed in detail in "overhead analysis" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics sttps://neurips.. hpppllc//tticsuuuiiiii dinins??
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader societal impacts the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss the safeguards along with socictal impact in the appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The assets properly cited in the paper. and licenses are provided in the code in supplementary.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https:/ http::pppp.p.pooeeeeee////////// for what should or should not be described.
20

---

A Experimental Results for Wan 2.1 Model
We present the results of applying PAROAttn and baseline sparsification and quantization methods to the Wan-2.1[41] 14B T2V model in Tab. 4, along with qualitative results in Fig. 8. Notably, when applying Sparge.Attention/ [63]] we observed numerical instability leading to NaN outputs; hence, its results are omitted from the table. These findings are consistent with those presented for the Cog VideoX model in Table the main paper. We summarize our key observations as follows:
(1) PAROAttn consistently outperforms baseline sparsification methods across varying density. As shown in Tab. 4, PAROAutn significantly outperforms the baseline method Sparse VideoGen across different metrics and settings. Remarkably, PAROAttn with 0.3 density still surpasses Sparse VideoGen at 0.5 density.
(2) PAROAttn preserves both visual quality and content even under more aggressive sparsity. As illustrated in Fig. PAROAttn at 0.3 density produces frames nearly identical to the dense baseline. In contrast, PAROAttn at 0.5 density introduces noticeable degradation with changes in both content and style.
(3) PA ROAttn achieves comparable performance to baseline quantization methods with higher speedups. As demonstrated in Tab. 4 and Fig. 8, PAROAum's quantization scheme further compresses the PV computation to INT8//NT4 on top of the SageAttn baseline, maintaining performance while delivering greater speedup.
Table 4: Performance of PAROAttentinn Wan 2.1 text-to-video generation on VBench prompts. Baselines are evaluated using their official codebases. For fair comparison, we configure Sparse ideoGen without skipping sparsification during the first 30% of timesteps.
Efficiency
Quality
Type
Method
Video Quality Merries
FP Dig Metries
Dense Rate
/ /Bitwidth
CLISSIM
VQAt
AtSeare]
PVD-HPI6
PSNRT
SSIM
ConSim"
FP16 Full Attn.
100.0%
0.215
93.49
0.000
0.000
00
1.000
1.000
SparseVideoGen (0.5)
50.0%
0.199
91.56
0.468
0.476
15.32
0.613
0.900
PAROAti (0.5)
50.0%
0.213
92.85
0.114
0.251
22.02
0.806
0.978
Sparse
VparseVideoGea (0.3)
30.0%
0.196
90.13
0.612
0.679
13.17
0.475
0.839
PAROAti (0.3)
30.0%
0.208
91.97
0.153
0.278
21.73
0.786
0.978
SageAItn Suge/itnV2
OX (INTS), PV (FP16)
0,201
92.24
0.126
0.209
20.43
0.720
0.970
QK (INT4), PV (FPS)
0.200
88.53
1.260
0.749
17.86
0.678
0.954
Quant
PAROAr (INT))
QK (INT8), (INTS)
0.213
92.89
0.128
0.362
20.13
0.706
0.967
PAROAII (INT4)
QK PV (INT4)
0.206
89.77
0.896
0.412
19.30
0.741
0.965
B Additional Qualitative Results and Analysis of Metrics Selection:
We present additional qualitative comparisons of spar- sification methods, along with their corresponding metric scores, in Fig. 9. We compare PAROAttn against SpargeAttn and Sparse 30%. As shown in the figure, PAROAttn produces nearly identical frames at both density levels, whereas StargeAttn and VideoGen introduce noticeable blurriness and content distortion. In particular, SpargeAttn at 30% density exhibits prominent square-shaped color blocks, and the generated content becomes barely recognizable. We further analyze the corresponding changes in metric scores in the next paragraph.
Analysis of Additional Qualitative Results:
on the Cog VideoX model under density levels of 50% and
Findings and Recommendation of Metrics: In Fig. 9, we observe that quality-related metric (VQA) and FP difference metric (PSNR) exhibit different trends as the dense rate varies. Specifically, for PAROAttn, the quality-related VQA metric remains stable even with 30% density, while the FP difference- related PSNR metric gradually decays. It reveals that the FP difference-relatee metrics are more challenging to maintain because they focus on low-level differences and can detect very minor detail changes that quality-related metrics might miss. As result, they are suitable for scenarios where only minor differences are expected. However, they may not be reliable when comparing samples with significant differences, in which case quality metries more appropriate.
21

---

FP16 Full Attn. (100%)
PAROAttn. (50%)
PAROAttn. (30%)
PAROAttn. (INTB)
PAROAttn. (INT4)
PSNR: VOA: 93.48
PSNR: 22.02 22.02
PSNR: 21.73 VQA: 91.97
PSNR: 20.13 VOA: 92.89
PSNR: 19.30
VOA: 92.85
VOA: 89.77
Speedup: 1.00x
Speedup: 1.73x
Speedup: 2.71x
Speedup: 1.83x
Speedup: 2.97x
SparseVideoGen. (50%)
SparseVideoGen. (30%)
SageAttn. INT8, PV FP16)
StgeAttnV2.
(OK INT4, PV FP8)
PSNR: 15.32
PSNR: 92.24
VQA: 91.56
VQA: 90.13
VOA: 88.53
Speedupp 1.42x
Speedup: 2.11x
Speedup: 1.72x
Speedup:
Figure 8: Qualitative results of Wan 2.1 model video generation.
FP16 Full Attn. (100%)
PAROAttn.
PAROAttn. (30%)
SpargeAttn. (50%)
SpargeAttn.
SparseVideoGer (50%)
SparseVideoGen. (30%)
(50%)
FSNR
22.90 VOA 92.66
PENR Speedup: 1.67%
PSNR 18.50 VOA: 90.14
PSNR: 17.53 VOA: 89.54 54
VOA: 92.53
Speedup: 1.73x
Speedup: 2.71x
Speedup: 2.62x
Spancuu: 1.42x
Speedup: 2.11x
Figure 9: Additional qualitative results sparsification for CogVideox VideoX.
Ablation study skipping scheme in Sparse VideoGen. In the original Sparse paper and its official code release, sparsification is deliherately omitted for the first two Transformer blocks and the initial 30% of timesteps. In the main paper, for fair comparison, scheme for Sparse VideoGinn We conduct an ablation study on the effect of this "skipping" scheme, with results shown in Sec. B and Fig. 11. As observed, removing the skipping strategy leads notable degradation in generation quality (PSNR drops from 25.37 to 18.50), significant content distortion, and visibly blurred outputs. In contrast, PAROAttn maintains high generation quality even without skipping early timesteps or transformer blocks.
donot adopt such "skipping"
C Additional Results for CUDA Kernel Efficiency Improvement
We provide detailed results comparing different CUDA kernel implementations in Sec. and Sec. The reported speedups are measured based on attention computation alone (excluding the QKVO projections), using a token length of correspooniiiiiinnggg to 6-second 720P video generation with Cog VideoX. Experiments for sparsification basclines conducted on NVIDIA A100 GPU
22

---

PAROAttn.
SparseVideoGen,
SpareeVideoGen, Seip)
(1005)
(D01))
PSNR 29.14
PÊNR, 25,37
PENIR: 18.50 V0.4-90.14
VoA 02 56
VOA:: $1.30
Speedup: 1.00x
Speeduxx 1.73k
Speeous: 1,42x
Speeellg:: 1.42c
Method
PSNRT
SSIM
CosSim
SparseVidcoGes VideoGia (0.5, w.o, skip)
18.50
0.755
0.960
SparscVideoGinn (0.5, skip)
25.37
0.871
0.984
PAROAtention
0.997
Figure 10: Ablation of skipping timestep and transformer blocks for Sparse CparseVideoGen.
Figure 11:
Qualitative examples from the ab- lation study skipping timesteps and Trans- former blocks in
(consistent with the supported hardware in the baseline code release), while quantization results evaluated on an RTX 4090 GPU to leverage support for INT4 and FP8 quantization.
Comparison of Lateney Speedups: We detailedly present the experimental results for PAROAttn's CUDA kernel implementation with baseline sparsification and quantization methods in Sec. C. We discuss the findings in Sec. 5.2 "Hardware Resource Savings" of the main paper in detail as follows:
Baseline Sparsification methods exhibit notable performance degradation. Both the VineoGnn and SpargeAttn achieve PSNR lower than 20, even with a relative high density of 50%, worse than the PAROAttn with both sparsification and quantization applied (0.3+INT8 with PSNR 21.49, 0.5+INT4 with PSNR 24.34).
PAROAttn's sparsification method can generate nearly identical frames, even with lower dense rate. As seen in Sec. C and Fig. 9, the PAROAttn generation result highly resembles the full-precision baseline, while achieveing substantial speedups.
The PARO token reordering is compatible with dynamic sparsification approaches. In Sec. C, the "SpargeAttention (0.3 PARO)" means adopting the PARO token reordering with SpargeAttn, it notably improves the performance to exceeding the "SpargeAttn (0.5)", and improve the speedup from 1.67x to 2. 1x.
PAROAttn introduces minimal overhead. The overhead of sparsification is presented in Sec. C, since the PAROAttn adopts static sparse scheme, avoids the online static mask generation overhead. The remaining overhead is the online permutation, and loading of the sparse mask, which are also diminished with the kernel fusion and prefetch techniques, discussed further in the "overhead of permutation/ppefetch" paragraph below.
PAROAttn supports quantization of PV to lower-bit formats Comparing the PAROAttn (INT8/4) with StgeAttn (V1/V2), PAROAttn could further quantizes the PV computation from FP16/FP8 to INT8/INT4 with similar performance, and notable better speedup (1.72x to 1.83x, and 2.56x to 2.97x).
Overhead of Permutation: We present an overhead analysis of integrating permutation within the Rotary Position Embedding (RoPE) operator. As shown, this integration introduces only negli- gible overhead (0.03%), demonstrating that permutation can be fused with prior operators without performance impact.
Overhead of Prefetch: As discussed in Section 4.2 of the main paper, static sparse attention introduces additional memory overhead due to the need to store the sparse mask in GPU memory. Since we adopt timestep-win -iie and transformer-block wise sparse masks, the memory cost of storing the binary sparse mask is approximately 1GB. To mitigate this cost, we introduce a prefetch scheme that only loads the sparse mask for the current transformer block and timestep, reducing memory usage to the KB level. Additionally, we employ double-buffering pipeline technique that allows for simultaneous sparse mask loading and attention computation, minimizing the time spent on sparse mask loading. Overall, the prefetching incurs only 0.33% of the total latency.
23

---

Method
PSNR
Speedupt
Overhead +
Method
PSNR
Speedup
FlashAttention
1.0xx
FlushAttention
1.00x
SpurgeAttention (0.5)
16.80
L67x
6%
SageAttaV1
29.58
1.72x
VparseVideoGen (0.5)
18.50
1.42x
10%
PAROAtt (INT8)
29.01
1.83x
PAROAnention (0.5)
29.14
1.73x
0%
SageAmtn22
24.46
2.56x
SpargeAttention (0.3)
15.22
2.62x
9%
PAROAR (INI4)
24.16
2.97x
SpargeAtenttion (0.3 PARO)
16.89
2.62x
9%
PAROAti (0.3 INT8)
21.49
5.72x
VidrseVi (0.3)
17.53
2.11x
15%
PAROAtO (0.5 INT4)
24.34
9.28x
PAROAttentio (0.3)
22.90
2.71x
0%
Table 6:
Comparison of latency speedup for quantization methods on NVIDIA RTX4090.
Table 5: Comparison of latency speedup for sparsifica- tion methods on NVIDIA A100.
Table 7: Overhead of permutation. The latency comparison of whether adopting permutation to rope operator. The "w." and "w.o." stands for with and without
W.O. permute
w. permute
overhead
Latency (ms)
1.2488
1.2492
0.03%
D Implementation Details and Analysis of Baseline Sparsification Methods
We select MInference [18), DiTFastAttn [57]. Sparse VideoGen [46], and SpargeAttention [63] for video generation. We visualize the sparse mask generated by baseline sparse methods in Fig. 12.
Implementation details:
MInference:
We adapt the sparse attention scheme designed for language models to visual attention masks, making the following modifications: First, since we skip sparsification for the larger text tokens, the "attention sink" meenomenon-whee the first few tokens are significantly larger than the rest-is not observed. As result, the "A-shaped" pattern degrades single diagonal pattern, which can be viewed special case of the "vertical- slash" pattern. We select between the remaining "vertical-slad and "block-wise" patterns based on cosine similarity, following the original implementation. Consistent with the original paper, the "vertical-slash" pattern is determined by selecting the top K% of vertical and diagonal lines with the largest summed values, where K can be tuned to adjust the sparsity rate. For the "block-wise pattern, we use 8x8 blocks and determine whether to retain each block based its summed value.
DitFastAttn: Consistent with the original paper, we determine the window length by selecting the smallest window where the sum values within the window reaches %% of the total attention values.
Sparse VideoGen: We use the official code implementation, the num-sampled-rows are cho- sen as the default value 32 and 64 for Cog Video and Wan. Specifically, for fair comparison, donot adopt skipping sparsification for the first timesteps, and set first-times-i as O. We also present the ablation such skipping scheme in Sec.
SpargeAttn: We use the official code implementation to test the performance of the Cog VideoX model. When adapting SpargeAttn to Wan, numerical stability issue arises, causing the attention computation to produce NaN values; therefore, we omit these results. The hyperparameters for SpargeAttn are tuned using the script provided in the official code.
Table 8: Overhead of prefetching. The latency comparison of whether adopting prefetch for attention.
prefetch
prefetch
overhead
Latency (ms)
1296.5
1300.8
0.33%
24

---

For a density of 50%, we set /1 = 0.09 and ptj1 0.095. For density of 30%, we choose [1 0.13 and p 0.135.
Visualization of attention masks: We present a comparison of sparse masks for PAROAttention and baseline sparse attention methods. The first column shows the post-softmax attention patterns. As can be seen, for the Sparse VideoGinn method, while successfully identifies the "diagonal in block" temporal attention pattern, the diagonal selection within the block remains inaccurate at relatively high dense rate. contrast, PAROAttn effectively preserves attention values while exploiting sparsity. For DiTFastAttn, the window-based attention struggles with the multiple diagonal pattern and fails to capture diagonals located far from the center. Similarly, MInference's diagonal pattern is unable to accurately preserve the naturally block-wise attention pattern.
SparseVideoGen
PAROAttn L1 Norm: 0.0135
RMSE: 0.1163
Cos: 0.6819
DITFastAttn
PAROAttn
Norm: 0.2
0.0321
0.9227
Cos: 0.9808
Cos: 0.9993
MInference L1 Norm: 0.0629
PAROAttn Norm:
RMSE: 0.0348
RMSE: 0.0027
Cos: 1.0000
Figure 12: Comparison of attention masks for PAROAttention and baseline sparse attention methods. We present the relative difference metries (L1 Norm, RMSE, CosSim) measure the difference between the original and masked attention map.
E
The Effectiveness of PAROAttention Quantization Technique
To demonstrate the effectiveness of PAROAttention's 's quantization technique, we present the data distribution within the quantization group (a 64x64 block) in Fig. 13. As shown, similar values successfully aggregated into localized blocks, and the outliers present in the original data distribution are significantly reduced. This reduces the incoherence range from 200-1200 to 12-20, and thus significantly reducing the quantization error.
Incoherence Analysis:
Comparison with FP8 quantization: Sage (Table 6), the authors analyze the cosine similarity between the quantized P matrix and its original FP counterpart, concluding that INT8 quantization introduces too much error and thus opting for FP8. However, after applying pattern- aware token reordering, the incoherence within the attention map data groups is significantly reduced,
25

---

Incoherence: 1273.8
Incoherence: 216.42
Incoherence: 12.72
Incoherence: 20.55
Figure 13: Incoherence for data within the quantization group before and after permutation.
Norm
RMSE
1 C.sSSim
4003
50
180
44
155
2312
Cdt 6325 6315
C.1 DENS
-1280
Valaes
Figure 14: Quantization error with respect to FP for quantization of the attention map P. The red "FP" stands for the FP8 quantization error.
leading to notable decrease in quantization enror. As shown in Fig. 14, the INT8 quantized attention map achieves significantly higher FP difference metrie scores compared its FP8 counterpart. This because INT8 provides mantissa bits to accurately represent subtle value differences.
Reasons for exploring interger quantization: Despite FP8 quantization achieves good performance and valid acceleration with easy deployment. We summarize the reasons for exploring integer quantization follows:
Lower quantization error: Low-bit floating-pons seatingg-oiittiie formats consist of both exponent bits and mantissa bits. For example, the E5M2 FP8 format has 5 exponent bits and 2 mantissa bits. The reduced number of mantissa bits limits its ability represent small value differences, potentially leading to performance degradation. In contrast, with the same bitwidth, integer formats provide more mantissa bits (e.g., equivalently mantissa bits for INT8), enabling them to preserve subtle data differences and achieve lower quantization error. By applying proper preprocessing remove outliers within data groups, the need for additional exponent bits to handle large dynamic variations is reduced. This advantage becomes even more pronounced lower bitwidths, such as 4-bit, where FP4 formats have only 1-2 mantissa bits. As presented in Fig. 5 in the main paper, the all INT4 PAROAttention quantized Flux model could still generate images with high quality. To conclude, integer quantization's 's representation power is essential for lower bitwidth quantization.
Support non-GPU hardware platforms. Despite Nvidia TensorCore [1] demonstrate simialr computing power for FP8 and INT8 matrix multiplication. However, for domain- specific accelerator design [2], adopting INT8 matrix multiplication could be more resource- efficient than FP8. Therefore, integer quantization is valuable for AI accelerator hardware design beyond GPU.
F Generalization of Sparse Attention Mask
present a visualization of the post-softmax attention patterns across different timesteps, prompts, and classifier-free guidance (CFG) settings in Fig. 15. The relative metric scores (L.1 Norm, RMSE, cosine similarity) are calculated based on the attention pattern timestep 5, as indicated by the red text. As shown, the type of attention pattern remains consistent across timesteps, prompts, and
26

---

CFG. However, the detailed attention pattern may vary over timesteps, gradually stabilizing in later timesteps. To address this, we design timestep-wise sparse masks and share the sparse mask for later timesteps.
T FARE LO0S
Oor 50613
9999
FMO 2011
ENGL L.W
PMG A3030
RASE 68334
har 03618
TNdD 1400
-003
115
Neve 8777
Deree 1884
Trenk 31341
1906 QUIER
WAA MURT
ARALL AURIS
Figure 15: Visualization of post softmax attention pattern for different timesteps, prompts, and -lassuaer-rree-uuudnne (CFG). The metric scores are calculated relative to the attention pattern with red text.
G Additional Visualization of Permutation for Flux
We present the attention pattern for flux under different permutations. The permutation also effectively produces concentrated and regular block-wise pattern.
60
60
60
60
40
20
20
60
20
40
20
40
60
20
40
60
[H, W]
Selected [W, H]
Selected [H, W]
[W. H]
60
60
40
40
40
20
20
201
0
20
40
60
20
40
60
0
20
40
60
0
20
40
60
[H, W]
Selected [W, H]
Selected [H, W]
[W,
H]
Figure 16: Visualization of the attention pattern for flux under different permutation.
27

---

H Discussion of application of PAROAMtn
As discussed in the "Discussion of Adaptability" section of the main paper, we introduce pattern- aware token reordering (permutation) as a universal and efficient preprocessing step for attention patterns. Its effectiveness stems from the unique properties of vision transformers, where 3D (or 2D) spatial information is flattened into ID token sequence, disrupting local adjacency. By concentrating attention patterns into more regular and block-wise structures, benefits wide range of application scenarios.
For Compression Techniques: The advantages of this approach extend beyond the specific design of PAROAttention and are applicable to various compression techniques, such as timestep-wise sharing [30], efficient reasoning [68. 10, 12], efficient architecture design [25, 52, 47]. For dynamic sparse attention methods, such as SpargeAAtt [63], the relative importance of blocks becomes more apparent, simplifying the task generating sparse masks from QK embeddings. Additionally, the concentrated patterns can improve caching strategies for feature reuse.
For Model Training: PAROAttention's permutation design also sheds light the meaning of attention patterns, which could inspire future improvements in model training. For instance, it could encourage different attention heads to focus aggregating information along different dimensions, leading to more specialized and efficient learning.
The effectiveness of permutation arises from the unique properties of vision transformers, but its applicability is not limited to visual generative models. could potentially be extended to multi-modal large language models and large vision models for perception tasks, offering similar benefits in these domains.
Beyond Visual Generative Models:
I Limitations and Broader Impacts
The methodology can be further improved from several perspectives. Permutation represents constrained subset of possible token reordering, and we adopt simple block sum as sparse metric, exploring more advanced reordering techniques sparse metric could further enhance performance. PAROAttn introduces a novel direction by leveraging token reordering to reorganize attention patterns. The idea not limited post-training compression, and could be extended broader applications. such as enabling native sparse attention or training acceleration.
28


Reviews:
[{"review_id": "qAsbw4bZKP", "reviewer": "Reviewer_wfYy", "summary": "The paper proposes ​PAROAttention, a novel method to accelerate attention mechanisms in visual generation models (e.g., diffusion transformers) via ​pattern-aware token reordering (PARO)​. Key innovations:\n​1) Token Reordering: Unifies diverse attention patterns (block-wise, multi-diagonal) into hardware-friendly block-sparse structures by permuting token dimensions ( [F,H,W] → [H,W,F]).\n​2) Joint Sparsification & Quantization: Enables ​20-30% density sparsity​ and ​INT8/INT4 quantization​ with near-lossless quality.\n3) ​Hardware Efficiency: Achieves ​1.9–2.7× speedup​ via CUDA-optimized kernels and minimal overhead (<1%).", "questions": "Question:\n) Insufficient Long-Sequence Validation​: PARO primarily evaluates 6-second clips, lacking validation for critical long-sequence scenarios (>10s videos) where 3D full-attention models typically struggle.\n\nSuggestion:\nA highly relevant study, \"Sparse VideoGen: Accelerating Video Generation with Spatial-Temporal Sparse Attention by 2× with High Fidelity\" (SVG Project), demonstrates significant performance improvements through token reordering strategies. While our approach shares the fundamental concept of optimizing local sparsity through token permutation, a systematic comparison would be valuable to elucidate both methodological similarities and key distinctions. Would it be possible to conduct a comprehensive comparison between these works, either through experimental validation or theoretical analysis?", "limitations": "In video generation, attention mechanisms typically manifest in two distinct forms: ​spatial attention, which focuses on local pixel relationships within individual frames, and ​temporal attention, which tracks object movements across frames. The paper fails to adequately differentiate between these fundamentally different attention patterns, potentially compromising the efficiency of its proposed approach.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) ​The primary contribution is indeed ​token reordering (PARO)​, a straightforward yet effective engineering approach to optimize attention computation. \n2) The proposed algorithm outperforms baselines (DiTFastAttn, SpargeAttn) by ​8–12% accuracy​ at 50% sparsity.\n\nWeaknesses:​\n1) The proposed algorithm lacks of theoretical depth. There is no analysis of why certain permutations work best or bounds on reordering error.\n2) Table 3 shows PARO's performance degrades without timestep sharing, indicating weaker adaptation to temporal dynamics.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer wfYy for the time and efforts, constructive feedback, and insightful suggestions, which are invaluable for improving our paper. We are encouraged by the reviewers’ recognition of the novelty of token reordering (PARO) and the solid improvement of the performance–efficiency trade-off. We address each of the raised weaknesses and questions as follows:\n\n> **Weakness 1:** *lack of theoretic analysis of why certain permutations work best or bounds on reordering error.*\n\n**Response to W1:** We agree with the reviewer that choosing proper reordering strategy is a critical issue. The scope of the current paper is to introduce the motivation for reorganizing attention patterns, and propose a feasible token reordering strategy to realize this idea. **The current reordering design is relatively simple and serves as a proof of concept to validate its effectiveness**, while leaving room for future improvements to further enhance its potential.\n- **The current PARO plan is empirically verified effective despite simplicty:** The current PARO design is guided by the strong prior of the local aggregation nature inherent in visual feature extraction, which is supported by extensive empirical observations. To reduce the complexity of optimization, we adopt an extremely small search space consisting of only 6 possible permutations. In this setting, identifying *\"which permutations work best\"* becomes tractable through simple enumeration. Despite the simplicity of the design and the limited search space, PARO demonstrates strong pattern reorganization capabilities, effectively generating unified block-wise patterns in most cases, as illustrated in Appendix Figures 5 and 8.\n- **The design of PAROAttn is compatible with more advanced optimization strategies for token reordering**, allowing the pursuit of \"lower bounds on reordering error\". Beyond the PARO reorder, PAROAttn also provides extensive analysis of sparsification design choices and finally adopts a static approach that determines both the sparsity and reordering strategies offline. This design enables more flexible and sophisticated optimization, as it avoids incurring additional online overhead. To further demonstrate this potential, we supplement our experimental results with an extension that integrates the existing PAROAttn framework with balanced K-Means clustering. \n\n|      Method (CogVideo)      | PSNR (videos) | SSIM (videos) |\n|:----:|:----:|:---:|\n| PARO      |     19.61     |     0.753     |\n| PARO+Cluster     |     20.37     |     0.790     |\n\nAs can be seen, combining balanced clustering with the current PAROAttn yields only a slight performance improvement. This is consistent with the results in Q2 below (PAROAttn achieves similar performance with the clustering-based SVG-V2). The underlying reason for this phenomenon is that the clustering objective focuses on grouping similar values together (similar to the \"local uniform\" in the main paper's Fig. 7), rather than achieving a complete \"sparse block\". **These results validate both the effectiveness of the current PARO strategy and its compatibility with more advanced optimization techniques.**\n\n---\n\n> **Weakness 2:** *Table 3 shows PARO performs worse without timestep sharing, suggesting limited temporal adaptability.*\n\n**Response to W2:** We would like to clarify that the purpose of this ablation study is to evaluate the effect of removing timestep sharing by applying a distinct sparse mask for each timestep. It is expected to yield improved results at the cost of increased resource consumption, the observation of no improvement, or even degradation in some metrics suggests that timestep-wise masking does not offer additional performance benefits. This, in turn, **supports the effectiveness and rationality of our proposed timestep-sharing strategy**. In PAROAttention, ***\"temporal adaptability\"*** can be ensured through timestep-wise sparse masks, and the associated memory overhead can be mitigated via prefetching, as discussed in Lines 190–201 of Sec. 4.2. We will carefully revise the manuscript to further improve clarity and prevent potential misinterpretation.\n\n---\n\n> **Q1:** *PARO lacks validation on longer sequences (>10s)*\n\n**Response to Q1:** We agree with the reviewer that long-sequence scenarios pose greater efficiency challenges. **Experimental results and analysis for 10s generation of Wan 2.1 model are provided in Appendix Sec. 1, Table 1, and Figure 1.** Due to space limits, for clarity, we present and discuss part of these results **in Q2 below**. We thank the reviewer for this valuable suggestion and will carefully reorganize and integrate the content into the main text in future revision.\n\n---\n\n> **Q2:** *Suggestion of comparing with SVG project*\n\n**Response to Q2:** We agree that the SVG project is closely related to our work. It consists of two research papers. We conduct a detailed comparison with the initial ICML submission (arXiv:2502.01776) as an important baseline. The updated SVG-V2 paper (arXiv:2505.18875), which introduces token reordering strategies, was published on arXiv on May 24—after the NeurIPS submission deadline of May 15. Therefore, it was not included in our original submission. We now provide a detailed discussion of SVG-V2 as follows:\n\n- **Methodology Comparison:** Both SVG-V2 and PAROAttn recognize the challenge posed by dispersed attention values and propose token reordering as a solution. However, compared to PAROAttn, SVG-V2 adopts a more advanced reordering strategy and dynamic sparsification, aiming for higher potential in algorithmic performance, with the cost of relatively compromised efficiency gains. \n     - **Distinction 1: How to determine the optimal reordering strategy.** As discussed in W1, PAROAttn adopts a relatively simple reordering scheme as a proof of concept, while remaining compatible with more advanced optimization strategies—such as the K-Means-based approach used in SVG-V2.\n         - As shown in the experiments in W1, PARO demonstrates strong effectiveness despite its simplicity, achieving performance comparable to SVG-V2. Moreover, it can be further improved when combined with more sophisticated reordering strategies.\n     - **Distinction 2: Dynamic vs. Static Sparse Approach.** PAROAttn conducts an extensive analysis of design choices (Sections 3 and 4) and ultimately adopts a static sparse approach, whereas SVG-V2 employs a dynamic approach in which both the reordering strategy and sparse mask are generated online.\n          - As discussed in W1, the static approach enables more flexible and sophisticated optimization by avoiding additional online overhead. For instance, it supports time-consuming techniques such as balanced K-Means, which ensures equal token counts across clusters (while SVG-V2 adopts normal kmeans due to online overhead). This design helps mitigate the need for specialized kernels to handle diverse block shapes and alleviates performance degradation caused by load imbalance.\n     - **Distinction 3: Discussion on effects of Quantization.** Additionally, PAROAttn provides discussion and experiments regarding how token reorder benefits quantization. \n\n- **Experimental Validation:** Due to the unavailability of the customized CUDA kernel code for SVG-V2, we focus our comparison on algorithmic performance and provide a discussion of hardware efficiency.\n    - **Algorithm Performance:** We report results on 10-second long-sequence generation using the Wan v2.1 model (as discussed in Q1). As shown, PAROAttn achieves comparable performance to SVG-V2 despite employing a simpler reordering strategy.\n    - **Hardware efficiency:** Compared to PAROAttn, SVG-V2 introduces additional overheads, including the following:\n         - **(1) Overhead from K-Means:** The cost of K-Means clustering is amortized across timesteps and accounts for about 1% of the attention cost. However, this percentage increases and becomes non-negligible for shorter sequences or higher sparsity rates.\n         - **(2) Increased Reordering Overhead:** Unlike PAROAttn, SVG-V2 determines the reordering strategy online, makes it difficult for kernel fusion of the reorder. The standalone reorder for Q, K, V, O can introduce an additional 5–10% overhead of attention cost.\n         - **(3) Load Imbalance:** K-Means clustering introduces variability in the number of tokens assigned to each cluster, requiring customized FlashAttention kernels to support arbitrary block sizes. It also causes load imbalance issues, notably reducing hardware efficiency.\n\n|      Method (Wan)       | PSNR (videos) | SSIM (videos) |\n|:---:|:----:|:-------------:|\n|   SVG-V2     |     25.81     |     0.854     |\n| PAROAttn     |     25.45     |     0.887     |\n\n---\n\n> **Limitation:** *Missing differentiation between spatial vs. temporal attention may hinder method efficiency.*\n\n**Response to Limitation:** We would like to clarify that PAROAttn **adaptively differentiates between spatial and temporal heads when determining the optimal permutation order**. Specifically, it selects the appropriate local aggregation dimension for each attention head. For example, in a permutation order such as [W, H, F], the frame dimension (F) remains contiguous in memory. This order will be selected when the permuted attention map exhibits a block-wise pattern, indicating that the current head is a temporal head; a similar rationale applies to spatial attention. Consequently, the pattern-aware reordering process effectively unifies both spatial and temporal attention into block-wise pattern. It preserves efficiency without requiring explicit differentiation between the two types of attention.\n\n---\n\nWe sincerely thank the reviewer for the time and efforts. We hope our responses have addressed your concerns. Please don’t hesitate to let us know if you have any additional questions or feedback on our response. We will do our best to address any concerns you may have :)"}, {"review_id": "lgxutOCIhS", "reviewer": "Reviewer_YVHL", "summary": "The authors propose PAROAttention, a lightweight post-training compression scheme for vision diffusion transformers. The key idea is to re-order tokens (PARO) so that the heterogeneous “multi-diagonal” attention maps typical of video/image generators become a single, block-sparse pattern. Once unified, the same block layout is exploited for static sparse masks and block-wise INT8/INT4 quantization, both implemented inside a customized FlashAttention-style CUDA kernel. Experiments on CogVideoX-5B and Flux shows that the proposed method achieves 1.9 – 2.7 × end-to-end speed-ups over FP-dense baselines and up to 10 × when sparsity and quantization are combined.", "questions": "Please refer to weaknesses above.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. The novelty of the paper is good. PAROAttention proposes to transform the pattern itself through reordering, which leads to more structured and predictable sparsity and quantization domains. \n\n2. The proposed PAROAttention aligns the sparsification and quantization granularity with FlashAttention’s block-wise execution model. The author of the paper provide CUDA compatible design, which is comprehensive.\n\n3. PAROAttention determines the optimal permutation and sparse masks with just 1–2 prompts, which demonstrate easy calibration process of using such design.\n\n4. The overall results are convincing. The gain of the proposed method is significant compared to baselines.\n\n**Weaknesses**\n\n1. All results are reported without variance or error bars. Could the author of the paper show more experiment detail like how many time performed for experiments? And what is the variance for the experimental results.\n\n2. The method uses static sparse masks determined offline using only 1–2 prompts, which is a good thing. However, at inference stage, I assume the prompt distribution should match the one in the calibration stage. This brings concerns when there is distribution shift between calibration and inference. This is also my question: how does the proposed PAROAttention deal with the distribution shift?", "was_revised": true, "final_justification": "This paper demonstrate solid work on efficient vision generation model. The author address my concerns in the rebuttal. Therefore I would be happy to see this paper is accepted.", "rebuttal": "We sincerely thank reviewer YVHL for the time and efforts, constructive feedback, and insightful suggestions, which are invaluable for improving our paper. We are encouraged by the reviewers’ recognition of the novelty, CUDA-aware comprehensive design, easy practial usage, and solid efficiency improvement. We address each of the raised weaknesses and questions as follows:\n\n> **Weakness 1:** *Variance of the experimental results (missing error bars).*\n\n**Response to W1:** We agree with the reviewer’s suggestion to include error bars to validate the variance in the experimental results. In the main paper, due to the high evaluation cost (video generation across numerous prompts), the experiments were conducted only once. To address this, we have supplemented the experiments by incorporating **additional random seeds to validate the controlled variance** of the evaluation.\n\n| Method (VBench)         | SSIM               | PSNR               |\n|:-----------------:|:------------------:|:------------------:|\n| **PAROAttn (0.5)**  | 0.909 (±0.0107)     | 28.227 (±1.195)    |\n\nAs can be seen, although random seeds have some effect on the absolute values of the metrics, likely due to variations in content which causes varying difficulty for preservation, **The overall variance remains relatively low**. It is also worth noting that, despite variations in the metric values, all methods exhibit a similar trend in value changes (i.e., when a seed has a higher metric value, the values for all methods tend to be relatively higher). This suggests that the variance is due to the \"relative difficulty\" of the seed rather than pure noise.\n\n---\n\n> **Weakness 2:** *How to deal with the distribution shift from calibration and inference prompts.*\n\n**Response to W2:** We agree with the reviewer that generalization across different conditions (prompts and noise) is a critical issue for static sparse attention approaches. **We do not specifically address the \"distribution shift' problem, as our observation shows that 'the attention pattern of the image part remains consistent across different conditions\"**. To support this finding, we provide extensive empirical evidence and hypotheses regarding the underlying reasons.\n - **Empirical observation:** We carefully verify that the attention pattern shows high similarity across different conditions (prompts and noise) through empirical analysis (as discussed in Lines 192-198 of the main paper). We observe an exceptionally high similarity across different prompts (cosine similarity ≥ 0.99). Appendix Figure 8 provides a visualization of the attention patterns, with each row representing a different prompt, and the patterns appear very similar. This indicates that the attention pattern, corresponding to the sparse mask, does not vary significantly across prompts.\n - **Assumption of underlying reasons:** The prompt primarily affects the text-image attention part, which accounts for only a small portion of the attention (<1%) and is therefore omitted for sparsification. The majority of the attention values, which describe the 'inter-image' aggregation, are more related to the model and are invariant to the input prompts. This phenomenon, where the attention pattern and sparse mask are primarily influenced by model weights rather than input activations, has also been observed in prior literature[1,2].\n - **Additional Experiments:** To further verify this finding, we also provide results for another prompt set (VBench V2), using the current sparse mask generated from an out-of-domain prompt set (cogvideo example prompts) to demonstrate the model's generalization ability across prompt domains. As can be seen, PAROAttn still demonstrates good generalization ability across prompts and random noises.\n\n| Method (VBench2)        | SSIM              | PSNR               |\n|:----------------:|:----------------:|:------------------:|\n| **PAROAttn (0.5)** | 0.919 (±0.0161)    | 28.442 (±1.054)     |\n\n\n[1] Jiang et. al., MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention.\n\n[2] Xi et. al., Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity.\n\n---\n\nWe sincerely thank the reviewer for the time and efforts. We hope our responses have addressed your concerns. Please don’t hesitate to let us know if you have any additional questions or feedback on our response. We will do our best to address any concerns you may have :)"}, {"review_id": "cNrIphdeyu", "reviewer": "Reviewer_CDie", "summary": "This paper proposes PAROAttention, a novel and efficient sparse and quantized attention mechanism for visual generation models based on the token reordering technique. By unifying diverse attention patterns into block-wise structures, PAROAttention makes attention sparsification and quantization more hardware-friendly and computationally efficient. Specifically, this work first highlights that the primary challenge in compressing attention in visual generation models arises from the diverse and scattered distribution of visual attention. Then, it further finds that these unique patterns result from the local aggregation of visual information across different dimensions, which can be transformed into localized block-wise structures by gathering the corresponding tokens through token reordering. Moreover, this reordering operation groups similar attention values together, simplifying quantization by eliminating the influence of outliers. Because of the block-wise design, the sparsification granularity naturally aligns with FlashAttention implementations. Consequently, PAROAttention achieves high-quality video generation under aggressive compression settings, including 20% sparsity and INT4 quantization, and delivers up to 3.8× end-to-end speedup, outperforming existing attention compression methods for visual generation models.", "questions": "(1) Table 3 illustrates that removing timestep-share cannot achieve better performance, which is kind of counterintuitive: with precise mask for each step, the performance is supposed to be better. Could the authors please try to explain this interesting results? \n\n(2) In the figure3, it seems that finally we need to do an inverse-reorder operation to recover the original sequence. However, there is no further elaboration in the paper for this operation. Is there anything important to note for this operation such as the overhead of this operation? Will the performance degrade a lot if we do not do this inverse-reorder?\n\n(3) PAROAttention harnesses the offline determined permute order. Are these determined permute patterns model-specific? Moreover, it claims that this offline mask generation is minute-level cost, is this because for different model and different input visual information, we need to do this offline mask generation from scratch?\n\n(4) How does this reorder-then-compress could potential benefits the large vision language models? could you please share some detailed thinkings about this? I hope to see that this idea can have broader impacts beyond visual generation domain.", "limitations": "The major limitations are mainly about the static, offline permutation patterns determination and the sparsity metrics, which are elaborated in the weaknesses and questions above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n**(1) Method:** The core idea of reordering tokens is novel and interesting for compressing attention in visual generation models. It opens up a new perspective on attention compression by avoiding the need to design complex sparsification and quantization methods to adapt to the diverse attention patterns typically observed in visual data.\n\n**(2) Experiments:** The experimental setup in the paper is clear and make sense. The authors provide a comprehensive evaluation across different settings and baselines on both video and image generation tasks. Compared to existing methods, including SageAttn and SparseAttn, PAROAttention achieves high generation quality under extreme compression scenarios, such as 20% density and INT4 quantization for both QK and PV, while delivering up to 3.8× end-to-end speedup. The ablation studies are convincing as well.\n\n**(3) Practicality:** The work also includes the design of specialized CUDA kernels to support PAROAttention, making the approach practical and accessible for evaluation by the community.\n\n**(4) Writing:** Overall, the structure and style of the paper are clear and easy to follow, despite some minor typos (e.g., line 155: “to to”) and some missing details. The figures effectively illustrate the methods and results.\n\n**Weaknesses:**\n\n**(1) Permutation Generality:** The paper claims that the six possible permutations are sufficient to produce unified, block-wise patterns. However, there is no thorough analysis to support this claim. This raises doubts about whether these six static permutations are adequate for handling the diverse and changing content across different samples and domains and visual generation models. This may be one reason the model’s performance could be constrained after compression. Additional discussion and empirical analysis of how permutation patterns influence effectiveness would strengthen the work.\n\n**(2) Parameter Sensitivity:** The parameter sensitivity is not fully elaborated in the paper. For example, the metrics for permutation selection involve several hyperparameters, but it is unclear how sensitive the chosen patterns are to these settings, which is important to understand. Moreover, the block size is set to 64 to align with FlashAttention, but whether 64 is the optimal value remains unexplored. If 64 is too large to effectively capture local information, reducing the block size might further improve accuracy. A more detailed analysis of how block size and other parameters affect performance would be valuable.", "was_revised": true, "final_justification": "After rebuttal and discussions with authors, my final results can be summarized as follows:\n\n**Weaknesses**\n\nFor the concerns about the permutation strategy design space and generalization ability, the authors reply make sense and make me convinced about the design choice;\n\nFor the concerns about the parameter Sensitivity for reorder strategy and sparse block size, the authors' motivation is clear and the results make sense. \n\n**Questions**\n\nAll the questions are answered convincingly by the authors, and the extra experiments on LLada further proves the effectiveness of PAROAttention design. \n\n**Summary**\n\nOverall, I think this work is robust and impact, and it reaches to the requirement of NeurIPS. Therefore, I give it borderline acceptance.", "rebuttal": "We sincerely thank reviewer CDie for the time and efforts, constructive feedback, and insightful suggestions, which are invaluable for improving our paper. We are encouraged by the reviewers’ recognition of the novelty of methodology design, comprehensive evaluation, practicality, and clear writing. We address each of the raised weaknesses and questions as follows:\n\n> **Weakness 1:** *Permutation strategy design space and generalization ability.*\n\n**Response to W1:** We agree with the reviewer that selecting the appropriate reordering (permutation) strategy is fundamental to PAROAttn, and that providing more analysis on this topic would be beneficial. \n\n- **Design space of the reorder strategy.** *\"Whether the six possible permutations are sufficient to produce unified, block-wise patterns.\"*\n  - The current reordering design is relatively simple and serves as a **proof of concept to validate its effectiveness**, while leaving room for future improvements to further enhance its potential.\n  - **The current PARO plan is empirically verified effective despite simplicty:** The current PARO design is guided by the strong prior of the local aggregation nature inherent in visual feature extraction. Despite the simplicity of the design and the limited search space, PARO demonstrates strong pattern reorganization capabilities, effectively generating unified block-wise patterns in most cases, as illustrated in Appendix Figures 5 and 8.\n  - **The design of PAROAttn is compatible with more advanced optimization strategies for token reordering.** PAROAttn adopts a static approach that determines both the sparsity and reordering strategies offline. This design enables more flexible and sophisticated optimization techniques, as it avoids incurring additional online overhead. To further demonstrate this potential, we supplement our experimental results with an extension that integrates the existing PAROAttn framework with balanced K-Means clustering. \n  \n|   Method (CogVideo)      | PSNR (videos) | SSIM (videos) |\n|:-------:|:-------------:|:-----------:|\n| PARO      |     19.61     |     0.753     |\n| PARO+Cluster     |     20.37     |     0.790     |\n  \nAs can be seen, combining balanced clustering with the current PAROAttn yields only a slight performance improvement.  These results validate both **the effectiveness of the current PARO strategy** and its **compatibility with more advanced optimization techniques.**\n\n- **Generalization of the Reorder Strategy:** *\"Are these six static permutations sufficient for handling the diverse and changing content across different samples, domains, and models?\"* We carefully investigate the generalization of the reorder strategy and sparse mask across multiple dimensions.\n  - **(1) Models:** Despite the diverse and changing patterns across models, the six permutations are designed based on a strong prior of the local aggregation nature of visual feature extraction. We present visualizations of the attention patterns for the CogVideo and Flux models in Appendix Fig. 8 and 9, which reveal a clear block-wise pattern, though leaving room for further improvement.\n  - **(2) Timesteps:** We observe that attention patterns vary across timesteps and adopt timestep-wise sparse masks. However, attention patterns of different timestep share similar trend in local aggregation dimension (as seen in Appendix Fig.8). Therefore, we choose the same permute plan across timesteps, but verify the effectiveness of current reorder plan. \n  - **(3) Conditions (Prompt and Random Noises):** The attention patterns exhibit high similarity (CosSim > 0.99) across different conditions, demonstrating that the reorder strategy and sparse mask generalize well. We further supplement our experiments with a new prompt set (VBench 2) and random noise, and the performance remains strong.\n\n| Method (VBench2)        | SSIM (video)             | PSNR (video)             |\n|:---:|:----:|:----:|\n| PAROAttn (0.5) | 0.919 (±0.0161)    | 28.442 (±1.054)     |\n\n---\n\n> **Weakness 2:**  *Parameter Sensitivity for reorder strategy and sparse block size.*\n\n**Response to W2:** We agree with the reviewer and provide an analysis and justification for our choice of hyperparameters.\n- **Hyperparameters for permutation selection:** Since the final task is a simple 1-out-of-6 classification, **the selection of hyperparameters is relatively robust**. We conducted additional experiments and found that varying the `sparse_percentage` from 75% to 95% and the `sparse_threshold` from 1e-4 to 5e-3 does not affect the permutation selection. The `alpha` is used to control the trade-off between sparsity and quantization, the final permutation plan only changes when alpha is close to 0.5. \n- **Block size selection:** The block size of 64 is selected **based on actual profiling.** Intuitively, speedups are positively correlated with block size. However, in practice, when the block size is relatively large and the sequence length is not long enough (e.g., in our case with a sequence length of 17K), larger block sizes, such as 128, perform worse than 64. This is due to the \"spilling\" phenomenon, where the cache and register usage for each block may exceed the hardware limits, causing the CUDA compiler to move some data to lower-speed memory, which sacrifices efficiency. \n\n| | Block 32 | Block 64 | Block 128 |\n|:-------:|:----:|:-------:|:-----:|\n| Seq 17K     | 2.91           | **3.55**           | 3.54           |\n| Seq 32K     | 2.97           | 3.63           | **3.79**          |\n\n---\n\n> **Q1:** *Why removing timestep sharing achieve better performance.*\n\n**Response to Q1:** Removing timestep sharing is expected to yield improved results, at the cost of increased resource consumption. The phenomenon where some metrics (specifically PSNR) appear lower, while SSIM and LPIPS improve, is **due to evaluation noise**, as the performance differences are minimal and difficult to discern by human observers. As seen in Appendix Fig. 8, the attention patterns for later timesteps are very similar, thus could be shared. \n\n---\n\n> **Q2:** *Essentialness of the inverse order and the overhead*\n\n**Response to Q2:** The inverse reordering process is essential to maintain mathematical equivalence with the reordering process. We omit the implementation details of the inverse reordering process as **it is analogous to the reordering process**. It can also be fused within consecutive kernels.\n\nIn practical implementation, the ideal fusion strategy would be to fusing both the reorder and inv-reorder within the QKV linear mapping at the beginning. However, due to the tightly integrated nature of PyTorch’s Linear implementation, it is relatively difficult to hack, and it is also coupled with many other operator-level optimization strategies. Instead, we fuse the reordering process into the relatively \"independent\" rotary embedding layer, making it easier and more flexible for code integration. As a result, the inverse reordering is performed separately. We verify that the overhead of this operation **remains negligible** (<1%), as shown below:\n\n| InvReorder time (ms) | Attn time (ms)  | Proportion | \n|:--:|:----:|:----:|\n| 0.135    | 20.8          |  0.65%          |\n\n---\n\n> **Q3:** *Are permute patterns model-specific? minute-level mask generation need to be conducted for different model and input visual information.*\n\n**Response to Q3:**\n- *\"Permute patterns model-specific? \"* **Yes**, the permute order is closely related to the local aggregation dimension of certain attention head, which corresponds to model weights. \n- *\"Minute-level mask generation conducted for different model and input visual information?\"* The sparse mask generation needs to be performed for each model, but **not** for each input of visual information. The mask generalizes well across different prompts and random noises, as discussed in detail in the W1 response.\n\n---\n\n> **Q4:** *How can \"reorder-then-compress\" benefit large vision language model, the broader impact beyond visual generation of the idea.*\n\n**Response to Q4:** The prefilling stage of autoregressive VLMs share similar computing flow like diffusion models, and the \"local aggregation\" nature is a general characteristic for all visual generation. Specifically, for VLMs, similar \"multi-diagonal\" pattern also appears, as presented in the figures for the papers below (We apologize for the indirectness since NeurIPS rebuttal does not support providing figure or links).\n- Right side of Figure 6 \"BlindSight: Harnessing Sparsity for Efficient VLMs\" (arxiv 2507.09071)\n- Figure 3 of \"MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention\" (arxiv 2504.16083)\n\nWe also apply PAROAttn to **LLada-V[1]**, a diffusion-based vision-language model, to demonstrate its applicability beyond visual generation. We select the MathVerse dataset because it involves high-resolution images, and the image processing accounts for the majority of the computational cost.\n\n| Methods | Mathverse-minivision Score  | \n|:----:|:--:|\n|  Origin    |   28.5         |\n|  Naive (0.75)    |   20.5         |\n|  PAROAttn (0.75)    |   27.8         |\n\n[1] You et. al., \"LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning\", arxiv 2505.16933\n\n---\n\n> **Limitations:**  *static, offline permutation patterns determination and the sparsity metrics*\n\n**Response to Limitations:** To conclude,\n- regarding the *\"static offline permute patterns\"*, we supplement analysis of design space and generalization (W1 & Q3)\n- regarding the *\"sparsity metrics\"*, we supplement analysis of hyperparameter robustness (W2), and generalization (W1 & Q3). \n\n---\n\nWe sincerely thank the reviewer for the time and efforts. We hope our responses have addressed your concerns. Please don’t hesitate to let us know if you have any additional questions or feedback on our response. We will do our best to address any concerns you may have :)"}, {"review_id": "4d8O5psnDA", "reviewer": "Reviewer_A1jm", "summary": "This manuscript propose a novel pattern-aware token reorder (PARO) method for efficient sparse and quantized attention. The authors provide a key insight of the challenges of the attention sparsification and quantifization, and proposed PAROAttention to reorganize and unify the attention. Experimental results the proposed method achieves lower density and bitwidth, and achieving a higher end-to-end latency speedup.", "questions": "Overall, this is a solid method. The following are my major concerns:\n1. The guarantee of the selected permutation. Do the 6 permutations definitely contain a case that minimizes the diversity of the attention variants under a certain value? If yes, are there empirical or theoretical justifications?\n2. The generalization of the proposed method. Is the proposed method applicable to other attention-based methods on different tasks? If not, please justify the reasons.\n3. The ability boundary of the proposed method is expected to be justified.\n\nI'd like to raise my rating if my major concerns are addressed.", "limitations": "The generalization and ability boundary of the proposed method are expected to be justified.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1.\tThe observation and motivation of the proposed method are interesting.\n2.\tThe proposed method PARPAttention, is hardware efficient.\n3.\tExperiment results demonstrate good performance. \n\nWeakness:\n\n1.\tA more detailed explanation of the figures is expected. For example, in Figure 2 (b), why is the left attention map hard to sparsify and quantize?\n2.\tFor the pattern-aware reordering, is there a guarantee, such as sparsity, on the best permutation choice from the 6 possible cases? What is the worst case of the chosen best permutation?", "was_revised": true, "final_justification": "The authors have addressed my concerns about the method in the worst-case scenarios", "rebuttal": "We sincerely thank reviewer A1jm for the time and efforts, constructive feedback, and insightful suggestions, which are invaluable for improving our paper. We are encouraged by the reviewers’ recognition of the motivation, hardware efficient and solid performance improvement. We address each of the raised weaknesses and questions as follows:\n\n> **Weakness 1:** *More detailed explanation of Fig.2, why left one is hard to sparsify and quantize.*\n\n**Response to W1:** We thank the reviewer for pointing out the clarity issue in the explanation of Fig. 2. The explanation is provided in Section 3 of the main paper, and we will discuss it in detail here and reorganize the description in a future version.\n\n- **Sparsity:** As discussed in Line 103, Section 3 of the main paper, structured sparsity is essential for achieving practical speedup. The lefter attention patterns in Fig.2  are **diverse in shape**, which requires a complex sparse mask design to match them accordingly. Additionally, the **large attention values are scattered and dispersed**, resulting in a lack of complete sparse local regions, which prevents the formation of hardware-friendly structured sparsity.\n- **Quantization:** As discussed in Line 117, Section 3 of the main paper, prior research [1] summarizes that the major source of quantization error arises from large variations within data groups. Since **the large attention values are scattered and dispersed**, significant variation exists within local data groups, which, in turn, increases quantization error.\n\n[1] Zhao et. al., ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation, arxiv 2406.02540\n\n---\n\n> **Weakness 2:** *Is there a guarantee of sparsity 6 permutation choices? What is the worst case of chosen best permute?*\n\n> **Q1:**  *Empirical or theoretical justifications The guarantee of the selected permutation*\n\n**Response to W2 and Q1:** We agree with the reviewer that the selection and effect of reorder strategy is a key issue. We first provide direct answer to the raised 2 questions, then provide analysis of the reorder strategy design. \n- ***\"Is there a guarantee of sparsity 6 permutation choices?\":*** Due to the naturally diverse patterns, there is no theoretical guarantee for the sparsity rate. However, the PARO reordering plan ensures the selection of the **\"relatively better\"** plan through simple enumeration of the carefully crafted search space. Since the search space extensively captures the local aggregation characteristics, which are inherent to visual feature extraction, it is **highly likely to contain a good sparse case**. This approach has been **empirically proven effective** in producing structured sparsity, as demonstrated by the visualization examples in Appendix Figures 5 and 8, and the experimental results in the \"Discussion of reorder strategy\" section below.\n- ***\"What is the worst case of chosen best permutation?\":*** For some cases, the simple permutation has room for improvement, such as presented in lower-right Fig. 9. \n- **Discussion of the reorder strategy:**\n  - The current reordering design is relatively simple and serves as a **proof of concept to validate its effectiveness**, leaving room for improvements to further enhance its potential.\n  - The design of PAROAttn **is compatible with more advanced optimization strategies for token reorder**, we adopt a static approach that determines both the sparsity and reordering strategies offline. This design enables more flexible and sophisticated optimization, as it avoids incurring additional online overhead. \n  - To further demonstrate this potential, we supplement our experimental results with an extension that integrates the existing PAROAttn framework with **balanced K-Means clustering**. \n\n|   Method (CogVideo)      | PSNR (videos) | SSIM (videos) |\n|:-------:|:-------------:|:-----------:|\n| PARO      |     19.61     |     0.753     |\n| PARO+Cluster     |     20.37     |     0.790     |\n\nAs can be seen, combining balanced clustering with the current PAROAttn yields only a slight performance improvement. These results empirically validate both **the effectiveness of the current PARO strategy** and its **compatibility with more advanced optimization techniques**.\n\n---\n\n> **Q2:** *Is PAROAttn applicable to other attention-based methods for different tasks?*\n\n**Response to Q2:** Yes, the redundancy of global attention in visual feature extraction tasks with a local aggregation nature is a common problem. The PAROAttn token reordering technique is applicable to **a variety of tasks beyond visual generation, such as  multimodal understanding (vision language models).** \n\nWe apply PAROAttn to **LLaDa-V [2]**, a diffusion-based vision-language model, to demonstrate its applicability beyond visual generation. We select the MathVerse dataset because it involves high-resolution images, and the image processing accounts for the majority of the computational cost.\n\n| Methods | Mathverse-minivision Score  | \n|:----:|:----:|\n|  Origin    |   28.5         |\n|  Naive (0.75)    |   20.5         |\n|  PAROAttn (0.75)    |   27.8         |\n\n[2] You et. al., \"LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning\", arxiv 2505.16933\n\nBeyond diffusion-based methods, Similar \"mutliple-diagnoal\" pattern also exists for the prefilling of VLMs, as illustrated in the figures in the papers below (We apologize for the indirectness since NeurIPS rebuttal does not support providing figure or links).\n- Right side of Figure 6 \"BlindSight: Harnessing Sparsity for Efficient VLMs\" (arxiv 2507.09071)\n- Figure 3 of \"MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention\" (arxiv 2504.16083)\n\n---\n\n> **Q3:** *Justification of the ability boundaries of PAROAttn*\n\n**Response to Q3:** We discuss the ability boundary of PAROAttn from the following aspects: \n- **Performance Upper Bound:** For standalone sparsity and quantization, as shown in Fig. 4 of the main paper, a 12.5% dense rate (8x sparse) causes a noticeable color shift, and INT4 leads to slight jittering across frames, indicating approaching performance boundary. When combining sparsity and quantization, \"30% dense rate + INT8\" and \"50% + INT4\" represent boundary cases that extensively preserve generation quality, achieving 5.8x and 9.1x end-to-end attention speedup, respectively. More aggressive compression plans are expected to cause notable degradation, except in extreme application scenarios, such as very long sequence generation, where there is inherently more redundancy.\n- **Application Field:** As discussed in Q2, PAROAttn addresses the generic problem of exploiting spatial redundancy for visual feature extraction. It can be widely applied not only to diffusion-based visual generation but also to the pre-filling stage of autoregressive models and multimodal understanding tasks. It is suitable for **any application scenario that requires compressing 'inter-image' attention**, but may need adaptation for attention mechanisms that describe 'text-image interaction' or the decoding stage of autoregressive models, which may exhibit distinct attention pattern characteristics.\n---\n\n> **Limitations:** *The generalization and ability boundary of the proposed method*\n\n**Response to Limitations:** We detailedly discuss the generalization of the method in Q2, and discuss the ability boundary in Q3. \n\n---\n\nWe sincerely thank the reviewer for the time and efforts. We hope our responses have addressed your concerns. Please don’t hesitate to let us know if you have any additional questions or feedback on our response. We will do our best to address any concerns you may have :)"}]

Author rebuttal:

Meta-review: The final ratings for this paper are unanimously positive (two "Accept" and two "Borderline Accept").

The paper introduces PAROAttention, a framework for accelerating visual generation models. The core idea of reorganizing attention patterns via a simple token-reordering strategy and speedups with minimal quality loss are strong contribution of this paper.

Initial reviews raised valid questions about the generalizability of this static approach. In a rebuttal, the authors addressed these concerns by demonstrating the method's robustness on an unseen dataset (VBench-V2) and applying it to a different architecture, a diffusion-based VLM (LLaDa-V).

The AC agrees that this is a practical contribution and recommends the paper for acceptance.
Decision: Accept (poster)