Title: PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Low-bit Quantization

Sparse Attention

Diffusion model

Acceleration
Abstract: In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolution image or multi-frame video generation. To address this, prior research has explored techniques such as sparsification and quantization.
However, these techniques face significant challenges under low density and reduced bitwidths. Through systematic analysis, we identify that the core difficulty stems from the dispersed and irregular characteristics of visual attention patterns. Therefore, instead of introducing specialized sparsification and quantization design to accommodate such patterns, we propose an alternative strategy: "reorganizing" the attention pattern to alleviate the challenges.
Inspired by the local aggregatin nature of visual feature extraction, we design a novel **P**attern-**A**ware token **R**e**O**rdering (**PARO**) technique, which unifies the diverse attention patterns into a hardware-friendly block-wise pattern. This unification substantially simplifies and enhances both sparsification and quantization.
We evaluate the performance-efficiency trade-offs of various design choices and finalize a methodology tailored for the unified pattern.
Our approach, **PAROAttention**, achieves video and image generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density (**20%-30%**) and bitwidth (**INT8/INT4**), achieving a **1.9 - 2.7x** end-to-end latency speedup.


Peer reviews:
comments: SUMMARY:
This manuscript propose a novel pattern-aware token reorder (PARO) method for efficient sparse and quantized attention. The authors provide a key insight of the challenges of the attention sparsification and quantifization, and proposed PAROAttention to reorganize and unify the attention. Experimental results the proposed method achieves lower density and bitwidth, and achieving a higher end-to-end latency speedup.

STRENGTHS AND WEAKNESSES:
Strength:

1.	The observation and motivation of the proposed method are interesting.
2.	The proposed method PARPAttention, is hardware efficient.
3.	Experiment results demonstrate good performance. 

Weakness:

1.	A more detailed explanation of the figures is expected. For example, in Figure 2 (b), why is the left attention map hard to sparsify and quantize?
2.	For the pattern-aware reordering, is there a guarantee, such as sparsity, on the best permutation choice from the 6 possible cases? What is the worst case of the chosen best permutation?

QUESTIONS:
Overall, this is a solid method. The following are my major concerns:
1. The guarantee of the selected permutation. Do the 6 permutations definitely contain a case that minimizes the diversity of the attention variants under a certain value? If yes, are there empirical or theoretical justifications?
2. The generalization of the proposed method. Is the proposed method applicable to other attention-based methods on different tasks? If not, please justify the reasons.
3. The ability boundary of the proposed method is expected to be justified.

I'd like to raise my rating if my major concerns are addressed.

LIMITATIONS:
The generalization and ability boundary of the proposed method are expected to be justified.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PAROAttention, a novel and efficient sparse and quantized attention mechanism for visual generation models based on the token reordering technique. By unifying diverse attention patterns into block-wise structures, PAROAttention makes attention sparsification and quantization more hardware-friendly and computationally efficient. Specifically, this work first highlights that the primary challenge in compressing attention in visual generation models arises from the diverse and scattered distribution of visual attention. Then, it further finds that these unique patterns result from the local aggregation of visual information across different dimensions, which can be transformed into localized block-wise structures by gathering the corresponding tokens through token reordering. Moreover, this reordering operation groups similar attention values together, simplifying quantization by eliminating the influence of outliers. Because of the block-wise design, the sparsification granularity naturally aligns with FlashAttention implementations. Consequently, PAROAttention achieves high-quality video generation under aggressive compression settings, including 20% sparsity and INT4 quantization, and delivers up to 3.8× end-to-end speedup, outperforming existing attention compression methods for visual generation models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n**(1) Method:** The core idea of reordering tokens is novel and interesting for compressing attention in visual generation models. It opens up a new perspective on attention compression by avoiding the need to design complex sparsification and quantization methods to adapt to the diverse attention patterns typically observed in visual data.\n\n**(2) Experiments:** The experimental setup in the paper is clear and make sense. The authors provide a comprehensive evaluation across different settings and baselines on both video and image generation tasks. Compared to existing methods, including SageAttn and SparseAttn, PAROAttention achieves high generation quality under extreme compression scenarios, such as 20% density and INT4 quantization for both QK and PV, while delivering up to 3.8× end-to-end speedup. The ablation studies are convincing as well.\n\n**(3) Practicality:** The work also includes the design of specialized CUDA kernels to support PAROAttention, making the approach practical and accessible for evaluation by the community.\n\n**(4) Writing:** Overall, the structure and style of the paper are clear and easy to follow, despite some minor typos (e.g., line 155: “to to”) and some missing details. The figures effectively illustrate the methods and results.\n\n**Weaknesses:**\n\n**(1) Permutation Generality:** The paper claims that the six possible permutations are sufficient to produce unified, block-wise patterns. However, there is no thorough analysis to support this claim. This raises doubts about whether these six static permutations are adequate for handling the diverse and changing content across different samples and domains and visual generation models. This may be one reason the model’s performance could be constrained after compression. Additional discussion and empirical analysis of how permutation patterns influence effectiveness would strengthen the work.\n\n**(2) Parameter Sensitivity:** The parameter sensitivity is not fully elaborated in the paper. For example, the metrics for permutation selection involve several hyperparameters, but it is unclear how sensitive the chosen patterns are to these settings, which is important to understand. Moreover, the block size is set to 64 to align with FlashAttention, but whether 64 is the optimal value remains unexplored. If 64 is too large to effectively capture local information, reducing the block size might further improve accuracy. A more detailed analysis of how block size and other parameters affect performance would be valuable.\n\nQUESTIONS:\n(1) Table 3 illustrates that removing timestep-share cannot achieve better performance, which is kind of counterintuitive: with precise mask for each step, the performance is supposed to be better. Could the authors please try to explain this interesting results? \n\n(2) In the figure3, it seems that finally we need to do an inverse-reorder operation to recover the original sequence. However, there is no further elaboration in the paper for this operation. Is there anything important to note for this operation such as the overhead of this operation? Will the performance degrade a lot if we do not do this inverse-reorder?\n\n(3) PAROAttention harnesses the offline determined permute order. Are these determined permute patterns model-specific? Moreover, it claims that this offline mask generation is minute-level cost, is this because for different model and different input visual information, we need to do this offline mask generation from scratch?\n\n(4) How does this reorder-then-compress could potential benefits the large vision language models? could you please share some detailed thinkings about this? I hope to see that this idea can have broader impacts beyond visual generation domain.\n\nLIMITATIONS:\nThe major limitations are mainly about the static, offline permutation patterns determination and the sparsity metrics, which are elaborated in the weaknesses and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose PAROAttention, a lightweight post-training compression scheme for vision diffusion transformers. The key idea is to re-order tokens (PARO) so that the heterogeneous “multi-diagonal” attention maps typical of video/image generators become a single, block-sparse pattern. Once unified, the same block layout is exploited for static sparse masks and block-wise INT8/INT4 quantization, both implemented inside a customized FlashAttention-style CUDA kernel. Experiments on CogVideoX-5B and Flux shows that the proposed method achieves 1.9 – 2.7 × end-to-end speed-ups over FP-dense baselines and up to 10 × when sparsity and quantization are combined.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The novelty of the paper is good. PAROAttention proposes to transform the pattern itself through reordering, which leads to more structured and predictable sparsity and quantization domains. \n\n2. The proposed PAROAttention aligns the sparsification and quantization granularity with FlashAttention’s block-wise execution model. The author of the paper provide CUDA compatible design, which is comprehensive.\n\n3. PAROAttention determines the optimal permutation and sparse masks with just 1–2 prompts, which demonstrate easy calibration process of using such design.\n\n4. The overall results are convincing. The gain of the proposed method is significant compared to baselines.\n\n**Weaknesses**\n\n1. All results are reported without variance or error bars. Could the author of the paper show more experiment detail like how many time performed for experiments? And what is the variance for the experimental results.\n\n2. The method uses static sparse masks determined offline using only 1–2 prompts, which is a good thing. However, at inference stage, I assume the prompt distribution should match the one in the calibration stage. This brings concerns when there is distribution shift between calibration and inference. This is also my question: how does the proposed PAROAttention deal with the distribution shift?\n\nQUESTIONS:\nPlease refer to weaknesses above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes ​PAROAttention, a novel method to accelerate attention mechanisms in visual generation models (e.g., diffusion transformers) via ​pattern-aware token reordering (PARO)​. Key innovations:\n​1) Token Reordering: Unifies diverse attention patterns (block-wise, multi-diagonal) into hardware-friendly block-sparse structures by permuting token dimensions ( [F,H,W] → [H,W,F]).\n​2) Joint Sparsification & Quantization: Enables ​20-30% density sparsity​ and ​INT8/INT4 quantization​ with near-lossless quality.\n3) ​Hardware Efficiency: Achieves ​1.9–2.7× speedup​ via CUDA-optimized kernels and minimal overhead (<1%).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) ​The primary contribution is indeed ​token reordering (PARO)​, a straightforward yet effective engineering approach to optimize attention computation. \n2) The proposed algorithm outperforms baselines (DiTFastAttn, SpargeAttn) by ​8–12% accuracy​ at 50% sparsity.\n\nWeaknesses:​\n1) The proposed algorithm lacks of theoretical depth. There is no analysis of why certain permutations work best or bounds on reordering error.\n2) Table 3 shows PARO's performance degrades without timestep sharing, indicating weaker adaptation to temporal dynamics.\n\nQUESTIONS:\nQuestion:\n) Insufficient Long-Sequence Validation​: PARO primarily evaluates 6-second clips, lacking validation for critical long-sequence scenarios (>10s videos) where 3D full-attention models typically struggle.\n\nSuggestion:\nA highly relevant study, \"Sparse VideoGen: Accelerating Video Generation with Spatial-Temporal Sparse Attention by 2× with High Fidelity\" (SVG Project), demonstrates significant performance improvements through token reordering strategies. While our approach shares the fundamental concept of optimizing local sparsity through token permutation, a systematic comparison would be valuable to elucidate both methodological similarities and key distinctions. Would it be possible to conduct a comprehensive comparison between these works, either through experimental validation or theoretical analysis?\n\nLIMITATIONS:\nIn video generation, attention mechanisms typically manifest in two distinct forms: ​spatial attention, which focuses on local pixel relationships within individual frames, and ​temporal attention, which tracks object movements across frames. The paper fails to adequately differentiate between these fundamentally different attention patterns, potentially compromising the efficiency of its proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}