DiTFastAttn: Attention Compression for Diffusion Transformer Models
Authors: Zhihang Yuan
Hanling Zhang
Lu Pu
Xuefei Ning
Linfeng Zhang
Tianchen Zhao
Shengen Yan
Guohao Dai
Yu Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Transformer
Attention
Acceleration
Primary area: diffusion_based_models
TL;DR: Attention Compression for Diffusion Transformer Models

Abstract: Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post-training compression method to alleviate the computational bottleneck of DiT.
We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (1) $\textit{Window Attention with Residual Sharing}$ to reduce spatial redundancy; (2) $\textit{Attention Sharing across Timesteps}$ to exploit the similarity between steps; (3) $\textit{Attention Sharing across CFG}$ to skip redundant computations during conditional generation.

DiTFastAttn: Attention Compression for Diffusion Transformer Models
Zhihang Yuan 1,2
Hanling Zhang 1,2
Pu Lu
Xuefei Ning
Linfeng Zhang
Tianchen Zhao 1.2
Shengen Yan?
Guohao D
Yu Wang'
Tsinghua University
2 Inniingencc Shanghai Jiao Tong University
Abstract
Diffusion Transformers (DiT) excel at image and video generation but face com- putational challenges due the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post- post-trainin trainini compression method to alleviate the com- putational bottleneck of DIT. We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (I) Window Attention with Residual Sharing to reduce spatial redundancy; (2) Attention Sharing across Timesteps to exploit the similarity between steps; (3) Attention Sharing across CFG to skip redundant computations during conditional generation. We apply DiTFastAttn to DIT, EexArt-Sigman for image generation tasks, and OpenSora for video generation tasks. Our results show that for image generation, our method reduces up 76% of the attention FLOOPs and achieves up to 1.8x end-to-end speedup at high-resolution (2k x 2k) generation.
Introduction
Recently, diffusion transformers (DīT) have gained increasing popularity in image (Peebles & Xie 2023 Chen et al., 2024) and video generation (Brooks et al. (002).. However, major challenge with DiTs is their substantial computational demand, particularly noticeable when generating high- resolution content. On the one hand, traditional transformer architectures, with their self attentioo mechanism, have O(L²) complexity to the input token length L. This quadratic complexity leads to significant computational costs as the resolution of images and videos escalates. As demonstrated in Fig. D. the attention computation becomes the primary computational bottleneck during the inference process as image resolution increases. Specifically, a 2K x 2K image tokenized into 16k tokens (Chen et al.) 2024), this requires several seconds for attention computation, on high-end GPUs such as the Nvidia A A100. On the other hand, the inference process of diffusion requires substantial number of neural network inferences due the multiple denoising steps and the classifier-free guidance (CFG) technique (Ho & Salimans 2022).
Previous efforts accelerate attention mechanisms, such Local Attention, Swin Transformer LLu ctt al. 2021), and Group Query Attention (GQA) (Ainslie et al. 2023),, mainly focused on designing the attention mechanism network architecture. While effective in reducing computational costs, these approaches necessitate large retraining costs. Due to the substantial data and computational requirements for training DIT, there is need for post-training compression methods.
In this work, we identify three types of redundancy in the attention computation of DiT inference and propose post-training model compression method, DitFastAttn, to address these redundancies:
Equal contribution. Corresponding author and project advisor: foxdoraamedgmail.com com http://nics-effe httpe/eeeaaaaagcccoiiaeettteeeee www.//ii/aaattttnnnnnmm
Project Website:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Fraction of FLOPs on different resolutions
1.0
0.8
FLOPs Fraction
Raw Attention
DiTFassALtn
Others
0.0
512x512
1024x1024
204Bx2048
Figure 1: Left: The efficiency benefits of applying DiTFastAttn on PixArt-Sigma (Chen al. 2024 when generating images of different resolutions. The Y-axis shows the #FLOPs fraction normalized by the #FLOPs of the original model. Right: The qualitative results of applying DiTFastAttn on 1024x 1024 Pigmrt-Sigma.
(1) Redundancy in the spatial dimension. Many attention heads primarily capture local spatial information, with attention values for distant tokens nearing zero. To reduce the redundancy, we opt to use window attention instead full attention for certain layers. However, directly discarding all attention computation outside the window leads to significant performance degradation. To maintain the performance a training-fre free way, we propose to cache the residual between the outputs of full and window attention at one step and reuse this residual for several subsequent steps. We refer to this technique as Window Attention with Residual Sharing (WA-RS).
(2) Similarity between the neighboring steps in attention outputs. The attention outputs of the same attention head across neighboring steps can be highly similar. We propose the Attention Sharing across Timesteps (AST) technique that exploits this step-wise similarity to accelerate attention computation.
(3) Similarity between the conditional and unconditional inference in attention outputs. We observe that in CFG, the attention outputs of conditional and unconditional inference exhibit significant simi- larity (SSIM 2 0.95) for certain heads and timesteps. Therefore, we propose the Attention Sharing across CFG (ASC) technique to skip redundant computation during unconditional generation.
We conduct extensive experiments to evaluate DiTFastAttn using multiple DiT models, including DIT-XL. (Peebles & Xie 2023) and PixAnt-Sigma (Chen et al. 2024) for image generation, and Open-Sora (Open-Sora 2024) for video generation. Our findings demonstrate that DttFastAtt consistently reduces the computational cost. Notably, the higher the resolution, the greater the savings in computation and latency. For instance, with Pix-rt-Sigma, DiTFastAttn delivers a 20% to 76% reduction in attention computation and end-to-end speedup of up to 1.8x during the generation of 2048x2048 images.
Related Work
2.1 Diffusion Models
Diffusion models (Ho al. 2020 Rombach al. 2022; Peebles Xie 2023 Chen al. 2024
Brooks et al., 2024) have gained significant attention due to their superior generative performance compared to GANs (Creswell et al. 2018). Early diffusion models (Ho et al. [2020] Rombach et
2022) are implemented based on the U-Net architecture,, To achieve better scalability, DiT (Peebles & Xie| 2023) utilizes transformer architecture instead of U-Net. Diffusion transformer's is applied in the fields of image and video generation. PixArt-Sigma (Chen et al. 2024) demonstrates the diffusion transformer's capability to generate high-resolution images up to 4K. Sora (Brooks et
2024) presents the diffusion transformer's ability to generate videos.

---

2.2 Vision Transformer Compression
The computational overhead of attention has garnered significant attention. FlashAttention (Dao
2023) divides the input tokens into smaller tiles minimize redundant memory accesses and optimize latency. Some studies highlight the quadratic complexity of attention computation and
improve efficiency through token pruning. achieved by filtering (Rao et al. 202T; Liu al., 2022 Wu et al. 2023) or merging (Lu et al. 2023] |Huang et al. 2023: Wu et al., 2023) tokens at different
stages of the network. Dynamic ViT (Rao al. 20221) employs a prediction network dynamically filter tokens. Adaptive Sparse ViT (Liu et al. (022) filters tokens by simultaneously considering the attention values and the L2 norm of the features. Lu et al. (202) trains a network with segmentation
labels to direct the merging operations of tokens in regions with similar content. [Huang et al. (2023) conducts attention computations after downsampling tokens and subsequently upsampling to recover the spatial resolution. [Wu et al. (023) demonstrates that deeper layers are more suitable for filtering tokens, whereas shallower layers are more appropriate for merging tokens.
2.3 Local Attention
Various studies have delved into the utilization of local attention patterns, where each token attends to a set of neighboring tokens within fixed window size, aiming to mitigate the computational burden associated with processing long sequences. The concept of local windowed attention was initially introduced by [Beltagy et al. (2020) in Longformer, presenting an attention mechanism that scales linearly with sequence length. Bigbird (Zaheer et al. 2020) extends this idea by incorporating window attention, random attention, and global attention mechanisms, enabling the retention of long-range dependencies while mitigating computational costs. In the realm of computer vision, Swin Transformer (Liu et al. 2021) adopts similar approach by confining attention computation to non- overlapping local windows, utilizing shifted windows across different layers to capture global context efficiently. Twins Transformer(CHe
et al. 2021), FasterViT Vasu et al. 2023). ) and Neighborhood attention transformer (Hassani et al. (023) employ window-base based attention to enhance computational efficiency, leveraging different module designs such global sub-sampled attention and hierarchical attention to exploit global context effectively. In our work, we employ fixed-sized window attention to accelerate pretrained Diffusion Transformer models and introduce a novel technique named Window Attention with Residual Sharing to preserve long-range dependencies for image tokens.
2.4 Attention Sharing
GQA (Ainslie et al., 2023) divides query heads into G groups. Each query retains its own param- eters, while each group shares key and value, reducing memory usage and improving efficiency. PSVIT (Chen et al. 2021) shows that attention maps between different layers in ViT have significant similarity and suggests sharing attention maps across layers reduce redundant computation. Deep- cache (Ma et al. 2023) demonstrates that high-level features in U-Net framework diffusion models are similar across timesteps. Deepeache proposes reusing U-Net's high-level features and skipping intermediate layers' computation to accelerate denoising process. TGATE (Zhang et al. 2024) shows that the cross-attention output of text-conditional diffusion models converges to a fixed point after several denoising timesteps. TGATE caches this output once it converges and keeps it fixed during the remaining denoising steps to reduce computational cost. In DiTFastAttn, we demonstrate the similarity of attention outputs both CFG-wise and step-wise. We also consider the differences in similarity across different layers at various steps to share attention outputs CFG-wise and step-wise.
2.5 Other Methods to Accelerate Diffusion Models
Network quantization is widely used technique for reducing the bitwidth of weights and activations,
effectively compressing both image generation models 4Shang et al. 2023 Zhao et al., 2024b) and video generation models (Zhao et al., (0024)). Scheduler optimization is another popular approach aimed at decreasing the number of timesteps in the denoising process (Song et al. 2020 Lu et al.) 2022 Liu et al.) 2023a). Additionally, distillation serves as an effective method for minimizing the timesteps required during denoising (Salimans & Ho 2022: Meng et al. 2023 Liu et al., 2023b). tttFastAtt offers a complementary solution, as operates independently of the specific quantizataniooo bitwidth, scheduler, and timesteps employed.
3

---

Stop Dimunsion
Window with Residual Share
add recitual
Uncondiicnal Irference
sligrnal
Conditional Inforence
Redundancy the spottal dimemion
Similaity between neghhoring
step =
step step-te t+1
Sharing across Timestop
Layer
Layar
stop
skip
share
mome Similarity
Unconditionsl
conditional and uncondittonal
Attention
Inference
Sharing
Layer
Layer
across
Figure 2: Types of Redundancy and Corresponding Compression Techniques. Left: Redundancy in the spatial dimension, denoising steps, and CFG. Right: Techniques implemented in DiTFastAttn to reduce redundancy for each type. DiTFastAttn employs window attention to minimize attention redundancy, while maintaining performance using residuals. Additionally, attention outputs are shared both step-wise and CFG-wise to reduce redundancy.
3 Method
3.1
Overview
In this section, we demonstrate the redundancy in the inference process of diffusion models with transformers. In the denoising process, identify three types of redundancy, as shown in Fig. 2 (1) Redundancy in the spatial dimension. (2) Similarity between the neighboring steps in attention oufputs. (3) Similarity between the conditional and unconditional inference in attention outputs. To
address these redundancies, we propose three compression techniques, as shown in Fig. 2 (1) In Sec. 3.2 introduce Window Attention with Residual Sharing reduce spatial redundancy. (2) In Sec. 3.3 we introduce Attention Sharing across Timesteps to exploit step-wise similarities, thereby enhancing model efficiency. (3) In Sec. 3.4 introduce Attention Sharing across CFG to reduce redundancy by utilizing similarity between conditional and unconditional generation. In Sec. 3.5 introduce simple greedy method to decide the compression plan, i.e., select the appropriate compression technique for each layer and step.
3.2 Window Attention with Residual Sharing (WA-RS)
We can observe the spatial locality of attention in many transformer layers in pre-trained DiTs. As shown in Fig. (),, attention values concentrate within window along the diagonal region of the attention matrix. Therefore, replacing full attention with fixed-size window attention for some layers can preserve most of the values in the attention matrix during inference. By computing attention values only within specified window, the computation cost of attention can be largely reduced.
However, tokens still attend to small of spatial distant tokens. Discarding these depen- dencies negatively affects model performance. Mitigating this issue using only window attention necessitates large window size to capture these dependencies. Consequently, this approach achieves minimal reduction in computational cost, thereby hindering acceleration efforts.
Cache and Reuse the Residual for Window Attention. To address the aforementioned issue, investigate the information loss caused by using window attention. As shown in Fig. B] a). the residual between the outputs of full and window attention exhibits small variation across steps, unlike the output of window attention. This observation motivates us to cache the residual of window attention and full attention in step and reuse it in subsequent steps.
Fig. ((b) illustrates the computation of WA-RS: at each step, for each window attention layer, we compute the window attention and add residual cached from the previous step to the output. We

---

0.1
dist
attention
Residuel calculation
411
vandoa and
0.08
Softmax
x Of
Of
Wi"
step
step 16
step
Window attentioon
Final attention output
Softmax >
650
step
step
0.1
Window
oulput
dual vindor
step 6+1
Softmaa
x Vies
WR.
Of
W/21 + RR
055
ventia
64
Window attentionn
Final attention output
step t+2 Softmad
Ve22 Wa2 Of
W22 + R²
0.
68
5
(a) Left: Examples of map shows sliding window pattern. Right: MSE proceeding
(b) Illustration of Window Attention with Residual Sharing
step attention
Figure 3: Window Attention with Residual Sharing. (a) Left: Example of the attention map showing the window pattern. Right: The MSE between the window attention outputs in the previous and current step (yellow line) versus the MSE between the output residuals of window and full attention in the previous and current step (blue line). The output residual exhibits minimal changes over the steps. (b) Computation of Window Attention with Residual Sharing. Window attention that illustrates significant changes recalculated. Residuals that change minimally are cached and reused in subsequent steps.
denote the set of steps that share the residual value R, as K, the full attention at step r as O,, the window attention step k as W.. For the first step in the set min (K), the computation of WA-RS goes as follows:
O, , Attention (rrtention(Q..fee K,, V,), AttenttAten Qr, K,., Vr),
W
(1)
R, O, WW,
For subsequent step in the set k E K, the computation of WA-RS goes as follows:
W k WindooAAtentiion (Qe, Kg. Vk);
(2)
O W + R,
3.3 Attention Sharing across Timesteps (AST)
The sequential nature of the denoising process in diffusion models is major bottleneck for inference speed. Here, we compare the attention outputs at different steps during the denoising process. We find that for some layers, the attention outputs certain steps show significant similarity to those of adjacent steps. Fig. a4a presents the cosine similarity between the attention outputs at different steps. We can draw two primary observations: (1) There is noticeable temporal similarity between the attention outputs; (2) This similarity varies across steps and layers.
To exploit this similarity to reduce the computational cost, propose the AST technique. Specifically, for set of steps with their attention outputs similar to each other, we cache the earliest step's s attention output O and reuse it, thereby skipping the computation at the subsequent steps.
3.4 Attention Sharing across CFG (ASC)
Classifier-free guidance (CFG) widely used for conditional generation (Ho & Salimans 2022
Ramesh et al. 2022; Saharia et al. 2022). In each step of the inference process for conditional
5

---

(b) Similarity of Attention Outputs between Conditional Unconditional Generation
Similarity Attention Different Different Layers
Figure 4: Similarity of Attention Outputs Across Step and CFG Dimensions in DiT. (a) Similarity of attention outputs across step dimension in different layers. (b) Similarity between conditional and unconditional attention outputs in various layers different steps
generation, CFG performs two neural network inferences: one with the conditional input and one without. This doubles the computational cost compared with unconditional generation. As shown in Fig. 4b) for many layers and steps, the similarity between the attention outputs in the conditional and unconditional neural network evaluations is high.
Based on this observation, we propose the ASC technique that reuses the attention output from conditional neural network evaluation in the unconditional neural network evaluation.
3.5 Method for Deciding the Compression Plan
The aforementioned techniques, including WA-RS, AST, and ASC, can effectively cut down the computational cost while maintaining the performance. As shown in Fig. 3 and Fig. different layers have different redundancies different time steps. Therefore, it is crucial to properly decide the compression plan, i.c., which techniques should be applied for each layer each step.
We develop a simple greedy method to select the appropriate strategy (a combination of techniques) from strategy list S =[AST, WA-RS ASC, WA-RS, ASCJ for each step and each layer. As shown in Alg. [ we determine the strategies step by step and layer by layer. For each step and transformer layer, apply each of the four compression strategies and calculate the loss between the model outputs with and without compression for the current step, L(O,O'). Then, we select the strategy with the highest computation reduction ratio with loss below a threshold [M 6, where is the layer index and JM is the number of layers in the model. If none of the four strategies meet the threshold, we not apply compression for this layer at that step.
Algorithm 1: Method for Deciding the Compression Plan
:Tannformer Model M, Total Step T', Compression Strategy List S, Threshold 8 Output: Output::ictionsss dictionary dict that stores selected compression techniques Initialize dict
Input
for step t in T' do
O compute the output of the uncompressed M for transformer layer in M do
for E S order by ascending compression ratio do compress layer i step using compression strategy m O' + compute the output of M (1(00'0) < aaj then
update m the selected strategy of layer i and step in dict break
return dict

---

DIT-XL-2 512x512, Threshold 0.15
PixArt-Sigma-XL 1024x1024, Threshold 0.15
PixArt-Siome-XL 2011x2048, Threshold 0.15
0
0
Full Arn
Full Artn
Full Atin
WA-R5
Tercsep
20
Terozap
20
Terostop
ASC
ASC
36
30
30
WA ASC
WA-RS-ASC
VIA RS ASC
AST
20
Layer
Layer
Layer
Figure 5: Compression plan for DiT-XL-512, PixArt-SigmaXLL-1022 and PixArt-Sigma-XLe XL-2K at D6 with the number of DPM-Solver steps set to 50.
Experiments
4.1
Settings
We evaluate DiTFastAttn on three commonly used diffusion transformers: DiT (Peebles & Xie 2023) and Pixart-Sigma (Chen et al. (024) for image generation tasks, and Open-Sora (Open-Sora
2024) for video generation tasks. To demonstrate compatibility with fast sampling methods, we build
our method upon 50-step DPM-Solver for DiT and Pixart-Sigma, and 200-step IDDPM (Nichol Dhariwal
202T for Open-Sora.
For calculating quality metrics, we use ImageNet the evaluation dataset for DiT and MS-COCO as the evaluation dataset for SikArt-Sigman MS-COCO 2014 caption is used as text prompt for Pixart-Sigma models' image generation. To evaluate generation quality, we generate 50k images for DiT models and 30k images for PixArt-Sigma models. Following previous studies, we employ
FID (Heusel et al., 2017). IS (Salimans et al. 2016) and CLIP score (Hessel et al. 2021) as the evaluation metrics. We measure the latency per sample
single Nvidia A100 GPU.
We mean relative absolute error for L(O, O') and experiment with and different thresholds intervals of 0.025. We denote these threshold settings as DI (6=0.025), D2 (8=0.05), ... D6 (5=0.15), respectively. We set the window size of WA-RS to 1/8 of the token size.
4.2 Results on Image Generation
Results of Evaluation Metries and Attention FLOPs. DiTFastAttn is applied to the pre-trained
DiT-XL-2-512, PixArt-Sigma-100 and PixArt-Sigma-2K models. Table displays the evaluation results of these models. For the DiT-XL-2-512 and PixArt-Sigma-110 -00---mgaa- models, configurations D1, D2, and D3 nearly match the performance of the original models terms IS and FID metrics. Comparison of compression effects and evaluation metrics between the three models reveals that as image resolution increases, DiTFastAttn not only achieves greater compression but also better preserves the generative performance of the models.
Compression Plan after Search. Fig. illustrates the compression plan under the D6 setting. For the DiT model, AST and ASC are utilized in the early timesteps, with full attention primarily appearing in the initial attention layers. the first two layers and in the middle attention layers during the intermediate timesteps, while the combination of WA-RS and ASC is notably predominant in the final timesteps. This variability in the distribution of different types of redundancy across models highlights the absence of universal compression strategy, underscoring the necessity for tailored plan searches. Additional compression plans for other settings are provided in A.5
contrast, the PixArt-Sigma model employs AST sporadically in
Visualization of DiTFastAttn's Generation Results. Fig. 6 presents image generation samples from DiTFastAttn. The DI, D2, and D3 configurations of the DYT-XL-2-512 and PixArt-Sigma-10 1024 models demonstrate visual generation quality comparable the original models. contrast, D4, D5, and D6 achieve greater compression, exhibiting slight variations in detail while still producing acceptable-quality images. The PixArt-Sigma-2K model maintains image quality similar to original up to D4, with DS and D6 also generating high-quality outputs. This suggests that our com- pression method effectively preserves generation quality, even when reducing attention computation by over 50% and compressing to 33% higher resolutions.
7

---

Table Image generation performance of DiTFastAttn various image resolutions under various compression ratios. The FID, IS, and CLIP results are marked in different makers. The 'Attn FLOPs represents the fraction computation in the multi-head attention module compared to the raw model.
Model
DiT-XL-2 512
PixArt-Sigma-Xi XL x024x 1024
PixArt-Sigma-XI x x x048
Attn FLOPs
Attn
Attn FLOPs
Score
FID
FID
CLIP
IS
FID
CLIP
FLOPs
Raw
408.16 16
25.43
100%
24.33
55.65
31.27
100%
23.67
51.89
31.47
100%
D1
412.24
25.32
85%
24.27
55.73
31.27
90%
23.28
52.34
31.46
81%
D2
412.18
24.67
69%
24.25
55.69
31.26
74%
22.90
53.01
31.32
60%
D3
411.74
23.76
59%
24.16
55.61
31.25
63%
22.96
52.54
31.36
46%
D4
391.80
21.52
49%
24.07
55.32
31.24
52%
22.95
51.74
31.39
36%
D5
370.07
19.32
41%
24.17
54.54
31.22
44%
22.82
51.21
31.34
29%
D6
352.20
16.80
34%
23.94
52.73
31.18
37%
22.38
49.34
31.28
24%
512x512 DIT
1024x1024 TixArt-Sigma
TAA
2048x2048 PixArt-Sigga
Raw
D1
D2
D3
D4
D5
D6
Figure 6: Image generation samples at various image resolutions under various compression ratios.
4.3 Results on Video Generation
We apply DitFastAttn on OpenSora for video generation with thresholds from 0.01 to 0.06. The results are shown in Fig. 7 Specifically, the reduction in attention computation for these configurations are as follows: 11.68% 27.63% 40.48%, 48. 48.98% 50.75%, 55.76% respectively. Latency at raw setting is 35.79 seconds where as performance with the aforementioned configurations were as follows: 34.79 seconds, 33.39 seconds, 31.74 seconds, 31.2 2. seconds, 31. 31.29 seconds, and 31.17 seconds, achieving 1.02x 1.15x end-to-end speedup. For an extended analysis, additional results
are provided in A.3

---

Figure 7: Comparison of video generation using OpenSora V1.1 at 240p resolution with 16 frames.
Table FLOPs fraction and latency fraction of DitFastAttn Diffusion Transformers comparing with original attention. The latency evaluated the Nvidia A100 GPU.
Model
Seqlen
Metric
ASC
WA-RS
WA-RS+ASC
AST
Atta FLOPs
50%
77%
38%
0%
DIT-XL-2512x512
1024
Attn Latency
59%
85%
51%
4%
Attn FLOPs
50%
51%
26%
0%
PixArt-Sigma-XL. 1024x 1024
4096
Attn Latency
54%
54%
31%
3%
Attn FLOPs
50%
33%
16%
0%
PixArt Siint-Siia--X- XL 2048x2048
16384
Attn Latency
52%
35%
19%
1%
4.4 #FLOPs Reduction and Speedup
Compression Results of FatFast on Various Sequence Lengths. We implement tiTF based on FlashAttention-2 (Dao 2023). Table Tshleesshows the FLOPs fraction and latency fraction of DiTFastAttn in Diffusion Transformers compared with original attention mechanisms. The ASC technique reduces attention computation by 50%, with latency reduction slightly increasing as resolution increases. As resolution increases, WA-RS can reduce attention computation from 77% to 33%, and latency reduction ranges from 85% to 35%. The WA-RS and ASC techniques are orthogonal; they can be used simultaneously without additional overhead.
Overall Latency of DiTFastAttn. Fig. 8 shows the latency for image generation and attention computation decreases when DiTFastAttn is applied. DiTFastAttn achieves end-to-end latency reduction for all three model at all compression ratio settings. PixArt-Sigma-2K shows the best performance, with overall generation latency at D6 being 56% of raw and overall attention latency at 37%. The result indicates that as resolution increases, DiTFastAttn achieves better performance in reducing latency for both overall attention and image generation.
4.5 Ablation Study
FisFastAttn Outperforms Single Methods. As shown on the left of Fig. FatFastAttn maintains higher quality metrics compared individual techniques with the same computation budget. Among single techniques, AST shows the best generation quality. However, beyond 2.2 FLOPs, further compression using AST significantly degrades the outputs, causing the search algorithm terminate. DiTFastAttn supports further compression while maintaining better quality.
Higher Steps Improve DiTFastAttn's Performance. As shown the middle of Fig. compared the performance of DiTFastAttn at different steps. It is evident that as the step increases, DiTFastAttn compress more computation while maintaining quality.
The Residual Caching Technique is Essential in Maintaining the Performance. As shown on the right of Fig. g Window Attention with Residual Sharing maintains better generative performance

---

512x512 DIT-XL
1024x1024 PixArt-Sigma-xL
2048X2048 PixArt-Sigma-XL
Latenoo (5)
Image Generation
Latency
Image Ceneration Attention
Latency
Image Generation
30
Attention
Attention
10
Attention TFLOPS
Attention TFLOPs
Attention TFLOPs
Figure 8: Latency of different resolutions of image generation under different compression ratios. DIT runs with batch size of 8, while PixArt-Sigma models with a batch size of The blue line delineates the latency for end-to-end image generation, whereas the orange line represents the latency of multi-head attention module.
Incepton Score
210
Inception Score
Incept Score
210
200
DiTFastAttn
steps-20
200
ASC
steps-30
WA-RS
steps-40
WA-RS
steps-50
190
1.8
2.0
2.6
0.6
0.8
1.0
1.5
2.0
2.5
Attention TFLOPs
Attention TELOPS Fraction
Attention TFLOPS
Figure 9: Ablation study on DIT-XL-2-512. Examination of methodological impact (Left), timesteps variability (Middle), and residual sharing (Right). "WA' denotes Window Attention without the Residual Share (RS).
than Window Attention at the same compression ratio. Without residuals, window attention results in significant performance drop.
Conclusion
In this paper, we introduce novel post-training compression method, Aiteeti,,,,ntion, to accelerate diffusion models. We identify three types of redundancy: (1) Redundancy in the spatial dimension. (2)
Similarity between the neighboring steps in attention outputs. (3) Similarity between the conditional and unconditional inference attention outputs. And we propose corresponding compression techniques: (1) Window Attention with Residual Sharing, (2) Attention Sharing across Timesteps, (3)
Attention Sharing across CFG. The experiments show that DiTFastAutention significantly reduces cost of attention and accelerates computation speeds.
Limitations. First, our method is post-training compression technique and therefore cannot take advantage of training to avoid the performance drop. Second, our method mainly focuses on inference acceleration instead of VRAM reduction. When AST is applied, the attention hidden states from previous timestep will be stored and will bring extra VRAM usage. Third, our simple compression method may not find the optimal compression plan. Fourth, our method only reduces the cost of attention module.
Acknowledgements
This work was supported by National Natural Science Foundation China (No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164) Tsinghua EE Xilinx AI Research Fund, and Beijing National Research Center for Information Science and Technology (BNRist).
10

---

References
Joshua Ainslie, James Lee Therp,, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-hcad checkpoints. arXiv preprint arXiv:2305.13245 2023.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 005:50004.05150, 2020.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue. Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang. and Aditya Ramesh. Video generation models as world simulators. 2024. URL htttps:/openaii com/research/
video- sileooggneratiinn mottoon-moelssseee as- wwwldwordsiie simulators
Boyu Chen, Peixia Li, Baopu Li, Chuming Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Psvit: Better vision transformer via token pooling and attention sharing. arXiv preprint arXiv:2108. 03428.. 2021.
Junsong Chen, Chongjian Ge, Enze Xie, Yuc Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- Asigma: toak-tt--ttrong training of diffusion transformer for 4k text-to-image generation. arXiv preprint wxXv:2203.04662, 2024.
Xiangxiang Chu, Zhi Tian. Yuqing Wang. Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia. and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers, Advances in neural information processing systems, 14:9355-9366, 2021.
Antonia Creswell, Tom White, Vincent Dumoulin. Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53 65, 2018
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2023.
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6185-6194, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12596 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6853 6851, 2020.
Huaibo Huang, Xiaoqiang Zhou, Jie Cao, Ran He, and Tieniu Tan. Vision transformer with super token sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22690-22699, 2023.
Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang, and Yu Wang. Oms- Optimizing the model schedule for diffusion probabilistic models. In International Conference on Machine Learning, pp. 21915 21936.. PMI.R, 2023a.
Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention arXiv preprint arXiv:2209, 13802, 2022.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-basees based text-to-image generation. In The Twelfth International Conference on Learning Representa- tions, 2023b.
Ze I Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
11

---

Chenyang Lu. Daan de Geus, and Gijs Dubbelman. Content-aware token sharing for efficient semantic segmentation with vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 23631-23662 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelcrating diffusion models for free. arXiv preprint arXtv:2312.00888, 2023.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14297-14306 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171, PMI.R, 2021.
Open-Sora. Project title.
Sora 2024. Accessed: 2024-05-22.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205 4205, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.66122 1(2:3, 2022.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems. 34: 34:13937-13949 1021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan. Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Chasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to eext-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479 36494, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling diffusion models. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022. URL.tttss: /ttp::////ppeeeeeiieee wttp://wwww.eeeeoodddddddaaaaooa
Tim Salimans, Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Yuzhang Shang. Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1972-1981. IEEE Computer Society, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Fastvit: A fast hybrid vision transformer using structural reparameterization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5785-5795, 2023.
Xinjian Wu, Fanhu Zeng. Xiudong Wang. Yunhe Wang, and Xinghao Chen. Ppt: Token pruning and pooling for efficient vision transformers. arXiv preprint a100v22310.018/2, 2023.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, ct al. Big bird: Transformers for longer sequences. Advances neural information prucessing systems, 13:17283-17297. 2020.
Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross- attention makes inference cumbersome text-to-image diffusson models. arXiv preprint arXiv:2404.02747 2024.
Tianchen Zhao, Tongcheng Fang, Enshu Liu, Wan Rui, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, et al. Vidit-q: Efficient and accurate quantization of diffusion transformers for image and video generation. arXiv preprint arXiv:2406. 025400. 2024a.
Tianchen Zhao, Xuefei Ning. Tongcheng Fang, Enshu Liu, Guyue Huang. Zinan Lin, Shengen Yan, Guohao text-to-image Dai, and Yu Wang. Mixdq: Memory-efficient efficient few-step text-to image diffusion models with metric-decoupee mixed precision guantization. arXiv preprint arXiv:2405.1788 17873, 2024b.
12

---

A Appendix
A.1 Societal Impacts
DiTFastAttention can enable more efficient deployment of diffusion transformer models, which have shown remarkable image and video generation capabilities. On the positive side, accelerating these models could democratize access powerful generative AI by reducing computational requirements, and allowing broader adoption for creative and educational applications. However, there are also potential negative societal impacts that must be carefully considered. Highly realistic synthetic media could be exploited to create deepfakes for misinformation, fraud, or non-consensual editing. There are also potential privacy risks generative models can reconstruct personal information from data, While DiTFastAttention does not inherently increase or reduce these risks compared to the original models, widening access makes misuse by malicious actors more likely. Therefore, safeguards against misuse and ethical guidelines for the responsible release of compressed models may be needed as this technology develops.
A.2 Details of the Method for Deciding the Compression Plan
This algorithm has computation complexity of ([S) x [T') x |M] 2²), where [S] is the number of compression strategies (4 in our case), [T'] is the number of denoising steps, |M is the number of transformer layers, and is the sequence length. While the inference time for generating an image using the DiT has computation complexity of O(TT) x [M| x s²). Therefore, the greedy algorithm takes about [) x |M of the image generation time. For example, the inference time for generating a 5 512x512 image using DiT-XL-2-512 is approximately 2 seconds, so the greedy algorithm takes around 224 seconds (2s x 28 transformer layers 4 method candidates) to decide the compression plan, which is a reasonable overhead compared to the overall inference time.
The mean relative absolute error is metric used to evaluate the performance of model by measuring the average relative deviation between the outputs O' and the raw outputs O. It is calculated as follows:
10; -9-0 mox( +
40,00) = 1011 Σ clip
0, 0010
In this equation, |0| represents the number of elements in the raw output vector O. The summation iterates over each element i in the vectors O and O'. For each element, the absolute difference between the raw output O and the output O is calculated. This difference is then divided by the maximum value between |O:] and 10%1 which serves as a normalization factor to make the error relative to the magnitude of the output values. To avoid numerical instability in cases where both O and O are very small or zero, a small positive constant (set to 10-6 in our experiments) is added to the denominator. The clip function ensures that the resulting ratio is clipped to the range [0, 10], preventing extreme values from dominating the overall error. The clipped ratios are summed and then divided by the total number of elements |O] 1 to obtain the mean relative absolute error. This metric provides normalized measure of the average relative deviation between the predicted and raw outputs, with values ranging from 0 to 10 (maximum allowed error).
A.3 Results for Video Generation
As shown in Fig. 10 the subjective evaluation of DiTFastAtt's application to video generation tasks revealed a significant performance distinction across configurations D1 through D6. Configurations DI to D3 demonstrated effective performance, balancing computational efficiency with the retention of visual quality in generated videos. The subjective assessment indicated that videos generated under these configurations were smooth, with natural transitions between frames and preserved details that are critical for video quality. The maintenance of these qualities suggests that the model was able to effectively leverage the redundancies identified and apply the compression techniques without noticeable loss the viewer.
In contrast, configurations D4 to D6, which applied more aggressive compression techniques, resulted in a noticeable deviation from the original video characteristics. The generated videos under D4 to D6 were still smooth and coherent, allowing them to represent the intended narrative or prompt with reasonable accuracy. This suggests that while aggressive compression can compromise certain
13

---

aspects of video quality, it can still be effective in scenarios where computational resources are limited and high level of detail is not paramount.
The subjective results underscore the importance of finding an optimal balance between computational efficiency and generation quality when applying DiTFastAttn to video generation tasks. While configurations DI to D3 offer a promising trade-off, the deviation observed in D4 to D6 highlights the need for careful consideration of the compression parameters. For practical deployment, it is crucial to select DiTFastAttn configuration that aligns with the specific requirements of the application in terms of both performance and output quality.
A.4 Latency Values in Different Settings
Table 3: Latency, FID and IS upon 50-step DPM-Solver. Attn Latency means the latency of self- attention. DiT-XL-2 runs with a batch size of 50000 images used to generate FID and IS score
Model
Resolution
Config
Latency (s)
Attn Latency (s)
FID
IS
Raw
6.66
2.26
25.43
408.16
D1
6.61
2.22
25.32
412.24
D2
6.45
2.05
24.67
412.18
DiT-XL-2 512x512
D3
6.27
1.87
23.76
411.74
D4
6.12
1.72
21.52
391.80
D5
5.96
1.56
19.32
370.07
D6
5.81
1.39
16.80
352.20
Table 4: Latency, FID and IS upon 50-step DPM-Solver. Attn Latency means the latency of self- attention. Models run with batch size of 1. 30000 images used to generate FID, IS, and CLIP score
Model
Config
Latency (s)
Attn Latency (s)
FIDD
IS
CLIP
Raw
12.76
5.30
55.65
24.33
31.27
D1
12.55
5.10
55.73
24.27
31.27
D2
11.98
4.49
55.69
24.25
31.26
PixArt-Sigma-X 1024x 1024
D3
11.42
4.01
55.61
24.16
31.25
D4
11.06
3.60
55.32
24.07
31.24
DS
10.73
3.25
54.54
24.17
31.22
D6
10.31
2.85
52.74
23.94
31.18
Raw
39.86
27.57
51.89
23.67
31.47
D1
35.75
23.62
52.34
23.28
31.46
D2
31.44
19.29
53.01
22.90
31.32
PixArt-Sigma-Xi
D3
28.99
16.51
52.54
22.96
31.36
2048 2048
D4
26.18
13.88
51.74
22.95
31.39
D5
23.86
11.66
51.22
22.82
31.34
D6
22.27
10.13
49.34
22.38
31.28
Table 5; Latency, FID and IS under DiT paper experiment setting (250-step IDDPM solver, cfg scale 1.5). Attn Latency means the latency of self-attention. DiT-XL-2 runs with a batch size
Model
Resolution
Config
Latency (s)
Attn Latency
FID
IS
Raw
32.62
11.40
3.16
219.97
D1
31.53
10.21
3,09
218.20
DiT-XL-2 512x512
D2
29.35
8.09
3.10
210.36
D3
27.80
6.56
3.54
196.05
D4
26.96
5.77
4.52
180.34
14

---

A.5
Compression Plan
Fig. I1 12 13 display compression plan obtained after our greedy search method in different model settings as heatmaps. Each block stands for one layer at specific step. Both models exhibit three different kinds of redundancy, but the distribution of these redundancies across time steps and layers is quite different. The results indicate that there is no uniform compression plan for different DiT models and search plan essential in this case.
A.6 Search Time
Table 6 show the plan search time in different configuratio.. Our greedy search method will try method that can achieve high compression ratio so plan search time inversely proportional to threshold.
Table Compression plan search time for three models
Model
Resolution
Config
Plan Search Time
Raw
04m39s
D2
04m08s
DiT-XL-2
512x512
D4
03m49s
D6
03m14s
Raw
22m02s
D2
20m 12s
PixArt-Sigma-XI
1024x
D4
17m50s
D6
15m49s
Raw
1h50m13s
D2
1h46m04s
PixArt-Sigma-XL
2048x2048
D4
1h22m53s
D6
1h23m01s
A.7 Metrics for Compression Plan Search
When designing the compression plan, have considered to use other metrics including LPIPS and SSIM, and finally chose the existing metric mainly because of the speed computation. We tested different SSIM compression schemes and found that when SSIM is chosen the metric, to ensure the quality of the images generated, the threshold should be set at small value of about 1/10 of the existing metric. We checked LPIPS into RGB space then calculate LPIPS on RGB space. We found that LPIPS is very insensitive to value changes in methods and always suggest to use sharing across timestep when threshold is set to 0.005 or smaller. Addistionally, it takes long time compute LPIPS. In this way, believe LPIPS is not suitable metric for compression plan search.
alternative metric by decoding the hidden states
cases. Only small value changes can be observed when switch diffrent
A.8 Negative Conditioning
Negative conditioning is technique that widely used improve generation quality by specifying what to exclude from the generated images. We explored the the impact of negative conditioning on method using general negative prompt like 'Low quality" PixArt-Sigmas In the case, found our method can preserve the effect of negative prompt on the generated images as shown in Fig. 14
15

---

Figure 10: Comparison of video generation using OpenSora V1.1 240p resolution with 16 frames. The left column displays the original video, and the right columns illustrate the outputs from the D1 D6 configuration.
16

---

DIT-XL-2 512x512. Thyeshold
DTT-L-2 512x512. Threshold 0.075
0
Full Aci
Full Altn
ful Atm
WWRS
Tirenter
20-
20
Tmmeso
ASC
ASC
30
30
WARS+AAC
WA-RSHASC
WWRGS-AK
40
20
20
Layer
DIT-XL-2 512x512. Threshold 0.1
DIT-KL-2 512x512. Threshold 0.125
DIT-XL-2 512x512. Threshold
0-
Hull Art
Full Altn
Full Arn
10
WWRS
WARS
WWMRS
20
Timeste
30
30
WA-RS-ACC
WA-SS-ASC
WW-RS-ASC
AST
20
20
20
Layer
Layer
Layer
Figure 11: Compression plan for 512 at different thresholds with DPM solver step set to 50
Pixkrt-Sigma-XL 1024x1024, Threshold 0.025
PixArt Sigma 1024x1124, Threshald 0.05 PixArt-Sigma-XL 1024x1024, Threshold 0.075
10
10-
WA
tirenio
20
20
Tireste
20
ASC
ASC
ASC
WH-S--ACC
WA-S5-ACC
40
AST
Layor
Pixrrt-Sigma-LL Threshold PixArt-Sigmaa 1024x1024, Threshold PisArt-Sigma-Xt
1024x1024, Threshold
Full
Tull Attn
Full Altn
firettp
20
Ttrastpp
20
ASC
ASC
ASC
WARS-ASC
WM-R5+ASC
WA- LASC
Figure 12: Compression plan for solver step set to 50
1024 different thresholds with DPM
17

---

Paxrrt-Sgmma-XL 2048x2048. Threshole 0.025 PixArt-Sigma- 2049x2048, Threshold 0.05 PuxArt-Sigma XL 2049x2048, Threshold 0.075
Ful ALD
Attn
Tull Attn
10
10-
WA-RS
WA-R5
WA
Tircctp
50
30-
30
WA-SS+ASC
W-RSHASC
WA-RHASC
40
40
AST
AST
20
20
Layer
Layer
Layer
PixArt-iggma XL 2048X204B, Thresheld 0.1 P.xrrt-Sigmn--L 2048x2048, Threshold 0.125 PixArt: Sigma XL 2048x204B, Threshold 0.15
Alth
Attn
10-
10
10
Timesoop
WA-RS
WA-R5
Timastee
20
WA-RS
ASC
30
WARSS+ASC
W-RSHASC
40
AST
AST
AST
Layer
Layer
Layer
Figure 13: Compression plan for -ww-rttSigmma-l--24488.. different thresholds with DPM solver step set to 50
Without Negative Prompt
With Negative
Prompt
Without
Negative
Prompt
With Negative
Prompt
D2
D4
D6
Figure 14: images generated by PixArt-Sigma- XL-1024 prompt
- at different thresholds with with/without negative
18

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They are properly stated.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
They are properly stated in Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
They are included in details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Codes included in Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwticccguie//eeeeeeeeeii CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are included in details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: They included in details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:::////uis./eeeeeeeee wwweeurrss.ccpppuuuo
Answer:
[Yes]
Justification: conforms in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Discussed in Appendix A
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Credits for all assets are well given.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing research with human subjects. The work focuses solely on developing a new compression method for diffusion transformer models using existing datasets. Therefore, no IRB approval equivalent review was required or obtained.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "uenq4Z5jOR", "reviewer": "Reviewer_CWF1", "summary": "This paper proposes a combination of novel techniques to reduce the self-attention computation in diffusion transformers (DiTs), without requiring finetuning. This methods leverage locality in attention scores, similarity in timesteps, and similarity with classifier free guidance. Overall, the proposed method is capable of maintaining image quality while significantly reducing self-attention FLOPS.", "questions": "1) You should cite Sora in the introduction and not OpenSora.\n2) While the inclusion of residual caching is novel, how is your windowed attention method different from neighborhood attention (Natten)? This should be cited with local attention.\n3) Including the 1-sigma boundaries (e.g. aggregated over 1k samples) in the MSE plots on Fig 3a would help understand how much variation there is for the attention layers. Additionally, is that DiT-XL or Pixart-Sigma? Identifying the model in the caption would be a good idea.\n4) You explored the impact with CFG but have you looked at the impact with negative conditioning? Does sharing the conditional attention perform similarly or degrade?\n5) There appears to be some interesting behavior in Figure 4, which may be interesting to explore from an explainability perspective. \n6) For your compression plan search, did you consider using other metrics such as LPIPS or SSIM? Would they impact the chosen plan?\n7) The use of 5k image samples for evaluation is unusual. Typically the evaluation is 50k for ImageNet and 30k for COCO, as a larger sample size will reduce statistical noise (FID can be very noisy below 15k samples). \n8) When you evaluated FID, IS, CLIP, did you do so on the same 5k samples used for determining the plan? I would have used the ImageNet/COCO validation set for the plan search and then evaluated on the test set to ensure the two distributions were non-overlapping.\n9) What dataset did you use for your OpenSora evaluation? \n10) Did you consider looking at the impact on VRAM? If there is no impact, stating so in the limitations would be appropriate. \n11) If increasing the step count improves the result in Fig 9, then this suggests that there is an optimal tradeoff between attention FLOPs and step count (iso-sampling FLOPS). Performing that analysis may strengthen adoption of your technique.\n12) Reporting relative performance as percentages is difficult to read. It would be clearer to report them as ratios (e.g. 0.93 vs 93%)\n\nFinally, Figure 5 is unreadable and requires revision. \n- FID and IS should not be plotted on the same axis as they are very different in scale. Use a dual-axis plot if you want to combine them.\n- The points should be connected with a line like in Figure 8 and 9.\n- I would only plot CLIP and FID (not IS) for Pixart, which will make the plot easier to read. \n- Your CLIP scores appear too high, where they should be between 0.2 and 0.35. \n\nPlease provide a revision to this figure or a table of the data plotted therein.", "limitations": "The authors adequately discuss limitations of their approach, although should include a note about VRAM.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposes three novel approaches with compelling empirical justification for each. The methods were then ablated and demonstrate good visual performance while reducing computational overhead during inference. While the authors acknowledge limitations of their kernel implementation, their results may have a strong impact in both downstream deployment and in explainable AI for generative image models.", "weaknesses": "While the visual results are compelling, the paper has limited quantitative evaluations, and may be in a regime with low statistical SNR requiring additional samples. Furthermore, the authors focus on improving FLOPS, however, they do not discuss VRAM usage which would be significant for consumer grade devices and generating longer videos. Finally, there are issues with figure 5 which significantly hinders readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the insightful and constructive comments. \n\n> Q1. You should cite Sora in the introduction and not OpenSora.\n\nThanks. We have added the citation of Sora in the introduction.\n\n> Q2. How is your windowed attention method different from neighborhood attention (Natten)? This should be cited with local attention. \n\nWe adopted 1D window attention, which can be recognized as 1D neighborhood attention, and we have added references to Natten in the article.  The evaluation shows our 1D window attention is more efficient (See global rebuttal Q4).\n\n> Q3. Including the 1-sigma boundaries (e.g. aggregated over 1k samples) in the MSE plots on Fig 3a would help understand how much variation there is for the attention layers. \n\nWe have added the MSE change plot on 1k samples and included the 1-sigma boundaries. We have also added the model type in the figure title and caption. The results indicate that the variation is not significant. Please refer to Figure 4.b in the attached rebuttal PDF.\n\n> Q4. You explored the impact with CFG but have you looked at the impact with negative conditioning? Does sharing the conditional attention perform similarly or degrade?\n\nWe investigated the impact of negative conditioning. The attentions with positive conditioning and negative conditioning have a high degree of similarity, and our method is also effective in reducing computation with negative conditioning. We provided an example in the attached PDF (Figure 4c, \"low quality\" as the negative prompt). We also added a plot to show that similarity across CFG with negative prompt behave similar as the one without negative prompt (Figure 4d). We will add more examples and analysis of negative conditioning in our paper.\n\n> Q5. There appears to be some interesting behavior in Figure 4, which may be interesting to explore from an explainability perspective. \n\nIn Figure 4a, we observe that the middle timesteps exhibit less similarity, which may suggest that the structural transformation occurs primarily in the middle timesteps, while the other timesteps focus on noise removal and refinement. In Figure 4b, we note that the middle layers show more differences between the conditional and unconditional cases, indicating that the middle layers are responsible for the condition-specific processing.\n\n> Q6. For your compression plan search, did you consider using other metrics such as LPIPS or SSIM? Would they impact the chosen plan?\n\nYes, we've considered LPIPS and SSIM for our comparison. LPIPS only works for image comparisons and requires the channel number to be set to 3, whereas we are comparing latent space (with 4 channels). LPIPS also needs to use a network as a feature extractor, which results in a significant inference overhead.\n\nWe also tried the SSIM metric, and the results are included in Table 4 of the global rebuttal. The SSIM metric appears to achieve comparable results to our own metric. However, SSIM is more computationally complex to calculate. Therefore, we have decided to use our own metric for this task. We have included the compression plan in Figure 2 of the rebuttal PDF. Interestingly, the SSIM metric selects more lower-level layers for compression, while the existing metric chooses to compress the higher layers. We will research this in the future.\n\n> Q7. Evaluation sample number\n\nPlease see global rebuttal Q1. Our conclusions remain unchanged at the larger evaluation size. \n\n> Q8. When you evaluated FID, IS, CLIP, did you do so on the same 5k samples used for determining the plan? I would have used the ImageNet/COCO validation set for the plan search and then evaluated on the test set to ensure the two distributions were non-overlapping.\n\nFor the calibration process, we only use class labels (for DiT) and text prompts (for PixArt-Sigma) to generate a small set of images (8 images for DiT, 6 images for PixArt 1K, and 4 for PixArt 2K). No real images are used in the calibration process, so there is no risk of overlapping.\n\n> Q9. What dataset did you use for your OpenSora evaluation?\n\nSince the video generation process is too resource-intensive, and we found the evaluation metrics to be unstable when the sample size is small, we did not apply these metrics to the generated videos. Instead, we included some sample generated examples. These example videos were generated using the default prompts provided by OpenSora.\n\n> Q10. Did you consider looking at the impact on VRAM? If there is no impact, stating so in the limitations would be appropriate.\n\nWhen AST or WA-RS is applied, the hidden states from the previous timestep are stored. The shape of stored hidden states of a layer is [batchsize*2, num_of_head, seqlength, hidden_size]. For example, with the DiT XL/2 512 model, at a batchsize of 1, the VRAM usage of the original model is 11652MB. To store the FP16 hidden states, an additional 2016MB of VRAM is needed, which is 17.3% of the original model's VRAM usage. We will include a VRAM analysis in the limitations section.\n\n> Q11. If increasing the step count improves the result in Fig 9, then this suggests that there is an optimal tradeoff between attention FLOPs and step count (iso-sampling FLOPS). \n\nIn the rebuttal PDF, we have added a FLOPs-IS Pareto front plot (Figure 4.a). Interestingly, we observed that different compression ratios require different optimal settings (steps and DiTFastAttn threshold). We will include this plot and the related discussion in the appendix of the future version of the paper.\n\n> Q12. Reporting relative performance as percentages is difficult to read. It would be clearer to report them as ratios \n\nThanks. We've changed the format of relative performance from percentage to ratio.\n\n> Fig 5 & CLIP issues\n\nFigure 5 is now changed accordingly (check global rebuttal and Figure 3 in the pdf). For CLIP, we use the package torchmetric to calculate it so the scale is different (0-100 vs 0-1). It is now changed to 0-1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "93K5fjBZMn", "reviewer": "Reviewer_e43K", "summary": "The paper  presents a post-training model compression method aimed at reducing the computational complexity of Diffusion Transformers. The authors identify three key redundancies in the attention computation. To address these, they propose three techniques. The proposed methods compress the model FLOPs and enable more efficient deployment in resource-constrained environments. The paper includes extensive experimental results demonstrating significant reductions in computational cost and latency while maintaining generation quality.", "questions": "- How do the authors ensure the generalizability of their method across different datasets and tasks? A more rigorous theoretical analysis could help understand the limitations and potential extensions of the proposed techniques.\n- What is the model complexity, running speed, and model size under different compression settings?\n- Can the authors offer clearer definitions and detailed illustrations for each redundancy type (spatial, temporal, and conditional redundancy)?", "limitations": "The authors have acknowledged several limitations of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The identification and analysis of spatial, temporal, and conditional redundancies in attention mechanisms are thorough and well-founded.\n- The authors conduct extensive experiments on multiple DiT models and various tasks, providing strong evidence of the effectiveness and generalizability of their methods.", "weaknesses": "- Limited Theoretical Analysis: I am in general agreement with the overall concept presented in the paper. However, the paper primarily focuses on empirical results, with limited theoretical analysis to support the proposed methods. A deeper theoretical foundation could help in understanding the generalizability and limitations of the techniques. \n- Complexity  and Scalability: The paper does not extensively discuss the complexity of the compressed models. The figure 1 presents the efficacy under different resolutions. But it does not discuss the model complexity, running speed, and model size in these settings.\n- Insufficient Comparisons: The paper does not compare the related methods, such as flashattention, and KV cache. This mainly weakens the contribution. A more detailed comparison with existing attention acceleration and model compression techniques would be beneficial. \n- Vague Illustration：The authors present three key redundancy strategy. However, the paper lacks sufficient annotations and explanations and does not provide clear definitions for the three scenarios.\n- Application: The paper mainly focuses on attention reduction, but It is not very effective at low resolutions because the attention mechanism accounts for a small proportion of the computational cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the insightful and constructive comments.\n\n> W1 & Q1. How do the authors ensure the generalizability of their method across different datasets and tasks? A more rigorous theoretical analysis could help understand the limitations and potential extensions of the proposed techniques.\n\nAbout generalizability, our experiments demonstrate the effectiveness of DiTFastAttn across multiple datasets and tasks:\n- Image generation: We evaluated on class conditional generation (for DiT) and text conditional generation (for PixArt-Sigma).\n- Video generation: We successfully applied DiTFastAttn to OpenSora for video generation tasks.\n\nOur method shows improved compression ratios and quality preservation as resolution increases from 512x512 to 2048x2048, indicating good scalability.\nThis diversity in datasets, tasks, and model architectures provides strong evidence for the generalizability of our approach. However, we acknowledge that further evaluation on additional domains could further strengthen this claim.\n\nFor theoretical analysis, we thank the reviewer for this valuable suggestion. While our current work focuses on empirical results, we agree that a more rigorous theoretical analysis would be beneficial.\n- We have provided rigorous computational complexity analysis for each proposed technique (WA-RS, AST, ASC) and their combinations.\nSome directions we could explore in future work include:\n- Investigate how the error propagates through the denoising steps when using approximated attention outputs.\n- Quantify the information loss when using window attention compared to full attention.\n\nWe will consider adding a discussion of these theoretical aspects in an extended version of the paper.\n\n\n> W2 & Q2. What is the model complexity, running speed, and model size under different compression settings?\n\nIn Appendix A.4 of our paper, we discussed the complexity of our compression method. Additionally, in the global rebuttal, we have added summary tables (Table 1 and 2) to present the exact FLOPs and latency (running speed). The result shows our method is effective to reduce both the complexity and improve the running speed. The model size will not change.\n\nFor model size and VRAM usage, when AST or WA-RS is applied, the hidden states from the previous timestep are stored. The shape of stored hidden states of a layer is [batchsize*2, num_of_head, seqlength, hidden_size]. For example, with the DiT XL/2 512 model, at a batchsize of 1, the VRAM usage of the original model is 11652MB. To store the FP16 hidden states, an additional 2016MB of VRAM is needed, which is 17.3% of the original model's VRAM usage. We will include a VRAM analysis in the limitations section.\n\n> W3. Insufficient Comparisons: The paper does not compare the related methods, such as flashattention, and KV cache. This mainly weakens the contribution. A more detailed comparison with existing attention acceleration and model compression techniques would be beneficial.\n\nThanks. Our baseline (we denoted as raw setting in our method) has used flashattention in their attention computation, and our method can achieve up to 1.6x speedup compared to it. We'll add a note to the paper to avoid confusion. \n\nKV cache is a method used in large language models for auto-regressive generation. In diffusion model, DiT don't need to attend the tokens from previous timesteps. KV cache isn't applicable to diffusion models like DiT. So we cannot compare our method with KV cache. Note that our method is inspired by caching concepts but applied differently:\n- Caching activation outputs across denoising timesteps and conditional/unconditional branches.\n- WA-RS technique caches residuals to maintain long-range dependencies with efficient local attention.\n\n\n> W4 & Q3. Can the authors offer clearer definitions and detailed illustrations for each redundancy type (spatial, temporal, and conditional redundancy)?\n\nAlthough we have provided some definitions in our paper, we agree that providing more clarity on these concepts will strengthen our paper. Here we denote $X$ as input, $Y$ as output\n\nWindow Attention with Residual Sharing (WA-RS):\n- Definition: The original paper Eq.1 and Eq.2.\n- Illustration: The original paper Figure.3. We will add a heat map visualization of the attention matrix, clearly showing the concentration of attention values along the diagonal.\n\nAttention Sharing across Timesteps (AST):\n- Definition: For step k, $Y_k=W_{o}O_k$ and $Y_{k+1} = \\\\{Y_k$  if  $AST_k = 1$; $W_{o}O_{k+1}$ if $AST_k = 0\\\\}$\n\n- Illustration: The original paper Figure.2 is clear\n﻿\nAttention Sharing across CFG (ASC):\n- Definition: For step k, $Q_{k} = \\\\{W_{Q}X_{k,:c/2}$ if $ASC_k = 1$, $W_{Q}X_{k}$ if $ASC_k = 0\\\\}$ and so as $K_{k}$ and $V_{k}$. The output is $Y_{k} = \\\\{[W_{o}O_k, W_{o}O_k]$ if $ASC_k = 1$; $W_{o}O_{k}$ if $ASC_k = 0\\\\}$\n- Illustration: The original paper Figure.2 is clear\n\n> W5. Application: The paper mainly focuses on attention reduction, but It is not very effective at low resolutions because the attention mechanism accounts for a small proportion of the computational cost.\n\nWhile it's true that our method provides greater benefits at higher resolutions, we believe this is actually a strength rather than a limitation:\n- Even at lower resolutions, our method still provides measurable improvements. For example, on the 512x512 DiT model, we achieve up to 31% FLOP reduction and 10% latency reduction for attention computation.\n- As transformer-based diffusion models continue to scale up in size and target higher resolutions, the relative importance of attention computation increases. Our method is thus well-positioned to provide even greater benefits for future large-scale models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sNb2xQBfaq", "reviewer": "Reviewer_5A6V", "summary": "The authors observe computational redundancies across the three main dimensions of the generation process in DiTs -- space, sampling time, and conditional vs. unconditional forward passes due to CFG. Based on these observations, the authors introduce a set of approaches leveraging them to enable more efficient inference with negligible quality loss for pre-trained DiTs.", "questions": "Questions regarding the main contributions of the paper\n- While their approach to only computing attention locally does seem to be effective, I am surprised by the fact that the authors chose to apply a 1D window attention on the flattened image token sequence instead of one of the many 2D-aware local attention mechanisms such as (Shifting) Window Attention. These should have been the obvious choice in this situation, and the authors fail to motivate their unorthodox choice. While the attention map patterns presented in Fig. 3a do look somewhat like the effective attention pattern given by 1D window attention, I would presume that the additional diagonal lines next to the main diagonal correspond to the same location, but offset by one row. Besides generally being a more intuitive choice, the aforementioned approaches should also correspond to this observed pattern significantly better.\n- Is it possible that the observed CFG redundancy is not an attention-specific aspect but a general one that is already well-described in previous literature? I would be very interested in a comparison with CFG not being applied at all for low noise timesteps, given the large range of prior works that found it beneficial to, substantially anneal the CFG scale for low noise timesteps (MDT, Gao et al., 2023; Analysis of Classifier-Free Guidance Weight Schedulers, Wang et al., 2024; and a wide range of non-published sources such as https://enzokro.dev/blog/posts/2022-11-28-sd-v2-schedules-1/, https://x.com/jeremyphoward/status/1584771100378288129, https://github.com/mcmonkeyprojects/sd-dynamic-thresholding), or even completely deactivate it (Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models, Karras et al., 2024).\n- What is the distribution of which acceleration method is used how often, and where in the generation process?\n\nAdditional small questions\n- Can the authors provide an explanation regarding the intuition of FID improving when reducing the attention FLOPS in Fig. 5?\n- Subjectively, more compression results in a loss of contrast, both for generated images (albeit slightly), as well as videos. Do the authors have an explanation as to what might be causing this?\n- For their similarity analyses in Fig. 4, the authors only analyze the cosine similarity of attention outputs. Are the magnitudes similar as well? Wouldn't this be a requirement to enable successful caching?\n\nPresentation suggestions\n- Fig. 5 is hard to parse due to the multitude and different scales of the presented metrics. I'd suggest that the authors separate the metrics into separate stacked graphs, which should also render these graphs more accessible for people with color perception impairments. I would also suggest adding arrows to indicate whether lower or higher is better for each individual metric. Similar improvements to other figures could also help make the paper more accessible.\n- For a camera-ready version, I think the paper would benefit from additional uncurated qualitative examples", "limitations": "The authors have adequately addressed the limitations and societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses an important avenue of research, as diffusion models are inherently very inefficient (as compared to, e.g., GANs) for inference, due to having to evaluate large models for a multitude of steps. While previous works such as DeepCache (Ma et al., 2023) or Cache Me If You Can (Wimbauer et al., 2023) have already investigated this problem in general, they do so specifically for diffusion U-Nets and many of these methods are not directly applicable to DiTs that do not share the hierarchical nature of U-Nets. Given the recent rise in popularity of DiT-based diffusion backbones for successful large-scale diffusion models (PixArt-alpha, Sora, Stable Diffusion 3, ...), providing methods that can successfully speed up the inference of these models is important. The authors present a method that is able to substantially reduce the cost incurred from evaluating attention, which makes up a large fraction of the FLOPS in high-resolution settings, in these diffusion transformers during inference while retaining most of the quality of the generated images, addressing this problem well.\nThe authors investigate and identify a set of redundancies during the generation process in diffusion transformers that relate to evaluating the attention mechanism. These findings are then used to motivate the presented methods.\n- Going beyond naive approaches to exploit the presented redundancies, the authors introduce further, non-obvious tricks such as residual sharing to make them work well.\n- The main method is thoroughly evaluated on two image diffusion models where it shows promising results.", "weaknesses": "- While the method as a whole is evaluated reasonably well, the evaluation of the effect of parts of the method & design choices seems lacking: there are missing ablations regarding different variants (e.g., naive approaches) of leveraging specific redundancies, and especially the evaluation of the compression plan algorithm is lacking and its hyperparameters (threshold $\\delta$, number of samples).\n- Open questions remain, especially regarding unorthodox design choices and whether the CFG redundancy is actually an attention redundancy as described or a general redundancy already thoroughly described in previous works. See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the insightful and constructive comments.\n\n> W1.  the evaluation of the effect of parts of the method & design choices seems lacking: there are missing ablations regarding different variants (e.g., naive approaches) of leveraging specific redundancies, and especially the evaluation of the compression plan algorithm is lacking and its hyperparameters (threshold $\\delta$, number of samples).\n\nWe show the effect of some naive approaches in Figure 9 in our paper, and we have added more images generated  (like Figure 5a in the attached pdf) and compression plan (like Figure 5c in the attached pdf) by naive approaches. We conducted an ablation study on number of samples, and we found that adding number of samples had little effect on the compression plan.\n\nDiT n_samples ablation\n\n| num of samples | Threshold | IS          | FID          |\n|---------|-----------|-------------|----------|\n| 8       | 0.05      | 206.43    | 29.97        |\n| 8       | 0.1       | 200.30    | 26.91        |\n| 8       | 0.15      | 179.13    | 23.11        |\n| 16      | 0.05      | 205.76    | 30.33        |\n| 16      | 0.1       | 197.16    | 27.62        |\n| 16      | 0.15      | 180.68    | 23.24         |\n| 32      | 0.05      | 205.63     | 30.35        |\n| 32      | 0.1       | 202.73    | 26.97         |\n| 32      | 0.15      | 180.22    | 24.55        |\n\nIn global rebuttal Q5, we added more ablation studies.\n\n> W2. & Q2. Open questions remain, especially regarding unorthodox design choices and whether the CFG redundancy is actually an attention redundancy as described or a general redundancy already thoroughly described in previous works. \n\nThanks for your comments. We believe that the issue of conditional guidance (CFG) redundancy is not limited to the attention mechanism alone. There has been some prior work exploring methods to reduce CFG redundancy. However, our approach differs from previous efforts, as we have examined the use of CFG sharing at a finer granularity. Specifically, we have assessed the influence of conditional and unconditional computations across different layers and different timesteps.\n\nOur work applies CFG sharing selectively, only on certain layers and time steps, in order to avoid significantly degrading the model's performance. In this way, our work acknowledges and builds upon existing research, while offering a novel perspective on addressing redundancy within the unique framework of diffusion models.\nGoing forward, we plan to extend our method to other model components in the future.\n\n> Q1. While their approach to only computing attention locally does seem to be effective, I am surprised by the fact that the authors chose to apply a 1D window attention on the flattened image token sequence instead of one of the many 2D-aware local attention mechanisms such as (Shifting) Window Attention. These should have been the obvious choice in this situation, and the authors fail to motivate their unorthodox choice. \n\nPlease see global rebuttal Q4. \n\n> Q3. What is the distribution of which acceleration method is used how often, and where in the generation process?\n\nPlease check Q2 in global rebuttal and Figure 1 in the rebuttal pdf attached.\n\n> Q4. Can the authors provide an explanation regarding the intuition of FID improving when reducing the attention FLOPS in Fig. 5?\n\nWe have thoroughly reviewed the evaluation code and confirmed that there are no issues on our end. We believe the discrepancy may be due to the inaccuracy of the FID metric, as indicated in previous research [1].\nThe FID does not fully capture the visual quality of the generated images. However, changes in the FID score can still be meaningful, as they reflect shifts in the distribution of the generated images. \n\n[1] Jayasumana S, Ramalingam S, Veit A, et al. Rethinking fid: Towards a better evaluation metric for image generation[C]// CVPR. 2024: 9307-9315.\n\n> Q5. Subjectively, more compression results in a loss of contrast, both for generated images (albeit slightly), as well as videos. Do the authors have an explanation as to what might be causing this?\n\nWe evaluated the images generated using different methods and found that the high threshold of ASC was likely the cause. We included an example generated by different methods and at varying ASC thresholds in the rebuttal PDF (Figure 5). We suspect that excessive CFG sharing can lead to a loss of detail and less pronounced edges in the generated images. If the trade-off between contract quality and ASC is a significant concern, the user should consider disabling the ASC method.\n\n> Q6. For their similarity analyses in Fig. 4, the authors only analyze the cosine similarity of attention outputs. Are the magnitudes similar as well? Wouldn't this be a requirement to enable successful caching?\n\nThanks. We plot the magnitude of attention output, the plot can be found in the rebuttal pdf (Figure 6). It is similar to cosine similarity.\n\n> Q7. Fig. 5 is hard to parse due to the multitude and different scales of the presented metrics. I'd suggest that the authors separate the metrics into separate stacked graphs, which should also render these graphs more accessible for people with color perception impairments. I would also suggest adding arrows to indicate whether lower or higher is better for each individual metric. Similar improvements to other figures could also help make the paper more accessible.\n\nThanks for your advice. Please see Q3 in global rebuttal.\n\n> Q8. For a camera-ready version, I think the paper would benefit from additional uncurated qualitative examples\n\nThanks. We plan to include more uncurated qualitative examples, ablation studies, plots, and tables in the future version of our paper, including the results presented in the rebuttal PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6Ne916h4s", "reviewer": "Reviewer_8Fw6", "summary": "In this paper, the authors introduce a novel post-training model compression method aimed at accelerating Diffusion Transformers (DiT) used for image and video generation tasks. They identify the spatial, temporal, and conditional redundancy in attention blocks and propose the corresponding method to tackle them.", "questions": "See weakness", "limitations": "See weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed methods are interesting and effective.\n+The writing is easy to understand.", "weaknesses": "- The authors sample 5K images to evaluate the generation quality while the standard setting is sampling 50K images (following original DiT).\n- What is the adopted compression strategy in each layer and each timestep? It would be better to illustrate the searched compression plan in a figure.\n- Can the proposed method be combined with other acceleration methods \\eg Deepcache?\n- Why do the authors only demonstrate the Inception Score in Figure 9? FID is the most common metric in image generation.\n- What are the FLOPs and Latency in Table 1 exactly? The authors should demonstrate the value rather than only the fraction since there is still enough space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the insightful and constructive comments. \n\n> W1. The authors sample 5K images to evaluate the generation quality while the standard setting is sampling 50K images (following original DiT).\n\nThanks. Following your suggestion, we have increased the evaluation size of ImageNet to 50K and COCO to 30K. Tables can be found in the global rebuttal (Table 1 & 2). More discussions can be found in the global rebuttal Q1. \n\n> W2. What is the adopted compression strategy in each layer and each timestep? It would be better to illustrate the searched compression plan in a figure.\n\nThanks for your suggestion. We generated figures for adopted compression strategy in different threshold settings, check Q2 in global rebuttal and Figure 1 in the rebuttal PDF. \n\n> W3. Can the proposed method be combined with other acceleration methods \\eg Deepcache?\n\nWe believed that Deepcache benefits from the hierarchical nature of U-Nets so it can not be directly applied on DiT model. However, we do think our method can be combined with other acceleration methods. For example, our method is orthogonal with quantization method, so we can apply both quantization and our method to the model to achieve further acceleration.\n\n> W4. Why do the authors only demonstrate the Inception Score in Figure 9? FID is the most common metric in image generation.\n\nShowing only one metric in a figure tends to be more clear. In a future version of our paper, we will include an ablation study plot to further demonstrate the FID results.\n\n> W5. What are the FLOPs and Latency in Table 1 exactly? The authors should demonstrate the value rather than only the fraction since there is still enough space.\n\nThanks. We make a summary table in the global rebuttal Q1 (Table 1 & 2) to contain exactly FLOPS and Latency. Summary tables like that will be included in the future version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks all the reviewers for the time and effort taken to provide valuable insights and comments on our work. Here we provide experiment results and answers for some common questions and comments:

> 1. The use of 5k image samples for evaluation is not enough.

We have increased the evaluation size of ImageNet to 50K samples and COCO to 30K samples. The table below showing the results of DiT (Table 1) and PixArt-Sigma 1K (Table 2). Plots of the ImageNet@50K and COCO@30K results can be found in the rebuttal PDF (Figure 3). The results are more stable and demonstrate the same trend of change as the previous 5K results. These updated results will be included in the future version of the paper.

#### Table1. DiT ImageNet 50K Results

| Threshold | MACs (GFLOPs) | Attn_MAC (GFLOPs) | Latency | Attn_Latency | IS | FID |
|-----------|------:|----------:|---------:|--------:|---:|----:|
| 0         | 20989 | 2706      | 2.841s    | 0.890s  | 400.64 | 23.99 |
| 0.025     | 20623 | 2383      | 2.863s    | 0.918s  | 402.24 | 23.86 |
| 0.05      | 20105 | 2088      | 2.854s    | 0.914s   | 400.28 | 22.95 |
| 0.075     | 19560 | 1832      | 2.892s    | 0.909s   | 401.37 | 21.75 |
| 0.1       | 19032 | 1598      | 2.782s    | 0.828s   | 385.48 | 19.74 |
| 0.125     | 18432 | 1361      | 2.769s    | 0.799s   | 330.35 | 18.09 |
| 0.15      | 17796 | 1209      | 2.658s    | 0.710s   | 328.21 | 15.10 |

#### Table2. PixArt 1K COCO 30K Results

| Threshold | MACs(GFLOPs)  | Attn_MAC (GFLOPs)| Latency| Attn_Latency | IS | FID | CLIP    |
|----|----:|----:|-----:|----:|---:|---:|-----:|
| 0         | 132693    | 46464    | 2.437s    | 0.937s  | 24.25| 55.70| 0.31377 |
| 0.025     | 129478    | 43662    | 2.459s    | 0.958s  | 24.28| 55.67| 0.31378 |
| 0.05      | 123701    | 38385    | 2.508s    | 0.993s  | 24.23| 55.58| 0.31371 |
| 0.075     | 117661    | 33041    | 2.472s    | 0.962s  | 24.18| 55.28| 0.31365 |
| 0.1       | 113351    | 29427    | 2.417s    | 0.922s  | 23.98| 55.11| 0.31342 |
| 0.125     | 108101    | 25330    | 2.372s    | 0.868s  | 23.74| 53.90| 0.31342 |
| 0.15      | 103403    | 22131    | 2.300s    | 0.798s  | 23.40| 51.68| 0.31314 |

> 2. What is the adopted compression strategy?

We generated plots for the adopted compression strategies under different threshold settings. Please refer to Figure 1 in the rebuttal PDF. The results show that all the strategies (WA-RS, ASC, WA-RS+ASC, and AST) are well-distributed, demonstrating that our strategy search algorithm is effective. An interesting observation is that the different strategies tend to be concentrated in specific layers and timesteps.

> 3. Figure 5 is unreadable.

  We have optimized the display of Figure 5. We show the result of each metrics seperately. Arrows are added to indicate whether lower or higher is better for each individual metric. We show 4 plots in the rebuttal pdf (Figure 3) as an example due to limited spaces. 

> 4. What kind of local attention is used? Why use 1d local attention instead of 2d local attention?

We use sliding window attention for WA-RS, which is the same as the sliding window attention used in Longformer and the 1D neighborhood attention. Our method can also be generalized to 2D local attention. However, we observe that 2D local attention is not efficient enough. This is because the GPU memory access of the K and V tensors is not sequential, resulting in extra data gathering overhead.
We evaluate the inference latency of adopting Natten 2D attention and our sliding window attention. We use a kernel size of 127 for the sliding window, and a kernel size of 5 for the Natten 2D attention. The latency results are shown in Table 3 below. Based on these results, we chose to use 1D local attention.

#### Table3. Latency of different kernel

| Kernel          | Attention Latency |
|-----------------|--------------------|
| Window Attention   | 0.828s              |
| Natten 2D Attention       | 0.926s              |

> 5. More ablations and examples are needed.

In the rebuttal PDF, 
   1. we conducted additional ablation studies on different metrics used in the search (Figure 2). We tested different SSIM compression schemes and found that when SSIM is chosen as the metric, to ensure the quality of the images generated, the threshold should be set at a small value of about 1/10 of the existing metric. 
  2. We investigated different  the impact of the negative prompt (Figure 4c) and found that negative prompt remain effective with our method. 
  3. We checked the effect of CFG sharing on contrast (Figure 5). Loss of contrast can happen when using a large threshold for ASC. 
  4. We provided examples of the Pareto front of number of steps vs. TFLOPS (Figure 4a), which can served as guidance for user to choose appropriate compression setting. 
  5. We provided the MSE plot of 1K samples with 1-sigma in DiT (Figure 4b) to show that window part have a higher MSE and larger variation across samples. 
  6. We plotted the Magnitude of Attention Outputs Across Step and CFG Dimensions in DiT (Figure 6) to show magnitude of attention output show a similar result with cosine similarity.

#### Table4. SSIM vs Current Metric

| Threshold | Metric | FID@5K    | IS@5K   | MACs (GFLOPs)| Attn_MAC (GFLOPs)| Latency | Attn_Latency |
|---|----|------|------|----|------|------|-----|
| 0.005     | SSIM   | 28.77 | 201.92 | 19324 | 1770    | 2.851s   | 0.862s   |
| 0.01      | SSIM   | 26.71 | 200.92 | 18611 | 1492    | 2.794s   | 0.825s   |
| 0.015     | SSIM   | 25.44 | 192.42 | 18042 | 1286    | 2.764s   | 0.774s   |
| 0.05      | Current   | 30.33 | 205.76 | 20102 | 2064    | 2.957s   | 0.950s   |
| 0.1       | Current   | 27.62 | 197.16 | 18976 | 1579    | 2.834s   | 0.851s   |
| 0.15      | Current   | 23.24  | 180.68 | 18028 | 1229    | 2.713s   | 0.756s   |

Meta-review: The paper proposed a combination of techniques to reduce the self-attention computation in diffusion transformers (DiTs), without requiring finetuning. The proposed method leverages locality in attention scores, similarity in timesteps, and similarity with classifier free guidance. Overall, the proposed method is capable of maintaining image quality while significantly reducing self-attention FLOPS. After rebuttal and discussions, the recommendations from the reviewers were 3XBorderline Accept and 1XAccept. The reviewers were consistent in the paper's contributions while they also made lots of comments in improveing the paper.
Decision: Accept (poster)