DiTFastAttn: Attention Compression for Diffusion Transformer Models

Authors: zhihang  yuan
hanling  zhang
pu  lu
xuefei  ning
linfeng  zhang
tianchen  zhao
shengen  yan
guohao  dai
yu  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:24:25

Abstract: Diffusion Transformers (DiT) excel at image and video generation but face computational challenges due to the quadratic complexity of self-attention operators. We propose DiTFastAttn, a post-training compression method to alleviate the computational bottleneck of DiT.
We identify three key redundancies in the attention computation during DiT inference: (1) spatial redundancy, where many attention heads focus on local information; (2) temporal redundancy, with high similarity between the attention outputs of neighboring steps; (3) conditional redundancy, where conditional and unconditional inferences exhibit significant similarity. We propose three techniques to reduce these redundancies: (1) $\textit{Window Attention with Residual Sharing}$ to reduce spatial redundancy; (2) $\textit{Attention Sharing across Timesteps}$ to exploit the similarity between steps; (3) $\textit{Attention Sharing across CFG}$ to skip redundant computations during conditional generation.

Main paper: 
 introduction: 
 Recently, diffusion transformers (DiT) have gained increasing popularity in image (Peebles & Xie, 2023; Chen et al., 2024) and video generation (Brooks et al., 2024) . However, a major challenge with DiTs is their substantial computational demand, particularly noticeable when generating highresolution content. On the one hand, traditional transformer architectures, with their self-attention mechanism, have an O(L 2 ) complexity to the input token length L. This quadratic complexity leads to significant computational costs as the resolution of images and videos escalates. As demonstrated in Fig. 1 , the attention computation becomes the primary computational bottleneck during the inference process as image resolution increases. Specifically, if a 2K × 2K image is tokenized into 16k tokens (Chen et al., 2024) , this requires several seconds for attention computation, even on high-end GPUs such as the Nvidia A100. On the other hand, the inference process of diffusion requires a substantial number of neural network inferences due to the multiple denoising steps and the classifier-free guidance (CFG) technique (Ho & Salimans, 2022) .Previous efforts to accelerate attention mechanisms, such as Local Attention, Swin Transformer (Liu et al., 2021) , and Group Query Attention (GQA) (Ainslie et al., 2023) , mainly focused on designing the attention mechanism or network architecture. While effective in reducing computational costs, these approaches necessitate large retraining costs. Due to the substantial data and computational requirements for training a DiT, there is a need for post-training compression methods.In this work, we identify three types of redundancy in the attention computation of DiT inference and propose a post-training model compression method, DiTFastAttn, to address these redundancies: (1) Redundancy in the spatial dimension. Many attention heads primarily capture local spatial information, with attention values for distant tokens nearing zero. To reduce the redundancy, we opt to use window attention instead of full attention for certain layers. However, directly discarding all attention computation outside the window leads to significant performance degradation. To maintain the performance in a training-free way, we propose to cache the residual between the outputs of full and window attention at one step and reuse this residual for several subsequent steps. We refer to this technique as Window Attention with Residual Sharing (WA-RS).(2) Similarity between the neighboring steps in attention outputs. The attention outputs of the same attention head across neighboring steps can be highly similar. We propose the Attention Sharing across Timesteps (AST) technique that exploits this step-wise similarity to accelerate attention computation.(3) Similarity between the conditional and unconditional inference in attention outputs. We observe that in CFG, the attention outputs of conditional and unconditional inference exhibit significant similarity (SSIM ≥ 0.95) for certain heads and timesteps. Therefore, we propose the Attention Sharing across CFG (ASC) technique to skip redundant computation during unconditional generation.We conduct extensive experiments to evaluate DiTFastAttn using multiple DiT models, including DiT-XL (Peebles & Xie, 2023) and PixArt-Sigma (Chen et al., 2024) for image generation, and Open-Sora (Open-Sora, 2024) for video generation. Our findings demonstrate that DiTFastAttn consistently reduces the computational cost. Notably, the higher the resolution, the greater the savings in computation and latency. For instance, with PixArt-Sigma, DiTFastAttn delivers a 20% to 76% reduction in attention computation and a end-to-end speedup of up to 1.8× during the generation of 2048×2048 images. 
diffusion models: 
 Diffusion models (Ho et al., 2020; Rombach et al., 2022; Peebles & Xie, 2023; Chen et al., 2024; Brooks et al., 2024) have gained significant attention due to their superior generative performance compared to GANs (Creswell et al., 2018) . Early diffusion models (Ho et al., 2020; Rombach et al., 2022) are implemented based on the U-Net architecture. To achieve better scalability, DiT (Peebles & Xie, 2023) utilizes the transformer architecture instead of U-Net. Diffusion transformer is applied in the fields of image and video generation. PixArt-Sigma (Chen et al., 2024) demonstrates the diffusion transformer's capability to generate high-resolution images up to 4K. Sora (Brooks et al., 2024) presents the diffusion transformer's ability to generate videos.The computational overhead of attention has garnered significant attention. FlashAttention (Dao, 2023) divides the input tokens into smaller tiles to minimize redundant memory accesses and optimize latency. Some studies highlight the quadratic complexity of attention computation and improve efficiency through token pruning, achieved by filtering (Rao et al., 2021; Liu et al., 2022; Wu et al., 2023) or merging (Lu et al., 2023; Huang et al., 2023; Wu et al., 2023) tokens at different stages of the network. DynamicViT (Rao et al., 2021) employs a prediction network to dynamically filter tokens. Adaptive Sparse ViT (Liu et al., 2022) filters tokens by simultaneously considering the attention values and the L2 norm of the features. Lu et al. (2023) trains a network with segmentation labels to direct the merging operations of tokens in regions with similar content. Huang et al. (2023) conducts attention computations after downsampling tokens and subsequently upsampling to recover the spatial resolution. Wu et al. (2023) demonstrates that deeper layers are more suitable for filtering tokens, whereas shallower layers are more appropriate for merging tokens.Various studies have delved into the utilization of local attention patterns, where each token attends to a set of neighboring tokens within a fixed window size, aiming to mitigate the computational burden associated with processing long sequences. The concept of local windowed attention was initially introduced by Beltagy et al. (2020) in Longformer, presenting an attention mechanism that scales linearly with sequence length. Bigbird (Zaheer et al., 2020) extends this idea by incorporating window attention, random attention, and global attention mechanisms, enabling the retention of long-range dependencies while mitigating computational costs. In the realm of computer vision, Swin Transformer (Liu et al., 2021) adopts a similar approach by confining attention computation to nonoverlapping local windows, utilizing shifted windows across different layers to capture global context efficiently. Twins Transformer (Chu et al., 2021) , FasterViT(Vasu et al., 2023) , and Neighborhood attention transformer (Hassani et al., 2023) employ window-based attention to enhance computational efficiency, leveraging different module designs such as global sub-sampled attention and hierarchical attention to exploit global context effectively. In our work, we employ fixed-sized window attention to accelerate pretrained Diffusion Transformer models and introduce a novel technique named Window Attention with Residual Sharing to preserve long-range dependencies for image tokens.GQA (Ainslie et al., 2023) divides query heads into G groups. Each query retains its own parameters, while each group shares a key and value, reducing memory usage and improving efficiency. PSVIT (Chen et al., 2021) shows that attention maps between different layers in ViT have significant similarity and suggests sharing attention maps across layers to reduce redundant computation. Deepcache (Ma et al., 2023) demonstrates that high-level features in U-Net framework diffusion models are similar across timesteps. Deepcache proposes reusing U-Net's high-level features and skipping intermediate layers' computation to accelerate denoising process. TGATE (Zhang et al., 2024) shows that the cross-attention output of text-conditional diffusion models converges to a fixed point after several denoising timesteps. TGATE caches this output once it converges and keeps it fixed during the remaining denoising steps to reduce computational cost. In DiTFastAttn, we demonstrate the similarity of attention outputs both CFG-wise and step-wise. We also consider the differences in similarity across different layers at various steps to share attention outputs CFG-wise and step-wise.Network quantization is a widely used technique for reducing the bitwidth of weights and activations, effectively compressing both image generation models (Shang et al., 2023; Zhao et al., 2024b) and video generation models (Zhao et al., 2024a) . Scheduler optimization is another popular approach aimed at decreasing the number of timesteps in the denoising process (Song et al., 2020; Lu et al., 2022; Liu et al., 2023a) . Additionally, distillation serves as an effective method for minimizing the timesteps required during denoising (Salimans & Ho, 2022; Meng et al., 2023; Liu et al., 2023b) . DiTFastAttn offers a complementary solution, as it operates independently of the specific quantization bitwidth, scheduler, and timesteps employed. Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: They are properly stated in Conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 In this section, we demonstrate the redundancy in the inference process of diffusion models with transformers. In the denoising process, we identify three types of redundancy, as shown in Fig. 2: (1) Redundancy in the spatial dimension. (2) Similarity between the neighboring steps in attention outputs.(3) Similarity between the conditional and unconditional inference in attention outputs. To address these redundancies, we propose three compression techniques, as shown in Fig. 2: (1) In Sec. 3.2, we introduce Window Attention with Residual Sharing to reduce spatial redundancy.(2) In Sec. 3.3, we introduce Attention Sharing across Timesteps to exploit step-wise similarities, thereby enhancing model efficiency. (3) In Sec. 3.4, we introduce Attention Sharing across CFG to reduce redundancy by utilizing similarity between conditional and unconditional generation. In Sec. 3.5, we introduce a simple greedy method to decide the compression plan, i.e., select the appropriate compression technique for each layer and step.We can observe the spatial locality of attention in many transformer layers in pre-trained DiTs. As shown in Fig. 3(a) , attention values concentrate within a window along the diagonal region of the attention matrix. Therefore, replacing full attention with fixed-size window attention for some layers can preserve most of the values in the attention matrix during inference. By computing attention values only within a specified window, the computation cost of attention can be largely reduced.However, some tokens still attend to a small set of spatial distant tokens. Discarding these dependencies negatively affects model performance. Mitigating this issue using only window attention necessitates a large window size to capture these dependencies. Consequently, this approach achieves minimal reduction in computational cost, thereby hindering acceleration efforts.Cache and Reuse the Residual for Window Attention. To address the aforementioned issue, we investigate the information loss caused by using window attention. As shown in Fig. 3 (a), the residual between the outputs of full and window attention exhibits a small variation across steps, unlike the output of window attention. This observation motivates us to cache the residual of window attention and full attention in one step and reuse it in subsequent steps. Fig. 3(b) illustrates the computation of WA-RS: at each step, for each window attention layer, we compute the window attention and add a residual cached from the previous step to the output. We denote the set of steps that share the residual value R r as K, the full attention at step r as O r , the window attention at step k as W k . For the first step in the set r = min (K), the computation of WA-RS goes as follows:)× 𝑉 ! = O ! " So3max( )× 𝑉 !#$ = 𝑊 !#$ " So3max( ) × 𝑉 ! = 𝑊 ! " O !#$ " = 𝑊 !#$ " + 𝑅 ! step t+2 So3max( )× 𝑉 !#% = 𝑊 !#% " 𝑅 ! = O ! " -𝑊 ! " O ! " O !#% " = 𝑊 !#% " + 𝑅 !O r = Attention(Q r , K r , V r ), W r = WindowAttention(Q r , K r , V r ), R r = O r -W r .(1)For a subsequent step in the set k ∈ K, the computation of WA-RS goes as follows:W k = WindowAttention(Q k , K k , V k ), O k = W k + R r .(2)The sequential nature of the denoising process in diffusion models is a major bottleneck for inference speed. Here, we compare the attention outputs at different steps during the denoising process. We find that for some layers, the attention outputs at certain steps show significant similarity to those of adjacent steps. Fig. 4 (a) presents the cosine similarity between the attention outputs at different steps.We can draw two primary observations: (1) There is a noticeable temporal similarity between the attention outputs;(2) This similarity varies across steps and layers.To exploit this similarity to reduce the computational cost, we propose the AST technique. Specifically, for a set of steps with their attention outputs similar to each other, we cache the earliest step's attention output O and reuse it, thereby skipping the computation at the subsequent steps.Classifier-free guidance (CFG) is widely used for conditional generation (Ho & Salimans, 2022; Ramesh et al., 2022; Saharia et al., 2022) . In each step of the inference process for conditional Based on this observation, we propose the ASC technique that reuses the attention output from the conditional neural network evaluation in the unconditional neural network evaluation.The aforementioned techniques, including WA-RS, AST, and ASC, can effectively cut down the computational cost while maintaining the performance. As shown in Fig. 3 and Fig. 4 , different layers have different redundancies in different time steps. Therefore, it is crucial to properly decide the compression plan, i.e., which techniques should be applied for each layer at each step.We develop a simple greedy method to select the appropriate strategy (a combination of techniques) from a strategy list S =[AST, WA-RS + ASC, WA-RS, ASC] for each step and each layer. As shown in Alg. 1, we determine the strategies step by step and layer by layer. For each step and transformer layer, we apply each of the four compression strategies and calculate the loss between the model outputs with and without compression for the current step, L(O, O ′ ). Then, we select the strategy with the highest computation reduction ratio with loss below a threshold i |M | δ, where i is the layer index and |M | is the number of layers in the model. If none of the four strategies meet the threshold, we do not apply compression for this layer at that step. For calculating quality metrics, we use ImageNet as the evaluation dataset for DiT and MS-COCO as the evaluation dataset for PixArt-Sigma. MS-COCO 2014 caption is used as text prompt for Pixart-Sigma models' image generation. To evaluate generation quality, we generate 50k images for DiT models and 30k images for PixArt-Sigma models. Following previous studies, we employ FID (Heusel et al., 2017) , IS (Salimans et al., 2016) and CLIP score (Hessel et al., 2021) as the evaluation metrics. We measure the latency per sample on a single Nvidia A100 GPU.We use mean relative absolute error for L(O, O ′ ) and experiment with and different thresholds δ at intervals of 0.025. We denote these threshold settings as D1 (δ=0.025), D2 (δ=0.05), ..., D6 (δ=0.15), respectively. We set the window size of WA-RS to 1/8 of the token size.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
results on image generation: 
 Results of Evaluation Metrics and Attention FLOPs. DiTFastAttn is applied to the pre-trained DiT-XL-2-512, PixArt-Sigma-1024, and PixArt-Sigma-2K models. Table 1 displays the evaluation results of these models. For the DiT-XL-2-512 and PixArt-Sigma-1024 models, configurations D1, D2, and D3 nearly match the performance of the original models in terms of IS and FID metrics. Comparison of compression effects and evaluation metrics between the three models reveals that as image resolution increases, DiTFastAttn not only achieves greater compression but also better preserves the generative performance of the models.Compression Plan after Search. Fig. 5 illustrates the compression plan under the D6 setting. For the DiT model, AST and ASC are utilized in the early timesteps, with full attention primarily appearing in the initial attention layers. In contrast, the PixArt-Sigma model employs AST sporadically in the first two layers and in the middle attention layers during the intermediate timesteps, while the combination of WA-RS and ASC is notably predominant in the final timesteps. This variability in the distribution of different types of redundancy across models highlights the absence of a universal compression strategy, underscoring the necessity for tailored plan searches. Additional compression plans for other settings are provided in A.5.Visualization of DiTFastAttn's Generation Results. Fig. 6 presents image generation samples from DiTFastAttn. The D1, D2, and D3 configurations of the DiT-XL-2-512 and PixArt-Sigma-1024 models demonstrate visual generation quality comparable to the original models. In contrast, D4, D5, and D6 achieve greater compression, exhibiting slight variations in detail while still producing acceptable-quality images. The PixArt-Sigma-2K model maintains image quality similar to the original up to D4, with D5 and D6 also generating high-quality outputs. This suggests that our compression method effectively preserves generation quality, even when reducing attention computation by over 50% and compressing to 33% at higher resolutions. We apply DitFastAttn on OpenSora for video generation with thresholds from 0.01 to 0.06. The results are shown in Fig. 7 . Specifically, the reduction in attention computation for these configurations are as follows: 11.68%, 27.63%, 40.48%, 48.98%, 50.75%, 55.76% respectively. Latency at raw setting is 35.79 seconds where as the performance with the aforementioned configurations were as follows: 34.79 seconds, 33.39 seconds, 31.74 seconds, 31.29 seconds, 31.29 seconds, and 31.17 seconds, achieving 1.02× to 1.15× end-to-end speedup. For an extended analysis, additional results are provided in A.3. Overall Latency of DiTFastAttn. Fig. 8 shows the latency for image generation and attention as computation decreases when DiTFastAttn is applied. DiTFastAttn achieves end-to-end latency reduction for all three model at all compression ratio settings. PixArt-Sigma-2K shows the best performance, with overall generation latency at D6 being 56% of raw and overall attention latency at 37%. The result indicates that as resolution increases, DiTFastAttn achieves better performance in reducing latency for both overall attention and image generation.DiTFastAttn Outperforms Single Methods. As shown on the left of Fig. 9 , DiTFastAttn maintains higher quality metrics compared to individual techniques with the same computation budget. Among single techniques, AST shows the best generation quality. However, beyond 2.2 FLOPs, further compression using AST significantly degrades the outputs, causing the search algorithm to terminate. DiTFastAttn supports further compression while maintaining better quality.Higher Steps Improve DiTFastAttn's Performance. As shown on the middle of Fig. 9 , we compared the performance of DiTFastAttn at different steps. It is evident that as the step increases, DiTFastAttn can compress more computation while maintaining quality. DiT runs with a batch size of 8, while PixArt-Sigma models with a batch size of 1. The blue line delineates the latency for end-to-end image generation, whereas the orange line represents the latency of multi-head attention module.1 than Window Attention at the same compression ratio. Without residuals, window attention results in a significant performance drop. 
conclusion: 
 In this paper, we introduce a novel post-training compression method, DiTFastAttention, to accelerate diffusion models. We identify three types of redundancy : (1) Redundancy in the spatial dimension. Limitations. First, our method is a post-training compression technique and therefore cannot take advantage of training to avoid the performance drop. Second, our method mainly focuses on inference acceleration instead of VRAM reduction. When AST is applied, the attention hidden states from previous timestep will be stored and will bring extra VRAM usage. Third, our simple compression method may not find the optimal compression plan. Fourth, our method only reduces the cost of attention module. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: Discussed in Appendix A.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Credits for all assets are well given. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: DiTFastAttention can enable more efficient deployment of diffusion transformer models, which have shown remarkable image and video generation capabilities. On the positive side, accelerating these models could democratize access to powerful generative AI by reducing computational requirements, and allowing broader adoption for creative and educational applications. However, there are also potential negative societal impacts that must be carefully considered. Highly realistic synthetic media could be exploited to create deepfakes for misinformation, fraud, or non-consensual editing. There are also potential privacy risks if generative models can reconstruct personal information from data. While DiTFastAttention does not inherently increase or reduce these risks compared to the original models, widening access makes misuse by malicious actors more likely. Therefore, safeguards against misuse and ethical guidelines for the responsible release of compressed models may be needed as this technology develops.A The mean relative absolute error is a metric used to evaluate the performance of a model by measuring the average relative deviation between the outputs O ′ and the raw outputs O. It is calculated as follows:L(O, O ′ ) = 1 |O| 1 i clip |O i -O ′ i | max(|O i |, |O ′ i |) + ϵ , 0, 10In this equation, |O| 1 represents the number of elements in the raw output vector O. The summation iterates over each element i in the vectors O and O ′ . For each element, the absolute difference between the raw output O i and the output O ′ i is calculated. This difference is then divided by the maximum value between |O i | and |O ′ i |, which serves as a normalization factor to make the error relative to the magnitude of the output values. To avoid numerical instability in cases where both O i and O ′ i are very small or zero, a small positive constant ϵ (set to 10 -6 in our experiments) is added to the denominator. The clip function ensures that the resulting ratio is clipped to the range [0, 10], preventing extreme values from dominating the overall error. The clipped ratios are summed and then divided by the total number of elements |O| 1 to obtain the mean relative absolute error. This metric provides a normalized measure of the average relative deviation between the predicted and raw outputs, with values ranging from 0 to 10 (maximum allowed error).As shown in Fig. 10 , the subjective evaluation of DiTFastAttn's application to video generation tasks revealed a significant performance distinction across configurations D1 through D6. Configurations D1 to D3 demonstrated effective performance, balancing computational efficiency with the retention of visual quality in generated videos. The subjective assessment indicated that videos generated under these configurations were smooth, with natural transitions between frames and preserved details that are critical for video quality. The maintenance of these qualities suggests that the model was able to effectively leverage the redundancies identified and apply the compression techniques without noticeable loss to the viewer.In contrast, configurations D4 to D6, which applied more aggressive compression techniques, resulted in a noticeable deviation from the original video characteristics. The generated videos under D4 to D6 were still smooth and coherent, allowing them to represent the intended narrative or prompt with reasonable accuracy. This suggests that while aggressive compression can compromise certain aspects of video quality, it can still be effective in scenarios where computational resources are limited and a high level of detail is not paramount.The subjective results underscore the importance of finding an optimal balance between computational efficiency and generation quality when applying DiTFastAttn to video generation tasks. While configurations D1 to D3 offer a promising trade-off, the deviation observed in D4 to D6 highlights the need for careful consideration of the compression parameters. For practical deployment, it is crucial to select a DiTFastAttn configuration that aligns with the specific requirements of the application in terms of both performance and output quality. Fig. 11, 12, 13 display compression plan obtained after our greedy search method in different model settings as heatmaps. Each block stands for one layer at specific step. Both models exhibit three different kinds of redundancy, but the distribution of these redundancies across time steps and layers is quite different. The results indicate that there is no uniform compression plan for different DiT models and a search plan is essential in this case.Table 6 show the plan search time in different configuration. Our greedy search method will try method that can achieve high compression ratio so plan search time is inversely proportional to threshold. When designing the compression plan, we have considered to use other metrics including LPIPS and SSIM, and finally chose the existing metric mainly because of the speed of computation. We tested different SSIM compression schemes and found that when SSIM is chosen as the metric, to ensure the quality of the images generated, the threshold should be set at a small value of about 1/10 of the existing metric. We checked LPIPS as an alternative metric by decoding the hidden states into RGB space then calculate LPIPS on RGB space. We found that LPIPS is very insensitive to value changes in our use cases. Only small value changes can be observed when switch diffrent methods and always suggest to use sharing across timestep when threshold is set to 0.005 or smaller. Addistionally, it takes a long time to compute LPIPS. In this way, we believe LPIPS is not a suitable metric for compression plan search.Negative conditioning is a technique that widely used to improve generation quality by specifying what to exclude from the generated images. We explored the the impact of negative conditioning on our method using general negative prompt like 'Low quality' on PixArt-Sigma-XL. In the case, we found our method can preserve the effect of negative prompt on the generated images as shown in Fig. 14 . Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper proposed a combination of techniques to reduce the self-attention computation in diffusion transformers (DiTs), without requiring finetuning. The proposed method leverages locality in attention scores, similarity in timesteps, and similarity with classifier free guidance. Overall, the proposed method is capable of maintaining image quality while significantly reducing self-attention FLOPS. After rebuttal and discussions, the recommendations from the reviewers were 3XBorderline Accept and 1XAccept. The reviewers were consistent in the paper's contributions while they also made lots of comments in improveing the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a combination of novel techniques to reduce the self-attention computation in diffusion transformers (DiTs), without requiring finetuning. This methods leverage locality in attention scores, similarity in timesteps, and similarity with classifier free guidance. Overall, the proposed method is capable of maintaining image quality while significantly reducing self-attention FLOPS.
strength_weakness: strength_weakness: While the visual results are compelling, the paper has limited quantitative evaluations, and may be in a regime with low statistical SNR requiring additional samples. Furthermore, the authors focus on improving FLOPS, however, they do not discuss VRAM usage which would be significant for consumer grade devices and generating longer videos. Finally, there are issues with figure 5 which significantly hinders readability.
questions: questions: 1) You should cite Sora in the introduction and not OpenSora.
2) While the inclusion of residual caching is novel, how is your windowed attention method different from neighborhood attention (Natten)? This should be cited with local attention.
3) Including the 1-sigma boundaries (e.g. aggregated over 1k samples) in the MSE plots on Fig 3a would help understand how much variation there is for the attention layers. Additionally, is that DiT-XL or Pixart-Sigma? Identifying the model in the caption would be a good idea.
4) You explored the impact with CFG but have you looked at the impact with negative conditioning? Does sharing the conditional attention perform similarly or degrade?
5) There appears to be some interesting behavior in Figure 4, which may be interesting to explore from an explainability perspective. 
6) For your compression plan search, did you consider using other metrics such as LPIPS or SSIM? Would they impact the chosen plan?
7) The use of 5k image samples for evaluation is unusual. Typically the evaluation is 50k for ImageNet and 30k for COCO, as a larger sample size will reduce statistical noise (FID can be very noisy below 15k samples). 
8) When you evaluated FID, IS, CLIP, did you do so on the same 5k samples used for determining the plan? I would have used the ImageNet/COCO validation set for the plan search and then evaluated on the test set to ensure the two distributions were non-overlapping.
9) What dataset did you use for your OpenSora evaluation? 
10) Did you consider looking at the impact on VRAM? If there is no impact, stating so in the limitations would be appropriate. 
11) If increasing the step count improves the result in Fig 9, then this suggests that there is an optimal tradeoff between attention FLOPs and step count (iso-sampling FLOPS). Performing that analysis may strengthen adoption of your technique.
12) Reporting relative performance as percentages is difficult to read. It would be clearer to report them as ratios (e.g. 0.93 vs 93%)

Finally, Figure 5 is unreadable and requires revision. 
- FID and IS should not be plotted on the same axis as they are very different in scale. Use a dual-axis plot if you want to combine them.
- The points should be connected with a line like in Figure 8 and 9.
- I would only plot CLIP and FID (not IS) for Pixart, which will make the plot easier to read. 
- Your CLIP scores appear too high, where they should be between 0.2 and 0.35. 

Please provide a revision to this figure or a table of the data plotted therein.
limitations: limitations: The authors adequately discuss limitations of their approach, although should include a note about VRAM.
paper_summary: paper_summary: The paper  presents a post-training model compression method aimed at reducing the computational complexity of Diffusion Transformers. The authors identify three key redundancies in the attention computation. To address these, they propose three techniques. The proposed methods compress the model FLOPs and enable more efficient deployment in resource-constrained environments. The paper includes extensive experimental results demonstrating significant reductions in computational cost and latency while maintaining generation quality.
strength_weakness: strength_weakness: - Limited Theoretical Analysis: I am in general agreement with the overall concept presented in the paper. However, the paper primarily focuses on empirical results, with limited theoretical analysis to support the proposed methods. A deeper theoretical foundation could help in understanding the generalizability and limitations of the techniques. 
- Complexity  and Scalability: The paper does not extensively discuss the complexity of the compressed models. The figure 1 presents the efficacy under different resolutions. But it does not discuss the model complexity, running speed, and model size in these settings.
- Insufficient Comparisons: The paper does not compare the related methods, such as flashattention, and KV cache. This mainly weakens the contribution. A more detailed comparison with existing attention acceleration and model compression techniques would be beneficial. 
- Vague Illustration：The authors present three key redundancy strategy. However, the paper lacks sufficient annotations and explanations and does not provide clear definitions for the three scenarios.
- Application: The paper mainly focuses on attention reduction, but It is not very effective at low resolutions because the attention mechanism accounts for a small proportion of the computational cost.
questions: questions: - How do the authors ensure the generalizability of their method across different datasets and tasks? A more rigorous theoretical analysis could help understand the limitations and potential extensions of the proposed techniques.
- What is the model complexity, running speed, and model size under different compression settings?
- Can the authors offer clearer definitions and detailed illustrations for each redundancy type (spatial, temporal, and conditional redundancy)?
limitations: limitations: The authors have acknowledged several limitations of their work.
paper_summary: paper_summary: The authors observe computational redundancies across the three main dimensions of the generation process in DiTs -- space, sampling time, and conditional vs. unconditional forward passes due to CFG. Based on these observations, the authors introduce a set of approaches leveraging them to enable more efficient inference with negligible quality loss for pre-trained DiTs.
strength_weakness: strength_weakness: - While the method as a whole is evaluated reasonably well, the evaluation of the effect of parts of the method & design choices seems lacking: there are missing ablations regarding different variants (e.g., naive approaches) of leveraging specific redundancies, and especially the evaluation of the compression plan algorithm is lacking and its hyperparameters (threshold $\delta$, number of samples).
- Open questions remain, especially regarding unorthodox design choices and whether the CFG redundancy is actually an attention redundancy as described or a general redundancy already thoroughly described in previous works. See the questions below.
questions: questions: Questions regarding the main contributions of the paper
- While their approach to only computing attention locally does seem to be effective, I am surprised by the fact that the authors chose to apply a 1D window attention on the flattened image token sequence instead of one of the many 2D-aware local attention mechanisms such as (Shifting) Window Attention. These should have been the obvious choice in this situation, and the authors fail to motivate their unorthodox choice. While the attention map patterns presented in Fig. 3a do look somewhat like the effective attention pattern given by 1D window attention, I would presume that the additional diagonal lines next to the main diagonal correspond to the same location, but offset by one row. Besides generally being a more intuitive choice, the aforementioned approaches should also correspond to this observed pattern significantly better.
- Is it possible that the observed CFG redundancy is not an attention-specific aspect but a general one that is already well-described in previous literature? I would be very interested in a comparison with CFG not being applied at all for low noise timesteps, given the large range of prior works that found it beneficial to, substantially anneal the CFG scale for low noise timesteps (MDT, Gao et al., 2023; Analysis of Classifier-Free Guidance Weight Schedulers, Wang et al., 2024; and a wide range of non-published sources such as https://enzokro.dev/blog/posts/2022-11-28-sd-v2-schedules-1/, https://x.com/jeremyphoward/status/1584771100378288129, https://github.com/mcmonkeyprojects/sd-dynamic-thresholding), or even completely deactivate it (Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models, Karras et al., 2024).
- What is the distribution of which acceleration method is used how often, and where in the generation process?

Additional small questions
- Can the authors provide an explanation regarding the intuition of FID improving when reducing the attention FLOPS in Fig. 5?
- Subjectively, more compression results in a loss of contrast, both for generated images (albeit slightly), as well as videos. Do the authors have an explanation as to what might be causing this?
- For their similarity analyses in Fig. 4, the authors only analyze the cosine similarity of attention outputs. Are the magnitudes similar as well? Wouldn't this be a requirement to enable successful caching?

Presentation suggestions
- Fig. 5 is hard to parse due to the multitude and different scales of the presented metrics. I'd suggest that the authors separate the metrics into separate stacked graphs, which should also render these graphs more accessible for people with color perception impairments. I would also suggest adding arrows to indicate whether lower or higher is better for each individual metric. Similar improvements to other figures could also help make the paper more accessible.
- For a camera-ready version, I think the paper would benefit from additional uncurated qualitative examples
limitations: limitations: The authors have adequately addressed the limitations and societal impact of their work.
paper_summary: paper_summary: In this paper, the authors introduce a novel post-training model compression method aimed at accelerating Diffusion Transformers (DiT) used for image and video generation tasks. They identify the spatial, temporal, and conditional redundancy in attention blocks and propose the corresponding method to tackle them.
strength_weakness: strength_weakness: - The authors sample 5K images to evaluate the generation quality while the standard setting is sampling 50K images (following original DiT).
- What is the adopted compression strategy in each layer and each timestep? It would be better to illustrate the searched compression plan in a figure.
- Can the proposed method be combined with other acceleration methods \eg Deepcache?
- Why do the authors only demonstrate the Inception Score in Figure 9? FID is the most common metric in image generation.
- What are the FLOPs and Latency in Table 1 exactly? The authors should demonstrate the value rather than only the fraction since there is still enough space.
questions: questions: See weakness
limitations: limitations: See weakness

Comments: title: Clarification on updated results
Clarification on updated results
Additional reply for the concerns about evaluation setting
Compression strategy search time
comment: After reviewing the authors’ rebuttal and all subsequent responses, I find that most of my concerns have been addressed. Consequently, I have decided to revise my rating to a positive one. I recommend that the additional experiments and analyses discussed be incorporated into the revised version of the manuscript.
Thank you kindly for this acknowledgement and recognition of our work.
Thank you for your reminder. We have tested the latency to the table for your reference. We observed a 30% reduction in DiT-XL/2 attention latency at a threshold of 0.05 (timesteps=250, 512x512 generation), showing that our method not only reduces FLOPs but also results in a significant speedup in the attention calculation part. Regarding the overall inference latency, your observation is correct that the latency reduction fraction is lower than the attention latency reduction fraction. However, the proportion of attention computation in the overall computation would increase along with resolution (as shown in Figure 1 of our paper). Our method can significantly reduce the overall latency on the generation of high-resolution images. For 2K image generation, attention computation contributes more than 70% of the total computation time, so a 30% reduction in attention latency will result in more than a 21% reduction in overall latency.

Regarding the number of calibration samples, we agree with your idea and will increase the number to further reduce degradation. However, it will take more computation cost. This is a trade-off between calibration time and performance. We will carefully evaluate the number of samples in the future. Thanks again for your remind and advice. If we have addressed all of your concern, we kindly ask you to reconsider the score.
Thank you for the additional context. 

I remind the authors that FLOPs is not a good metric for inference speedup, especially considering that many inference workloads are memory bound. While FLOPs can serve as a heuristic, the bigger picture is more complicated. Instead, overall inference latency is a more appropriate metric of which a significant deviation is not observed for thresholds below 0.1. From the other results, it is my belief that your upper performance bound is limited by the number of calibration samples, which results in plans that are unable to effectively generalize across the standard evaluation metrics. The authors should focus on that aspect: reducing degradation for higher thresholds to achieve significant latency reductions, rather than more results with thresholds below 0.05.
Our findings indicate that even at a threshold of 0.025, the method was able to significantly reduce the attention FLOPs by 46%, while maintaining a satisfactory generation effect. **At a threshold of 0.05, the attention FLOPs were reduced by 63% without significant degradation**. The results show that as the number of steps increases, the threshold required to achieve the same compression ratio decreases. For example, **with 250 steps, a threshold of 0.025 can achieve the same compression ratio as a threshold of 0.125 with 20 steps**. This means that for larger timesteps, a smaller threshold is needed to reach the same compression ratio.

Therefore, we believe our results demonstrate that in the experiment setting of 250 steps, a small threshold can indeed achieve good compression performance. We will adjust the threshold values and include more results in the range of 0 to 0.05 to better illustrate this point.
We appreciate your valuable feedback. In the revised version, we will include the LPIPS results and additional evaluation metrics, as well as more uncurated qualitative examples, as you suggested.

Regarding your interpretation of the updated evaluations (Q1), there seems to be some misunderstanding. Our findings indicate that even at a threshold of 0.025, the method was able to significantly reduce the attention FLOPs by 46%, while maintaining a satisfactory generation effect. **At a threshold of 0.05, the attention FLOPs were reduced by 63% without significant degradation**. The results show that as the number of steps increases, the threshold required to achieve the same compression ratio decreases. For example, **with 250 steps, a threshold of 0.025 can achieve the same compression ratio as a threshold of 0.125 with 20 steps**. This means that for larger timesteps, a smaller threshold is needed to reach the same compression ratio.

Therefore, we believe our results demonstrate that in the experiment setting of 250 steps, a small threshold can indeed achieve good compression performance. We will adjust the threshold values and include more results in the range of 0 to 0.05 to better illustrate this point.

We kindly ask you to reconsider the score if we have addressed all of your concern.
Thank you for the follow-up clarification.

Q1.

The updated FID and IS scores are closer to what is expected. While I understand your desire to evaluate under more practical settings, maintaining a consistent evaluation protocol is necessary to compare with other works. This requirement for comparison is much more sensitive to image count and cfg scale rather than sampler, which is why many recent works deviate in sampler algorithm and step count (computational practicality).

From your updated evaluations, I interpret the results as significant degradation beyond a threshold of 0.05, below which the speedup is not significant. 

Q2.

The results you described with LPIPS are interesting, and should be included in the revision for completeness. If anything, they will at the very least indicate that a less computational method achieves similar or better results for less overhead.

Q3. 

Your results exhibit a significant improvement using more samples for a threshold above 0.1. When interpreting FID, a shift of 1-2 points can be significant, although this may be reduced with the updated conditions from Q1.  

I remain concerned with the low number of images used in calibration, which I believe may serve to highlight issues in the generality of FID and IS as standard evaluation metrics. While I understand that more samples are computationally expensive, the cost largely becomes irrelevant if it need only be performed once per model.


---

I thank the authors for their effort in addressing my questions and those of the other reviewers. Additionally, I commend the authors on their detailed investigations, which make this an interesting technical work. As such, I am inclined to raise my score. However, I believe the variable quality as a function of calibration sample count brings into question the efficacy of the results. Furthermore, the results presented bring into question the appropriateness of using FID and IS as primary evaluation metrics rather than as a sanity check against significant degradation. Given the aforementioned pros and cons, I raise my score from 4 to 5.
Thank you for running the additional evaluation with CFG dropping. I'm surprised all the efficiency improvements actually improve the FID, which is counterintuitive. I'd have appreciated additional FLOP/MAC numbers to put these values into perspective, as my comment was primarily concerned with the question of whether just dropping CFG for the final few steps is a sufficient alternative for the proposed method, but looking at the MAC counts in the comment you posted in the main thread, I'd estimate the computational savings for your method at threshold 0.1 to be substantially greater than just CFG dropping. It's also interesting that CFG dropping apparently greatly boosts the IS. For me, this creates many interesting questions about the effect of CFG dropping that I wasn't thinking of before, but it has definitely addressed my concern about this work.

Given that the authors have adequately addressed all my major concerns and questions and have addressed the question about the unusually high FID scores raised by the other reviewers, I will raise my score from 6 to 7 and hope for acceptance.
Dear Reviewer e43K, Thanks so much again for the time and effort in our work. According to the comments and concerns, we provided more analysis and further discuss the related points. We also conduct a variety of ablation studies and experiments that we showed in the Global rebuttal.

As the rebuttal period is about to close, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are welcome to address them. Thanks again for taking the time to review our work and provide insightful comments.
## Explanation
The experiment settings are different in the following ways:
1. The number of timesteps. To compare DiT with ADM and LDM, DiT uses a relatively high number of diffusion timesteps (250). For most image generation applications, 20-50 steps would be reasonable. Other methods, such as SD-XL and PixArt series, also evaluate on 20-50 steps. That is why we use 20-50 steps in our paper.
2. Evaluation software. We use pytorch-fid & torchmetric to calculate FID and IS scores, while DiT uses the TensorFlow evaluation suite from the ADM paper. This resulted in some differences in the FID and IS scores.
3. We use a cfg_scale of 4, which is the default setting for both DiT official code and diffusers DiT pipeline. In the DiT paper, they use a cfg_scale of 1.5 to match the settings used for ADM and LDM.

## Results after changing experiment setting to align with DiT paper

| Threshold | IS@50K | FID@50K | macs | attn_mac |
|-----------|--------|---------|------|----------|
| Raw setting | 219.9721 | 3.16   | 262359 | 33823   |
| 0.025     | 218.1955 | 3.09   | 236265 | 18041   |
| 0.050     | 210.3559 | 3.10   | 218865 | 12339   |
| 0.075     | 196.05  | 3.54   | 203420 | 8777    |
| 0.100     | 180.34  | 4.52   | 195682 | 7137    |

## Analysis of results

From this table, we observe that when the threshold is increased from 0 to 0.10, the IS drops from 210 to 180, and the FID decreases from 3.16 to 3.09, then increases to 4.52. The behavior of the FID in different settings follows the pattern reported in the previous discussion[1]. We observe a higher reduction in FLOPs and Attention FLOPs in this setting. We believe the current results demonstrate the effectiveness of our method across different settings.

[1] Jayasumana S, Ramalingam S, Veit A, et al. Rethinking fid: Towards a better evaluation metric for image generation[C]// CVPR. 2024: 9307-9315.
Thanks for your patience. Here we post the results for DiT after changing our experiment setting to align with DiT paper. From this table, we observe that when the threshold is increased from 0 to 0.10, the IS drops from 210 to 180, and the FID decreases from 3.16 to 3.09, then increases to 4.52. The behavior of the FID in different settings follows the pattern reported in the previous discussion[1]. We observe a higher reduction in FLOPs and Attention FLOPs in this setting. We believe the current results demonstrate the effectiveness of our method across different settings.
  
| Threshold | IS@50K | FID@50K | macs | attn_mac |
|-----------|--------|---------|------|----------|
| Raw setting | 219.9721 | 3.16   | 262359 | 33823   |
| 0.025     | 218.1955 | 3.09   | 236265 | 18041   |
| 0.050     | 210.3559 | 3.10   | 218865 | 12339   |
| 0.075     | 196.05  | 3.54   | 203420 | 8777    |
| 0.100     | 180.34  | 4.52   | 195682 | 7137    |

So far, we increased evaluation sample size, changed experiments settings and provided results that aligned with the original paper according to W1; provided compression plan and search time that asked in W2 and FLOPs number that asked in W4; explained the combination with other acceleration model for W3; revised our paper according to W5. We are happy to answer additional questions and provide more clarification if needed. We kindly ask you to reconsider the score since we nearly reached the end of the discussion period.  

[1] Jayasumana S, Ramalingam S, Veit A, et al. Rethinking fid: Towards a better evaluation metric for image generation[C]// CVPR. 2024: 9307-9315.
Thank you for your patience. Here we present the results for DiT after changing our experimental settings to align with the DiT paper.
| Threshold | IS@50K | FID@50K | macs | attn_mac |
|-----------|--------|---------|------|----------|
| Raw | 219.9721 | 3.16   | 262359 | 33823   |
| 0.025     | 218.1955 | 3.09   | 236265 | 18041   |
| 0.050     | 210.3559 | 3.10   | 218865 | 12339   |
| 0.075     | 196.05  | 3.54   | 203420 | 8777    |
| 0.100     | 180.34  | 4.52   | 195682 | 7137    |

From this table, we observe that when the threshold is increased from 0 to 0.10, the IS drops from 210 to 180, and the FID decreases from 3.16 to 3.09, then increases to 4.52. The behavior of the FID in different settings follows the pattern reported in the previous discussion[1]. We observe a higher reduction in FLOPs and Attention FLOPs in this setting. We believe the current results demonstrate the effectiveness of our method across different settings.

So far, we provide explanations, revision and examples according to each concerns. We are open to more discussion and will to provide more clarification if needed. As we nearly reached the end of the discussion period, we kindly ask you to reconsider the score.

[1] Jayasumana S, Ramalingam S, Veit A, et al. Rethinking fid: Towards a better evaluation metric for image generation[C]// CVPR. 2024: 9307-9315.
Thanks for your comments and valuable suggestions.

> Regarding your response to my CFG sharing comment, I'd like to clarify that I was primarily referring to the practice of omitting CFG for the final few steps altogether, which also seems to have a negligible effect in practice, and is substantially simpler.

Thanks for your clarification. We conducted an ablation study on CFG dropping in the final steps. The results are shown in the following table. We observed that the method of CFG dropping in the final steps is effective in DiT. We also observed that our DiTFastAttn can work well with this method. We will add this to the appendix in our revised version of the paper, along with more evaluation results and generated images.

| Setting             | IS@5K | FID@5K  |
|---------------------|----------|------|
| Raw                 | 208.3134 | 31.65|
| CFG Dropping final 10% steps       | 278.4405 | 30.5 |
| Threshold = 0.1     | 200.2979 | 26.91|
| Threshold = 0.1 + CFG Dropping final 10% steps    | 262.5407 | 26.77|

Note that the method only focuses on CFG dropping in the final steps, while our CFG sharing in our method reduces redundancy in other steps. Since the CFG dropping is manually set now, it is an interesting idea to automatically search for where to drop the CFG. We are considering researching this point in the future.

> I'd also suggest including that in a future revised version of the paper and potentially still running practical quality evaluations with Natten, as it should be substantially more FLOP-efficient and might catch up w.r.t. practical speed with innovations like flex attention. For now, the provided context is sufficient for me.

Thanks for your suggestion. We will include the Natten results in the revised version of our paper.

So far, we have provided the results of 1D vs 2D local attention, n_samples, loss of contrast, magnitude, and the compression plan for Q1, 3, 5, 6, and 7. We have also provided a reasonable explanation for Q2, and an analysis of CFG sharing to address W2 and Q2. We are open to further discussion and happy to answer any additional questions. As we are nearing the end of the discussion period, we kindly ask you to reconsider the score.
Thanks for your additional questions. Here is our reply:

> Q1. Evaluation metrics

The experiment settings are different in the following ways:
- 1. The number of timesteps. To compare DiT with ADM and LDM, DiT uses a relatively high number of diffusion timesteps (250). For most image generation applications, 20-50 steps would be reasonable. Other methods, such as SD-XL and PixArt series, also evaluate on 20-50 steps. That is why we use 20-50 steps in our paper.
- 2. Evaluation software. We use pytorch-fid & torchmetric to calculate FID and IS scores, while DiT uses the TensorFlow evaluation suite from the ADM paper. This resulted in some differences in the FID and IS scores.
- 3. We use a cfg_scale of 4, which is the default setting for both DiT official code and diffusers DiT pipeline. In the DiT paper, they use a cfg_scale of 1.5 to match the settings used for ADM and LDM.

We have modified the code to reproduce the results. However, it is not feasible to resample the 50K images under the timestep=250 setting during the rebuttal period (this would take approximately 55 GPU hours). We will provide a partial set of the results soon.

For PixArt-Sigma, they used a curated set of 30,000 images instead of COCO to calculate FID. Since the dataset is not publicly released, we cannot reproduce those results. We will clearly explain the experiment settings in the appendix.

> Q2. LPIPS implementation

Thank you for the suggestion. We have implemented LPIPS as an additional evaluation metric (by decoding the hidden states into RGB space). However, we found that LPIPS has problems for our use case: 
- 1. Insensitivity to Value Changes: We observed that LPIPS is quite insensitive to value changes in our experiments. Only small value changes were observed when switching between different methods, and LPIPS always suggested using sharing across timesteps when the threshold was set to 0.005 or smaller.
- 2. Computational Overhead: Using LPIPS as a metric takes significantly more time. Therefore, we were unable to provide the LPIPS results within the rebuttal period due to the computational requirements.

> Q3. Number of calibration images

We conducted an ablation study on the number of samples generated during the plan search, and found that increasing the number of samples had little effect on the resulting compression plan.

| n samples | threshold | IS    | FID           |
|---------|-----------|-------|---------------|
| 8       | 0.05      | 206.4319 | 29.97 |
| 8       | 0.1       | 200.2979 | 26.91  |
| 8       | 0.15      | 179.1331 | 23.114 |
| 16      | 0.05      | 205.7642 | 30.33 |
| 16      | 0.1       | 197.1631 | 27.62 |
| 16      | 0.15      | 180.6786 | 23.24  |
| 32      | 0.05      | 205.626  | 30.35 |
| 32      | 0.1       | 202.7274 | 26.97 |
| 32      | 0.15      | 180.2231 | 24.55 |

From the results, the IS and FID did not change significantly as we varied the number of samples. Nevertheless, users can easily set 'n' to a larger number if desired, as it is a configurable parameter in our code.

>Q4. downscaled method like HiDiffusion be applicable?

We have carefully reviewed the HiDiffusion approach, which is designed for UNet-based diffusion models. We have considered how this method may be applicable to our framework:
- 1. HiDiffusion proposed using local attention to replace global attention in the top layers of the UNet, as global attention in the upper blocks is computationally dominant. However, in DiT, the amount of self-attention computation is equal across each layer. Therefore, we do not need to focus solely on specific layers, and instead use a search-based method to automatically identify which layers can be replaced.
2. HiDiffusion use the Modified Shifted Window Attention, where the window area is set differently across diffusion timesteps. This approach may be applicable to our method as well. However, as mentioned in the Global rebuttal (Q4), the inference speed of these attention mechanisms could be slower. It is also an interesting topic to research.

>Q5. Pareto front question

Thanks for your suggestion. We have followed your advice and added marker shapes. Here is our observation from the plot:
- The point representing compression at 30 steps (yellow marker at (2.7 TFLOPs, 208 IS)) is superior to using no compression at 20 steps (blue marker at (2.9 TFLOPs, 207 IS)). This is because the compressed 30-step model has lower computational cost (TFLOPs) while achieving a higher Inception Score (IS).
- For a high FLOPs budget (> 3.5T), it is better to use moderate compression with 50 steps.
- For a medium FLOPs budget (2-3.5T), it is preferable to use slight compression with 20 steps or moderate compression with 30 steps.
- For a low FLOPs budget (< 2T), the optimal choice would be to use high compression with either 20 or 30 steps.
I also have the same question: 'The FID scores are too high (DiT-XL/2 should be around 2.2 with CFG, and Pixart-sigma around 9). Similarly, your IS scores for DiT-XL are too high and too low for Pixart.'
Thank you for the extensive response and for running the additional ablations in the short rebuttal timespan!

Regarding your response to my CFG sharing comment, I'd like to clarify that I was primarily referring to the practice of omitting CFG for the final few steps altogether, which also seems to have a negligible effect in practice, and is substantially simpler.

Regarding the Natten comparison, I'm really surprised that the Natten kernel is seemingly so inefficient despite the substantially smaller number of elements, and I thank the authors for providing that valuable context. I'd also suggest including that in a future revised version of the paper and potentially still running practical quality evaluations with Natten, as it should be substantially more FLOP-efficient and might catch up w.r.t. practical speed with innovations like flex attention. For now, the provided context is sufficient for me.
Thank you for pointing out this observation.
We have thoroughly reviewed the evaluation code and confirmed that there are no issues on our end. We believe the discrepancy may be due to the inaccuracy of the FID metric, as indicated in previous research [1]. The FID does not fully capture the visual quality of the generated images. However, changes in the FID score can still be meaningful, as they reflect shifts in the distribution of the generated images.

We checked a large number of the generated images, and found that the quality does decrease with an increase in the threshold. Based on this finding, we will add more generated samples and human evaluation into the paper.

[1] Jayasumana S, Ramalingam S, Veit A, et al. Rethinking fid: Towards a better evaluation metric for image generation[C]// CVPR. 2024: 9307-9315.
The results in Table 1 and Table 2 are strange.  Lower IS and lower FID? I think that when the quality of the image decreases, the IS will be lower but FID will be higher.
I thank the authors for their effort with additional evaluations, the SSIM difference and compression plans are especially interesting.

There are a few points that I would like clarification on.

**Q1.**

There appears to be an issue with your evaluation metrics. The FID scores are too high (DiT-XL/2 should be around 2.2 with CFG, and Pixart-sigma around 9). Similarly, your IS scores for DiT-XL are too high and too low for Pixart. Were you able to reproduce the metrics from the respective papers without your compression method?

**Q2.**

> LPIPS only works for image comparisons and requires the channel number to be set to 3...

This is partially true, where it would be possible to retrain a vgg model as per [1] in the latent space. However, it would also be feasible to simply decode the latent images and apply LPIPS in RGB space (this is also a differentiable operation).

**Q3.**

> For the calibration process, we only use class labels (for DiT) and text prompts (for PixArt-Sigma) to generate a small set of images (8 images for DiT, 6 images for PixArt 1K, and 4 for PixArt 2K). No real images are used in the calibration process, so there is no risk of overlapping.

Is this a sufficient number of images to establish a robust compression plan? If the goal is to find a method for downstream inference performance, I would want to have at least 1k samples (maybe more).

**Q4.**

Given the early focus on full self-attention in the compression plans, would some downscaled method like HiDiffusion [2] be applicable? 

**Q5.**

> ...we have added a FLOPs-IS Pareto front plot...

Interpreting this plot is difficult, perhaps adding marker shapes for each compression ratio would make it clearer. However, if I understand correctly, it suggests that using no compression at 20 steps performs similarly to using moderate compression at 30 steps? If so, can you give an example for where your approach would be preferable to simply reducing the sampling steps?

[1] Zhang, R., et.al. (2018), "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric"

[2] Zhang, S., et.al. (2023), "HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Models"
Thanks for your comment.

Search time varies for different models and parameter settings. We use a greedy method in compression plan search so increase threshold (more compression) will result in a shorter search time. The maximum search time for a model can be determined by setting the threshold to 0.  One thing to note is that once the search is completed, the compression strategy will be locally cached, eliminating the need for a repeated search. The cached strategy will be utilized the next time our method is applied for image or video generation. For reference, here we post the search times for DiT and PixArt-Sigma 1K using the default settings.

#### Table1. Search time for DiT-XL-2-512

| Threshold | Search Time |
|-----------|-------------|
| 0         | 04m39s      |
| 0.05      | 04m08s      |
| 0.1       | 03m49s      |
| 0.15      | 03m14s      |

#### Table2. Search time for PixArt-Sigma-XL-2-1024-MS

| Threshold | Search Time |
|-----------|-------------|
| 0         | 22m02s      |
| 0.05      | 20m12s      |
| 0.1       | 17m50s      |
| 0.15      | 15m49s      |
How much time do we need to search the compression strategy?