FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality

Authors: Zhengyao Lv, Chenyang Si, Junhao Song, Zhenyu Yang, Yu Qiao, Ziwei Liu, Kwan-Yee K. Wong
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Efficient Video Synthesis

Abstract:
In this paper, we present \textbf{\textit{FasterCache}}, a novel training-free strategy designed to accelerate the inference of video diffusion models with high-quality generation. By analyzing existing cache-based methods, we observe that \textit{directly reusing adjacent-step features degrades video quality due to the loss of subtle variations}. We further perform a pioneering investigation of the acceleration potential of classifier-free guidance (CFG) and reveal significant redundancy between conditional and unconditional features within the same timestep. Capitalizing on these observations, we introduce FasterCache to substantially accelerate diffusion-based video generation. Our key contributions include a dynamic feature reuse strategy that preserves both feature distinction and temporal continuity, and CFG-Cache which optimizes the reuse of conditional and unconditional outputs to further enhance inference speed without compromising video quality. We empirically evaluate FasterCache on recent video diffusion models. Experimental results show that FasterCache can significantly accelerate video generation (\eg 1.67$\times$ speedup on Vchitect-2.0) while keeping video quality comparable to the baseline, and consistently outperform existing methods in both inference speed and video quality. \textit{Our code will be made public upon publication.}

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730360541964

thread_type:
official_review

<thread_object>
creator: reviewer_znbc

<content>
summary:
This paper presents a method to accelerate the computation for video diffusion models. Based on the observation that adjacent-time features exhibit subtle differences,  the authors propose a dynamic feature reuse scheme. Based on the observation that the conditional and unconditional generation features have strong similarity in CFG, the authors propose a CFG-Cache scheme to save computations of unconditional stage.

strengths:
this paper is relatively well organized and written.

weaknesses:
this paper lacks novelty and insight. the proposed method is not solid in terms of logical and theoretical justification. the speedup of computation could be achieved using other methods.

questions:
1. what is the reason of using equation (5) for feature reuse? it seems to be quite heuristic.
2. why in cfg, the similarity of two stages has such a wired characteristics (time-varing low and high frequency similarity)? there should be more analysis and insight.
3. in terms of saving the computation of unconditional stage, how about existing and well-adopted methods such as "On Distillation of Guided Diffusion Models"? 
4. how does your method work with video generation models using flow matching?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550106661

thread_type:
official_review

<thread_object>
creator: reviewer_jhfk

<content>
summary:
This paper propose a training-free method for efficient video generation. Their method is based on the attention feature map and output redundancy observed in the forward pass of the denoising process. To this end, they propose to (1) use the feature map dynamics to restore the missing details in the previous naive feature map reuse method. (2) use the frequency difference to compensate the cached conditional output in CFG. The proposed method has been validated with several models and showed promising results.

strengths:
- The paper is well-written and easy to read.
- The proposed attention feature map dynamics could be a generic method and applied into all other cache based efficient generation methods.
- the proposed frequency residual method for CFG is novel and seems also a generic method that can be applied into CFG for all diffusion based image/video generation models.

weaknesses:
- Distillation based efficient generation methods are less discussed and not added into the comparison. I can understand that this work mainly focused on the cache based method. But:
  - What's the delta of this method compared to distillation based method?
  - Are they compatible to each other? If not, in real world applications, which method we need to use?

- the overall method seems to perform better than all other methods. What's the contribution of each individual method? What the cost of additional computation in each individual method?

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707432743

thread_type:
official_review

<thread_object>
creator: reviewer_4a6e

<content>
summary:
This paper presents an innovative approach to accelerate video diffusion models without sacrificing quality. By analyzing existing cache - based methods and classifier - free guidance (CFG), the authors identify limitations and redundancies. They introduce FasterCache, which includes a dynamic feature reuse strategy for attention modules and CFG-Cache. The dynamic feature reuse strategy adjusts features across timesteps, maintaining distinction and continuity. CFG-Cache optimizes the reuse of conditional and unconditional outputs. Experimental results on multiple video diffusion models demonstrate significant acceleration while keeping video quality comparable to the baseline, outperforming existing methods.

strengths:
In terms of originality, the proposed FasterCache method is highly innovative. It combines a dynamic feature reuse strategy that considers subtle differences between adjacent timesteps and a CFG - Cache component for handling conditional and unconditional outputs in a novel way. This is a departure from existing methods and reveals new acceleration opportunities through a pioneering investigation of classifier - free guidance (CFG). 

Regarding quality, a thorough analysis of existing cache - based methods and CFG is provided. This includes identifying problems and limitations, and conducting in - depth investigations into feature reuse and CFG outputs, which forms a solid foundation for the method. The experimental design is robust, with testing on multiple video diffusion models and the use of various evaluation metrics. Ablation studies further demonstrate the effectiveness of each component. 

For clarity, the paper has a clear structure, starting with an introduction to the problem and existing solutions, followed by a detailed description of the method in the methodology section. The experimental results are presented clearly, and the discussion and conclusions are well - organized. Complex concepts are explained accessibly, such as through diagrams and detailed descriptions of the key components. 

In terms of significance, the research has practical implications as it can reduce the time and computational resources required for video generation, which is important for applications like video content creation, virtual reality, and augmented reality. It also advances the field by filling gaps in existing research on video diffusion models, addressing limitations of current acceleration methods, and inspiring further studies on improving efficiency and performance.

weaknesses:
Hyperparameter Explanation. Some of the hyperparameters used in the method, such as the weighting function are not explained in sufficient detail. Although default values are provided and it is mentioned that they work well for most models, a more in-depth discussion on how these hyperparameters are chosen and their impact on the performance for different datasets and models would make the method more reproducible and understandable.

Uncertainty in complex scenes. The paper mentions that in complex scenes with substantial video motion, the method may occasionally produce degraded results. This may limit the practical applicability of the method once video generation model baselines are imporved and complex video content is common. More discussions refer to the Question 1.

Weak theoretical underpinning: The paper focuses more on the empirical evaluation and practical implementation of the FasterCache method. Although the experimental results are strong, the theoretical foundation behind some of the proposed techniques could be strengthened. For example, a more in-depth analysis of why the dynamic feature reuse strategy and CFG-Cache work as expected in different scenarios from a theoretical perspective would add more credibility to the method.

questions:
Questions:
1. Is the effectiveness of the Dynamic Feature Reuse Strategy because the videos generated by existing video generation methods are relatively smooth, with little difference between adjacent frames, e.g. fixed backgrounds? If so, once the motion of the video is large and the adjacent frames change greatly, then is the proposed method still effective?

2. Negative prompt will affect the quality of video generation, which generally improves it. So will CFG-Cache and the separation of high and low frequencies have an impact on negative prompt setting?

3. For Open-Sora-Plan, although FasterCache achieved an acceleration of 1.68x, the generated video has significant varying in pixel domain compared to the original one (PSNR 23.72). What is the reason for this bias? Blurriness, noise, or changes in semantic level?

4. Will the setting of g value in CFG formula 4 (Line 167) affect the performance of FasterCache?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720519700

thread_type:
official_review

<thread_object>
creator: reviewer_4sx2

<content>
summary:
This paper proposes a training-free feature caching strategy for video diffusion models, named FasterCache, that improves how features are reused for accelerating inference. Instead of directly reusing the adjacent-step features as in previous works, FasterCache interpolates the neighboring features to efficiently synthesize features for the missing steps. In addition, the paper introduces CFG-Cache, which aims to effectively reuse the conditional outputs for computing the unconditional outputs. Experimental results demonstrate notable visual quality and efficiency improvements compared with the existing methods.

strengths:
1. The paper clearly demonstrates the motivation and the problem of existing works with illustrative examples (Fig. 3-7). Overall, the limitations of vanilla cache-based acceleration methods with CFG is analyzed very thoroughly.

2. Qualitative results demonstrate notable improvements in visual details and the spatio-temporal consistency of the generated frames.

weaknesses:
1. Slightly missing justifications for the design choices of the proposed method. For instance, is the linear interpolation strategy sufficient for dynamically building the missing features? Are there any other options that the authors have considered? Using the proposed Dynamic Feature Reuse, how similar are the computed features compared to the existing caching methods? (Please guide me if I missed the descriptions)

2. Limited video-specific contributions. The proposed methodologies seem to applicable to any image diffusion model, of which comparison to previous works is missing, and I could not find any video-related novelties proposed. The notation $t = \{1 ... T\}$ in Equations seem to denote the diffusion sampling steps, and I think there is no notation for the temporal axis? Please correct me if I misunderstood any equations.

3. (Minor) Figure captions are not very descriptive. I would suggest to include the main points (Fig. 2, 8) or where to focus on (qualitative figures).

questions:
1. For CFG-Cache, should we interpret the feature differences in the frequency domain in a similar way as MSE? 

2. In Figure 7b, it seems to me that low-frequency components do not gradually shift into high-freq as the authors mention in L300-301, since the high-freq components differ both at the beginning and near the end. What kind of insights can we gain from this analysis?

3. In the experiments, PAB seems to significantly outperform $\Delta-DiT$, but there are no PAB results for CogVideoX and Vchitect. Also, why is the performance of $\Delta-DiT$ so low? Is it because of the inherent randomness of image diffusion models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732112010865

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response and summary of changes in the revision

comment:
We sincerely thank all reviewers for their time and effort in reviewing our paper!


In this work, we identify a common issue in previous cache-based methods: while features at adjacent timesteps are similar, ignoring the subtle differences and simply reusing them results in loss of detail. To address this, we propose the **Dynamic Feature Reuse** strategy, which introduces an additional feature bias term to preserve quality during acceleration. Additionally, through a comprehensive analysis of the video diffusion framework, we uncover significant redundancy in the unconditional computations of CFG, often overlooked by prior training-free acceleration methods. We introduce **CFG-Cache** to reduce this redundancy. By combining Dynamic Feature Reuse with CFG-Cache, FasterCache achieves significant acceleration while maintaining high fidelity. For example, on Vchitect 2.0, FasterCache achieved **80.84% VBench score and 1.67$\times$ speedup**, significantly outperforming the second-best result of **79.56% VBench score and 1.26$\times$ speedup**.


Below, we summarize the changes made in the revised paper:


1. We added the additional experiments about the justifications behind the hyperparameters or design choices in Appendix A.3.1 and A.5. We have also added supplementary explanations in the corresponding sections of the main text (L287 - L288). (Reviewer: 4sX2, 4a6E, zNbc)
2. We added Fig. 13 (b) to visualize the similarity of features computed by Dynamic Feature Reuse and existing caching based method (Vanilla Feature Reuse). (Reviewer: 4sX2)
3. We revised and supplemented the captions in Fig. 2, Fig. 8 and Fig. 9 for clarity. (Reviewer: 4sX2)
4. We have revised and supplemented Figure 7 (b) and the related explaination in Section 2.4 (L297 - L304). We also added a visual varations of CFG output in Fig. 14 and Appendix A.3.2. (Reviewer: 4sX2, zNbc)
5. We added the quantitative results of PAB on CogVideoX and Vchitect 2.0 in Table 1. (Reviewer: 4sX2)
6. We added the visual results of FasterCache based on larger scale video diffusion model (CogVideoX-5B and Mochi-10B) in Fig. 21 in Appendix A.4. (Reviewer: 4a6E)
7. We added a theoretical discussion on the effectiveness of Dynamic Feature Reuse and CFG-Cache in Appendix A.3.1 and Appendix A.3.2.  (Reviewer: 4a6E)
8. We added an experiment on the settings of CFG scale and negative prompt in Fig. 15 in Appendix A.3.3. (Reviewer: 4a6E)


The revised content is highlighted in blue.

We sincerely thank all reviewers again for their valuable suggestions, which have greatly helped strengthen our paper.


If you have any further questions, we would be happy to discuss them!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733121276811

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-up on Reviewers Feedback

comment:
We sincerely thank all reviewers for their constructive comments and the time and effort they dedicated to evaluating our paper. We hope we have adequately addressed all concerns and respectfully request the reviewer to take this into account in the final evaluation. 

Thank you very much!
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: This work proposes a dynamic feature-caching mechanism for inference-time video diffusion sampling. Unlike straightforward re-use of adjacent-time features, the method interpolates features of neighboring time steps in a weighted manner. Additionally, the authors identify and address a redundancy in classifier-free guidance (CFG), where conditional and unconditional features at the same time step overlap. They propose reusing conditional outputs to compute unconditional outputs, reducing redundancy. Experimental results demonstrate notable improvements in image quality and runtime performance.
Strengths: The main contribution is well-motivated, clearly illustrated, and supported by both qualitative and quantitative analyses. The work is highly relevant for accelerating video sampling, a process often hindered by computational expense.
Weaknesses: The contribution is somewhat narrow, focusing specifically on image/video diffusion sampling. The paper lacks a comparison with distillation-based methods that train smaller networks for similar acceleration goals.
Acceptance Reasoning:The proposed ideas, while simple, are highly relevant for speeding up diffusion sampling processes. The authors have sufficiently addressed most weaknesses raised by reviewers, leading to rating increases from some. Although not all reviewers responded after the rebuttal, the authors provided comprehensive responses to the concerns, addressing them effectively.