FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality

## Abstract

Abstract In this paper, we present FasterCache , a novel training-free strategy designed to accelerate the inference of video diffusion models with high-quality generation. By analyzing existing cache-based methods, we observe that directly reusing adjacent-step features degrades video quality due to the loss of subtle variations . We further perform a pioneering investigation of the acceleration potential of classifier-free guidance (CFG) and reveal significant redundancy between conditional and unconditional features within the same timestep. Capitalizing on these observations, we introduce FasterCache to substantially accelerate diffusion-based video generation. Our key contributions include a dynamic feature reuse strategy that preserves both feature distinction and temporal continuity, and CFG-Cache which optimizes the reuse of conditional and unconditional outputs to further enhance inference speed without compromising video quality. We empirically evaluate FasterCache on recent video diffusion models. Experimental results show that FasterCache can significantly accelerate video generation ( e.g. , 1.67 × \times × speedup on Vchitect-2.0) while keeping video quality comparable to the baseline, and consistently outperform existing methods in both inference speed and video quality.

## 1 Introduction

Diffusion transformers (DiT) (Peebles & Xie, ) have achieved notable success in image (Chen et al., ; ; Esser et al., ) and video generation (Ma et al., ; Zheng et al., ; Lab & etc., ), attracting significant attention for their potential. Although iterative denoising, classifier-free guidance (CFG) (Ho & Salimans, ), and transformer attention mechanisms have significantly improved the generative capabilities of diffusion models, they also lead to substantial computational costs and increased memory requirements for inference, especially for video generation which typically takes 2-5 minutes to synthesize a 6-second 480P video, limiting their practical use. This calls for the development of new techniques that
require less computational cost for diffusion models (Salimans & Ho, ; Ma et al., ; Chen et al., ; Zhao et al., ).

Among the recently proposed solutions, cache-based acceleration has emerged as one of the most widely adopted approaches. This approach speeds up the sampling process by reusing intermediate features across timesteps, thereby reducing redundant computations and significantly improving computational efficiency. Besides, it requires no additional training costs for inference acceleration and offers straightforward generalization to other video diffusion models. Examples include the cache-based methods for U-Net based diffusion models (Ma et al., ; Li et al., ), residual caching in $\Delta$-DiT (Chen et al., ) for the transformer based diffusion models, and hierarchical attention caching of PAB (Zhao et al., ) for video generation. Despite their proven effectiveness, there exist two critical concerns: 1) Whether directly reusing intermediate features aligns with the iterative denoising mechanism, considering the inherent feature variations between timesteps. 2) Current cache-based methods focus primarily on the attention features within the transformer networks, with limited exploration of accelerating different parts of the pipeline. In this work, we aim to address these two concerns.

Figure: Figure 2: Vanilla cache-based acceleration method.
Refer to caption: x2.png

To thoroughly investigate the acceleration potential of DiT inference for video generation, we delve into the feature reuse process of existing cache-based methods. As shown in Fig. 2, these methods typically assume a high degree of feature similarity between adjacent timesteps in the iterative denoising process, and achieve accelerated inference by sharing features across consecutive timesteps. However, our investigation reveals that while features in the same attention module (*e.g.*, spatial attention) appear to be nearly identical between adjacent timesteps, there exist some subtle yet discernible differences. As a result, a naive feature caching and reuse strategy often leads to degradation of details in generated videos, as shown in Fig. 3 (a).

Figure: Figure 3: (a) Vanilla cache-based methods typically lead to detail loss. (b) Time overhead proportions of different components in video models.
Refer to caption: x3.png

Based on the above insights, we propose a novel strategy, termed FasterCache, to accelerate the inference of video diffusion models while ensuring high-quality generation and remaining training-free. Specifically, we first introduce a dynamic feature reuse strategy for attention modules which dynamically adjusts the reused features across different timesteps, ensuring both distinction and continuity of features between adjacent timesteps are maintained. This strategy preserves the subtle variations essential for the iterative denoising process while ensuring temporal consistency, resulting in accelerated inference with minimal loss of details in the generated videos. Furthermore, we introduce CFG-Cache, an innovative technique that stores the residuals between conditional and unconditional outputs, dynamically enhancing their high-frequency and low-frequency components before reuse. This significantly accelerates inference while preserving details in generated videos.

We evaluate our FasterCache on various video diffusion models, including Open-Sora 1.2 (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), Latte Ma et al. (), CogVideoX (Yang et al., ), and Vchitect-2.0 (Vchitect, ). Experimental results demonstrate that FasterCache can significantly accelerate inference while preserving high-quality video generation across all tested models. Specifically, on Vchitect-2.0, FasterCache achieves 1.67$\times$ speedup, with performance comparable to the baseline (VBench: baseline 80.80% $\rightarrow$ FasterCache 80.84%). Furthermore, our method outperforms existing approaches in both inference speed and video generation quality, highlighting its effectiveness and efficiency in real-world applications.

Overall, the contributions of this work are as follows:

- •
We analyze the feature reuse process in cache-based methods and discover that while adjacent-step features in attention modules appear to be similar, their subtle differences can degrade output quality if ignored.
- •
We conduct a pioneering investigation of CFG’s potential for acceleration, finding high redundancy within the same timestep but weaker similarity across adjacent timesteps, revealing new acceleration opportunities.
- •
We propose FasterCache, a training-free strategy that dynamically adjusts feature reuse, preserving both feature distinction and continuity. It also introduces CFG-Cache to accelerate inference while preserving details in generated videos.
- •
We empirically evaluate our approach on various video diffusion models, demonstrating significant improvement in inference speed while maintaining high video quality.

## 2 Methodology

### 2.1 Preliminary

Diffusion model is a generative model consisting of a forward process and a reverse process.
Specifically, its forward diffusion process progressively adds noise to the data $\bm{x}_{0}\sim p_{data}(\bm{x}_{0})$, eventually destroying the signal. This can be formulated as:

$$ $\displaystyle q(\bm{x}_{t}|\bm{x}_{0})=\mathcal{N}(\bm{x}_{t};\sqrt{\alpha_{t} }\bm{x}_{0},\sqrt{1-\alpha_{t}}\bm{I}),$ (1) $$

where $\{\alpha_{t}\}^{T}_{t=1}$ controls the noise schedules and T represents the total number of diffusion timesteps.
The reverse process is typically parameterized as a UNet or transformer architecture $\bm{\epsilon}_{\theta}$ which is trained to predict the noise with the following loss function:

$$ $\displaystyle\mathcal{L}_{DM}=\mathbb{E}_{\bm{x},\bm{\epsilon}\sim\mathcal{N}( 0,1),t}[||\bm{\epsilon}-\bm{\epsilon}_{\theta}(\bm{x}_{t},t)||^{2}_{2}].$ (2) $$

A clean signal $\bm{x}_{0}$ can be recovered through iterative inference steps which predict $\bm{x}_{t-1}$ from $\bm{x}_{t}$ using $\bm{\epsilon}_{\theta}$. This can formulated as:

$$ $\displaystyle p(\bm{x}_{t-1}|\bm{x}_{t})=\mathcal{N}(\bm{x}_{t-1};\mu_{\theta} (\bm{x}_{t},t),\Sigma_{\theta}(\bm{x}_{t},t)),$ (3) $$

where $\mu_{\theta}$ and $\Sigma_{\theta}$ are the mean and variance parameterized with learnable $\theta$.

Video diffusion models recently employ diffusion transformers as the backbone for noise prediction. This work explores video synthesis acceleration based on Open-Sora 1.2 (Zheng et al., ). This model is composed of 56 stacked transformer layers, with alternating spatial and temporal layers. Each layer contains not only a spatial or temporal attention module but also a cross-attention and a feed-forward network.
Latte (Ma et al., ) and Open-Sora-Plan (Lab & etc., ) also adopt a similar architecture as their noise prediction networks.

Classifier-Free Guidance (CFG) has proven to be a powerful technique for enhancing the quality of synthesized images/videos in diffusion models. During the sampling process, CFG computes two outputs, namely $\bm{\epsilon}_{\theta}(\bm{x}_{t},\bm{c})$ for the conditional input $\bm{c}$ and $\bm{\epsilon}_{\theta}(\bm{z}_{t},\emptyset)$ for the unconditional input $\emptyset$ (often an empty or negative prompt).
The final output is given by:

$$ $\displaystyle\tilde{\bm{\epsilon}}_{\theta}(\bm{x}_{t},\bm{c})=(1+g)\bm{ \epsilon}_{\theta}(\bm{x}_{t},\bm{c})-g\bm{\epsilon}_{\theta}(\bm{z}_{t}, \emptyset),$ (4) $$

where $g$ is the guidance scale. As shown in Fig. 3 (b), while CFG significantly enhances visual quality, it also increases computational cost and inference latency due to the additional computation required for unconditional outputs.

### 2.2 Rethinking Attention Feature Reuse

Figure: Figure 4: Comparison of the mean squared error (MSE) of attention features between the current and previous diffusion steps. Smaller values indicate higher similarity.
Refer to caption: x4.png

Figure: Figure 5: Visual quality degradation caused by Vanilla Feature Reuse (left) and feature differences between adjacent timesteps (right).
Refer to caption: x5.png

To investigate the underlying causes of this phenomenon, we subsequently visualize the attention features between adjacent timesteps and analyze their differences. The results indicate that while the attention features between adjacent timesteps are highly similar, there exist noticeable differences between them. These subtle variations between timesteps are essential for preserving fine details in video generation. Therefore, directly reusing features without accounting for these differences leads to the loss of important visual information, resulting in smoother but less detailed outputs. This highlights the need for a more refined approach to feature reuse, *i.e.*, one that can retain computational efficiency while preserving key inter-step variations.

### 2.3 Feature Redundancy in CFG

Figure: Figure 6: (a) The MSE between conditional and unconditional outputs at the same timestep as well as across adjacent timesteps. (b) Directly reusing unconditional outputs from previous timesteps will lead to a significantly degraded visual quality.
Refer to caption: x6.png

Following the observation of feature redundancy in attention modules across adjacent timesteps, we further extend our investigation into other critical components of the diffusion models. Through this broader analysis of the entire denoising process, we find that classifier-free guidance (CFG) significantly increases inference time, as it requires the computation of both conditional and unconditional outputs at every timestep. While CFG has been widely adopted for enhancing visual quality, there is little exploration to reduce its computational burden, leaving this aspect largely uncharted.

To explore potential redundancy within CFG, we first conduct a quantitative analysis of the similarity between conditional and unconditional outputs at the same timestep as well as across adjacent timesteps based on mean squared error (MSE). As shown in Fig. 6 (a), the results reveal that, in the mid to later stages of sampling, the similarity between conditional and unconditional outputs at the same timestep is remarkably high, significantly surpassing that of adjacent steps. Hence, as illustrated in Fig. 6 (b), directly reusing unconditional outputs from adjacent timesteps, as suggested in existing methods, leads to significant error accumulation, resulting in a decline in video quality. These results indicate substantial redundancy in the CFG process and highlight the necessity for a new strategy to accelerate CFG without compromising the quality of the generated outputs.

Figure: Figure 7: (a) Simply reusing the conditional output from the same time step results in the poor generation of intricate details. (b) Trend curves of high and low-frequency biases between conditional and unconditional outputs change as sampling progresses.
Refer to caption: x7.png

### 2.4 FasterCache for Video Diffusion model

Capitalizing on the above discoveries, we introduce an innovative approach, FasterCache, which accelerates inference for video diffusion models while preserving high-quality generation. This is accomplished through a Dynamic Feature Reuse Strategy that maintains feature distinction and temporal continuity. Furthermore, we introduce CFG-Cache to optimize the reuse of conditional and unconditional outputs, further enhancing inference speed without compromising visual quality.

Dynamic Feature Reuse Strategy As discussed in Section 2.2, vanilla attention feature reuse strategy neglects the feature differences between adjacent timesteps which leads to visual quality degradation. Hence, instead of directly reusing previously cached features at the current timestep, we propose a Dynamic Feature Reuse Strategy that can more effectively capture and preserve critical details in the generated videos. Specifically, for the attention modules in diffusion models, we compute the attention module outputs at every alternate timestep. For example, we calculate the attention outputs for each layer at $t+2$ and $t$ timesteps, denoted as $\bm{F}_{t+2}$ and $\bm{F}_{t}$, and store them in the feature cache as $\bm{F}_{cache}^{t+2}$ and $\bm{F}_{cache}^{t}$. To dynamically adjust feature reuse, we compute the difference between the adjacent cached features. This serves as a bias for approximating the feature variation trend and enables the reused features to more accurately capture the evolving details across timesteps. For the intermediate $t-1$ timestep, its features can be computed as:

$$ $\displaystyle\bm{F}_{t-1}=\bm{F}_{cache}^{t}+(\bm{F}_{cache}^{t}-\bm{F}_{cache }^{t+2})*w(t),$ (5) $$

where $w(t)$ is a weighting function that modulates the contribution of the feature difference to account for variation between adjacent timesteps, ensuring both efficiency and the preservation of fine details in the generated videos. In our experiments, $w(t)$ gradually increases as the sampling process progresses, allowing the model to place greater emphasis on the feature differences at later stages of generation.
Consequently, our approach significantly accelerates inference while preserving the visual quality of the synthesized videos.

CFG-Cache As analyzed in Section 2.3, the conditional and unconditional outputs at the same timestep exhibit high similarity in CFG, indicating significant information redundancy. A naive approach to take advantage of this would be to directly reuse the conditional features for the corresponding unconditional outputs at the same timestep. However, this often leads to a noticeable degradation in detail generation. As illustrated in Fig 7 (a), this approach results in poor generation of intricate details, such as the texture of the spacesuit which shows a lack of details and clarity. Since both the conditional and unconditional outputs in CFG represent predicted noise, and drawing inspiration from the Dynamic Feature Reuse Strategy and FreeU (Si et al., ), we analyze the differences between these two outputs in the frequency domain. As shown in Fig 7 (b), we observe that, in the early and mid-stages of the sampling process, the conditional and unconditional outputs at the same timestep exhibit a significant bias in the low-frequency components, which progressively shifts to the high-frequency components in the later steps. This observation suggests that despite their overall similarity, key differences in frequency components must be addressed to avoid the degradation of fine details.

Figure: Figure 8: Overview of the CFG-Cache.
Refer to caption: x8.png

Building on this discovery, we propose CFG-Cache, a novel approach designed to account for both high- and low-frequency biases, coupled with a timestep-adaptive enhancement technique. Specifically, as shown in Fig. 8, at timestep $t$, a full inference is performed to obtain both the conditional output $\epsilon_{\theta}(x_{t},t,c)$ and the unconditional output $\epsilon_{\theta}(x_{t},t,\emptyset)$. We then separately calculate the biases for the high-frequency ($\Delta_{HF}$) and low-frequency ($\Delta_{LF}$) components between these two outputs:

$$ $\displaystyle\Delta_{LF}=\mathcal{FFT}(\bm{\epsilon}_{\theta}(\bm{x}_{t},t, \emptyset))_{low}-\mathcal{FFT}(\bm{\epsilon}_{\theta}(\bm{x}_{t},t,\bm{c}))_{low}$ (6) $$

$$ $\displaystyle\Delta_{HF}=\mathcal{FFT}(\bm{\epsilon}_{\theta}(\bm{x}_{t},t, \emptyset))_{high}-\mathcal{FFT}(\bm{\epsilon}_{\theta}(\bm{x}_{t},t,\bm{c}))_ {high}.$ (7) $$

These biases ensure that both high- and low-frequency differences are accurately captured and compensated during the reuse process.
In the subsequent $n$ timesteps (from $t-1$ to $t-n$), we infer only the outputs of the conditional branches and compute the unconditional outputs using the cached $\Delta_{HF}$ and $\Delta_{LF}$ as follows:

$$ $\displaystyle\bm{\hat{\epsilon}}_{\theta}(\bm{x}_{t-i},t-i,\emptyset)$ $\displaystyle=\mathcal{IFFT}(\mathcal{F}_{low},\mathcal{F}_{high}),$ (8) $$

$$ $\displaystyle\mathcal{F}_{low}$ $\displaystyle=\quad\Delta_{LF}*w_{1}+\mathcal{FFT}(\bm{\epsilon}_{\theta}(\bm{ x}_{t-i},t-i,\bm{c}))_{low}$ (9) $\displaystyle\mathcal{F}_{high}$ $\displaystyle=\Delta_{HF}*w_{2}+\mathcal{FFT}(\bm{\epsilon}_{\theta}(\bm{x}_{t -i},t-i,\bm{c}))_{high}$ (10) $$

Here, $w_{1}$ and $w_{2}$ are adaptively adjusted based on the sampling timestep $t$, with greater emphasis on different frequency components at distinct sampling phases. The weighting scheme is defined as:

$$ $\displaystyle w_{1}=1+\alpha_{1}\cdot\mathbb{I}(t>t_{0}),w_{2}=1+\alpha_{2} \cdot\mathbb{I}(t<=t_{0}),$ (11) $$

where $\alpha_{1}$ and $\alpha_{2}$ are hyperparameter weights, $t_{0}$ is the manually set switching timestep, and $\mathbb{I}(\cdot)$ is the indicator function. This formulation ensures that mid-low frequencies are prioritized in the mid-sampling phase, while high-frequency components receive more attention in the later phase.

## 3 EXPERIMENTS

### 3.1 Experimental Settings

Base models and compared methods To demonstrate the effectiveness of our method, we apply our acceleration technique to various video diffusion models, including Open-Sora 1.2 (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), Latte (Ma et al., ), CogVideoX (Yang et al., ), and Vchitect-2.0 (Vchitect, ).
We compare our base models with recent efficient video synthesis techniques, including PAB (Zhao et al., ) and $\Delta$-DiT (Chen et al., ), to highlight the benefits of our approach. Notably, $\Delta$-DiT was originally designed as an acceleration method for image synthesis. Here we have adapted it for video synthesis to facilitate comparison.
Please refer to the Appendix for more details of the base models and compared methods.

Evaluation metrics and datasets To assess the performance of video synthesis acceleration methods, we focus primarily on two aspects, namely inference efficiency and visual quality.
To evaluate inference efficiency, we employ Multiply-Accumulate Operations (MACs) and inference latency as metrics.
We utilize VBench (Huang et al., ), LPIPS (Zhang et al., ), PSNR, and SSIM for visual quality evaluation. VBench is a comprehensive benchmark suite for video generative models. It is well-aligned with human perceptions and capable of providing valuable insights from multiple perspectives.
LPIPS, PSNR, and SSIM measure the similarity between videos generated by the accelerated sampling method and those from the original model.
PSNR quantifies pixel-level fidelity between outputs, LPIPS measures perceptual consistency, and SSIM assesses structural similarity. In general, higher similarity scores indicate better fidelity and visual quality.

Implementation details All experiments conduct full attention inference for spatial and temporal attention modules every 2 timesteps to facilitate dynamic feature reuse.
The weight $w(t)$ increases linearly from 0 to 1 starting from the beginning of dynamic feature reuse until the end of sampling.
For CFG output reuse, full inference is conducted every five timesteps, starting from $1/3$ of the total sampling steps (e.g., for Open-Sora 1.2, which has 30 total sampling steps, this begins at step 10).
The hyperparameters $\alpha_{1}$ and $\alpha_{2}$ are set to a default value of 0.2, which performs well for most models.
All experiments are carried out on NVIDIA A100 80GB GPUs using PyTorch, with FlashAttention (Dao et al., ) enabled by default.

**Table 1: Comparison of efficiency and visual quality on a single GPU.**
| Method | Efficiency | Visual Quality |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MACs (P) $\downarrow$ | Speedup $\uparrow$ | Latency (s) $\downarrow$ | VBench $\uparrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ |  |
| Open-Sora 1.2 (192 frames, 480P) |  |  |  |  |  |  |  |
| Open-Sora 1.2 ($T=30$) | 6.30 | $1\times$ | 192.07 | 78.79% | - | - | - |
| $\Delta$-DiT ($N_{c}=14,N=2$) | 5.51 | $1.14\times$ | 168.69 | 77.43% | 0.2834 | 0.7403 | 17.77 |
| $\Delta$-DiT ($N_{c}=28,N=2$) | 4.72 | $1.34\times$ | 143.14 | 76.60% | 0.3321 | 0.7092 | 16.24 |
| PAB | 5.33 | $1.23\times$ | 156.73 | 78.15% | 0.1041 | 0.8821 | 26.43 |
| Ours | 4.13 | 1.62$\times$ | 118.44 | 78.46% | 0.0835 | 0.8932 | 27.03 |
| Open-Sora-Plan (65 frames, 512$\times$512) |  |  |  |  |  |  |  |
| Open-Sora-Plan ($T=150$) | 10.30 | $1\times$ | 103.76 | 80.16% | - | - | - |
| $\Delta$-DiT ($N_{c}=14,N=3$) | 8.60 | $1.19\times$ | 86.88 | 78.12% | 0.4515 | 0.4813 | 16.08 |
| $\Delta$-DiT ($N_{c}=28,N=3$) | 6.90 | $1.46\times$ | 70.99 | 77.71% | 0.4819 | 0.4467 | 15.42 |
| PAB | 7.39 | $1.32\times$ | 78.72 | 80.06% | 0.2423 | 0.7126 | 20.29 |
| Ours | 5.51 | 1.68$\times$ | 61.68 | 80.19% | 0.1348 | 0.8138 | 23.72 |
| Latte (16 frames, 512$\times$512) |  |  |  |  |  |  |  |
| Latte ($T=50$) | 3.05 | $1\times$ | 29.22 | 77.05% | - | - | - |
| $\Delta$-DiT ($N_{c}=14,N=2$) | 2.67 | $1.23\times$ | 23.80 | 76.27% | 0.1731 | 0.8107 | 22.69 |
| $\Delta$-DiT ($N_{c}=28,N=2$) | 2.29 | $1.43\times$ | 20.38 | 76.01% | 0.2245 | 0.7620 | 21.00 |
| PAB | 2.24 | $1.28\times$ | 22.84 | 76.70% | 0.2904 | 0.7083 | 18.98 |
| Ours | 1.97 | 1.54$\times$ | 18.98 | 76.89% | 0.0817 | 0.8948 | 28.21 |
| CogVideoX (48 frames, 480P) |  |  |  |  |  |  |  |
| CogVideoX ($T=50$) | 6.03 | 1$\times$ | 78.48 | 80.18% | - | - | - |
| $\Delta$-DiT ($N_{c}=4,N=2$) | 5.62 | 1.08$\times$ | 72.72 | 79.61% | 0.3319 | 0.6612 | 17.93 |
| $\Delta$-DiT ($N_{c}=8,N=2$) | 5.23 | 1.15$\times$ | 68.19 | 79.31% | 0.3822 | 0.6277 | 16.69 |
| $\Delta$-DiT ($N_{c}=12,N=2$) | 4.82 | 1.26$\times$ | 62.50 | 79.09% | 0.4053 | 0.6126 | 16.15 |
| Ours | 3.71 | 1.62$\times$ | 48.44 | 79.83% | 0.0766 | 0.9066 | 28.93 |
| Vchitect-2.0 (40 frames, 480P) |  |  |  |  |  |  |  |
| Vchitect-2.0 ($T=100$) | 14.57 | 1$\times$ | 260.32 | 80.80% | - | - | - |
| $\Delta$-DiT ($N_{c}=6,N=3$) | 13.00 | 1.11$\times$ | 233.59 | 79.98% | 0.4153 | 0.5837 | 14.26 |
| $\Delta$-DiT ($N_{c}=12,N=3$) | 11.79 | 1.24$\times$ | 209.78 | 79.50% | 0.4534 | 0.5519 | 13.68 |
| Ours | 8.67 | 1.67$\times$ | 156.13 | 80.84% | 0.0282 | 0.9224 | 31.45 |

### 3.2 Main Results

Quantitative comparison Table 1 presents a quantitative comparison of our method with $\Delta$-DiT and PAB in terms of efficiency and visual quality.
We synthesize videos with prompts provided by VBench and use the synthesized videos to compute the VBench metrics as well as calculate LPIPS, SSIM, and PSNR with videos sampled by the original model.
The results demonstrate that our method achieves stable acceleration efficiency and superior visual quality across different base models, sampling schedulers, video resolutions, and lengths.

Figure: Figure 9: Comparison of visual result quality among different methods.
Refer to caption: x9.png

Visual quality comparison Fig. 9 compares the videos generated by our method against those by the original model, PAB, and $\Delta$-DiT. The results demonstrate that our method can effectively preserve the original quality and fine details. More visual results can be found in the Appendix.

### 3.3 Ablation Study

To comprehensively assess the effectiveness and efficiency of our method, we perform extensive ablation studies based on Open-Sora, synthesizing videos of 48 frames at 480P.

**Table 2: Impact on inference efficiency.**
| Variants | MACs (P) | Latency (s) | $\Delta$ (s) |  |  |
| --- | --- | --- | --- | --- | --- |
| Vanilla FR | Dynamic FR | CFG-Cache |  |  |  |
|  |  |  | 1.54 | 41.28 | - |
| ✓ |  |  | 1.33 | 33.25 | -8.03 |
|  | ✓ |  | 1.33 | 33.50 | -7.78 |
|  |  | ✓ | 1.16 | 31.32 | -9.96 |
|  | ✓ | ✓ | 1.01 | 26.12 | -15.16 |

Efficiency Table 2 compares the efficiency of the original Open-Sora and its variants with different acceleration components. There are two key observations. (1) The Dynamic Feature Reuse Strategy and CFG-Cache independently contribute to significant reductions in inference costs. When combined, they further minimize inference overhead.
(2) Compared to Vanilla Feature Reuse, the proposed Dynamic Feature Reuse strategy has a negligible impact on efficiency.

Visual quality Table 4 compares the visual quality of the original Open-Sora with its variants implementing different acceleration components. Note that vanilla feature reuse leads to a performance drop in VBench and LPIPS.
The introduction of the dynamic feature reuse strategy mitigates the loss of information and thereby improves the performance of these metrics (*e.g.*, VBench: 78.34% $\rightarrow$ 78.69%).
Fig. 10 (a) provides a visual comparison of the results.
It can be observed that vanilla feature reuse shows reduced details (*e.g.*, the moon and snowflakes), whereas dynamic feature reuse strategy can significantly alleviate this problem.
The Feature MSE curves show that adding the bias term can lower the MSE between intermediate features from the original and accelerated sampling process, aligning with the visual results.

**Table 3: Impact on visual quality.**
| Variants | VBench | LPIPS | SSIM | PSNR |
| --- | --- | --- | --- | --- |
| Original Open-Sora | 78.99% | - | - | - |
| Vanilla FR | 78.34% | 0.0657 | 28.20 | 0.8785 |
| Full (w/ Dynamic FR) | 78.69% | 0.0590 | 28.41 | 0.8938 |
| CFG-Cache w/o Enhancement | 78.42% | 0.0709 | 27.97 | 0.8727 |
| Enhance LF only | 78.58% | 0.0617 | 28.29 | 0.8894 |
| Enhance HF only | 78.49% | 0.0686 | 28.08 | 0.8834 |
| Full (w/ full CFG-Cache) | 78.69% | 0.0590 | 28.41 | 0.8938 |

Figure: Figure 10: Comparison of Feature MSE curves and visual results from the ablation study.
Refer to caption: x10.png

Referring to Table 4, it can be seen that introducing CFG-Cache without enhancement reduces the visual quality.
On the other hand, CFG-Cache with dynamic enhancement of either the low- or high-frequency bias helps to improve the visual quality, and their combined effect achieves the best visual quality.
Fig. 10 (b) shows that enhancing low-frequency bias improves the fidelity of low-frequency components (e.g., clouds, tornado outlines) while enhancing high-frequency bias enriches high-frequency details (e.g., lightning).
The Feature MSE curve of CFG-Cache without enhancement aligns with the reduced visual quality. Dynamic enhancement helps to mitigate error accumulation, leading to higher visual fidelity.

### 3.4 Scalability and Generalization

Scaling to multiple GPUs To evaluate the sampling efficiency of our method on multiple GPUs, we adopt the approach used in PAB and integrate Dynamic Sequence Parallelism (DSP) (Zhao et al., ) to distribute the workload across GPUs.
Table 4 illustrates that, as the number of GPUs increases, our method consistently enhances inference speed across different base models, surpassing the performance of the compared methods.

Performance at different resolutions and lengths To evaluate the effectiveness of our method in accelerating sampling for videos of varying sizes, we conduct tests across different video lengths and resolutions and report the results in Fig. 11. Our method maintains stable acceleration performance when faced with increasing resolutions and frame counts in videos, demonstrating its potential to accelerate sampling longer and higher-resolution videos in line with practical demands.

Figure: Figure 11: Acceleration efficiency of our method at different video resolutions and lengths.
Refer to caption: x11.png

I2V and image synthesis performance We integrate our acceleration method to the state-of-the-art image-to-video model DynamiCrafter (Xing et al., ) and image synthesis model PixArt-sigma (Chen et al., ). As shown in Fig. 12, our method significantly accelerates sampling while maintaining visual fidelity, demonstrating its potential for extension to various base models.

Figure: Figure 12: Visual results and inference time of our method on I2V and image synthesis models.
Refer to caption: x12.png

## 4 RELATED WORK

### 4.1 Diffusion Models for Video Synthesis

Diffusion models have demonstrated potential in high-quality image synthesis (Ho et al., ; Rombach et al., ; Chen et al., ; ), attracting significant attention. Subsequent work has adapted these models for video synthesis to generate high-fidelity videos (Ho et al., ).
Motivated by advancements in image synthesis, early studies typically employed the diffusion UNet architecture (Blattmann et al., ; Wang et al., ; Zhang et al., ; Wu et al., ).
As the scalability of diffusion transformer (Peebles & Xie, ) was validated in image synthesis, an increasing number of works have adopted the diffusion transformer as the noise estimation network (Ma et al., ; Zheng et al., ; Lab & etc., ; Yang et al., ).

### 4.2 Efficiency Improvements in Diffusion Models

Despite the impressive performance of diffusion models in image and video synthesis, their substantial inference cost limits their practicality. Prior research on improving the efficiency of diffusion models has primarily focused on two perspectives, namely reducing the number of sampling steps and lowering the inference cost per sampling step.
Regarding the reduction of sampling steps, most approaches achieve high-quality samples with fewer sampling steps by employing efficient SDE or ODE solvers (Song et al., ; Lu et al., ; ).
Other methods reduce the number of sampling steps by progressively distilling the model (Salimans & Ho, ; Sauer et al., ; Lin & Yang, ; Li et al., ) or employing consistency models (Luo et al., ; Song et al., ).

More works have focused on reducing the inference cost per timestep.
Some approaches improve network efficiency through pruning (Zhang et al., ) or quantization (Shang et al., ; So et al., ; He et al., ; Li et al., ; Sui et al., ; Zhao et al., ), while others obtain more lightweight network architectures through search techniques (Li et al., ; Yang et al., ).
However, these methods often require additional computational resources for fine-tuning or optimization.
Some training-free approaches (Bolya & Hoffman, ; Wang et al., ) focus on the input tokens, accelerating the sampling process by reducing the number of tokens to be processed by eliminating token redundancy in image synthesis.
Other methods reuse intermediate features between adjacent sampling timesteps, avoiding redundant computations (Wimbauer et al., ; So et al., ).
TGATE (Zhang et al., ) accelerates image generation by caching and reusing attention outputs at scheduled timesteps.
DeepCache (Ma et al., ) and Faster Diffusion (Li et al., ) employ a feature caching mechanism to indirectly alter the UNet diffusion for acceleration. $\Delta$-DiT (Chen et al., ) adapts this mechanism to the diffusion transformer architecture by caching the residuals between attention layers. PAB (Zhao et al., ) caches and broadcasts intermediate features at different timestep intervals based on the characteristics of varying attention blocks.
Although these methods have achieved some improvements in diffusion efficiency, the efficiency enhancements for diffusion transformers in video synthesis remain insufficient.

## 5 CONCLUSION AND DISCUSSION

In this work, we present FasterCache, a training-free strategy that significantly accelerates video synthesis inference while preserving high-quality generation.
Through analysis of existing cache-based methods, we find that directly reusing adjacent-step features in attention modules can degrade video quality. Additionally, we investigate the acceleration potential of CFG, identifying redundancy between conditional and unconditional features at the same timestep.
Leveraging these insights, FasterCache integrates a dynamic feature reuse strategy that maintains feature distinction and temporal continuity, and CFG-Cache which optimizes the reuse of conditional and unconditional outputs to further boost speed without sacrificing detail quality.
Extensive experiments demonstrate its strong performance in both efficiency and synthesis quality across diverse video models, sampling schedules, video lengths and resolutions, highlighting its potential for real-world applications.

Limitation Despite the effectiveness shown by our method, certain limitations remain. When the synthesis quality of the model is suboptimal, our acceleration method is unlikely to yield satisfactory results either. We believe that advancements in base video models will mitigate this issue. Additionally, in complex scenes with substantial video motion, our method may occasionally produce degraded results. At present, this can be remedied through manual adjustments of hyperparameters. In the future, we plan to investigate strategies for adaptive caching to further enhance performance.

## Appendix A Appendix

### A.1 Further Details of base models

In this work, we applied our FasterCache to various video synthesis models, including Open-Sora 1.2 (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), Latte (Ma et al., ), CogVideoX (Yang et al., ), and Vchitect 2.0 Vchitect (). Open-Sora 1.2 (Zheng et al., ) integrates 2D-VAE and 3D-VAE to enhance video compression and employs ST-DiT blocks for the diffusion process.
Open-Sora-Plan (Lab & etc., )
adopts CausalVideoVAE to compress visual representations better and 3D full attention architecture to capture joint spatial and temporal features.
Latte Ma et al. () extracts spatio-temporal tokens from input videos and then adopts a series of transformer blocks to model video distribution in the latent space.
CogVideoX (Yang et al., ) employs a 3D VAE to compress videos along spatial and temporal dimensions and an expert transformer with the expert adaptive LayerNorm to facilitate the fusion between the two modalities.

### A.2 Further Details of compared methods

PAB (Zhao et al., ) employs a pyramid-style broadcasting mechanism to propagate attention outputs across subsequent steps. It optimizes efficiency by applying distinct broadcast strategies to each attention layer based on their respective variances. Additionally, the method introduces broadcast sequence parallelism to enhance the efficiency of distributed inference. This paper follows the default parameter configuration of PAB.

$\Delta$-DiT (Chen et al., ) accelerates inference by caching feature offsets instead of the full feature maps while preventing input information loss.
It caches the residuals of the blocks in the latter part of DiT for approximation during early-stage sampling and caches the residuals of the blocks in the earlier part during later-stage sampling.
In $\Delta$-DiT, the parameters that need to be configured are the residual cache interval $N$, the number of cached blocks $N_{c}$, and the timestep boundary $b$ for determining the position of the cached blocks.
Since the source code of $\Delta$-DiT is not publicly available, we implemented its method based on the paper for accelerating video synthesis.
Following the guidelines in $\Delta$-DiT, we experimented with different configurations of $N_{c}$ and $N$ to balance visual quality and inference speed, allowing for a fair evaluation of the method.

### A.3 Additional Qualitative Experiments

More visual results on Text-to-Video models The additional visual comparison results for Open-Sora 1.2 (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), and Latte (Ma et al., ) are presented in Fig. 13, Fig. 14, and Fig. 15, while further comparisons for CogVideoX (Yang et al., ) and Vchitect-2.0 Vchitect () are shown in Fig. 16. Our method demonstrates reliable fidelity across various models and styles or content in video synthesis, while simultaneously achieving acceleration.

More visual results on Image-to-Video models We conducted image-to-video sampling acceleration experiments based on DynamiCrafter (Xing et al., ), achieving a 1.52$\times$ speedup on a single GPU. Additional visual results are provided in Fig. 17. Our method demonstrates good fidelity in the acceleration of image-to-video models, indicating broad potential for practical applications.

Figure: Figure 13: More visual results on Open-Sora (480P 192 frames). Zoom in for details.
Refer to caption: x13.png

Figure: Figure 14: More visual results on Open-Sora-Plan (512$\times$512 65 frames). Zoom in for details.
Refer to caption: x14.png

Figure: Figure 15: More visual results on Latte (512$\times$512 16 frames). Zoom in for details.
Refer to caption: x15.png

Figure: Figure 16: More visual results on CogVideoX (480P 48 frames) & Vchitect-2.0 (480P 40frames).
Refer to caption: x16.png

Figure: Figure 17: More visual results on DynamiCrafter (1024$\times$576 16frames). Zoom in for details.
Refer to caption: x17.png

### A.4 Additional Quantitative Experiments

User preference study To assess the effectiveness of our FasterCache, we additionally conduct a human evaluation. We randomly selected 30 videos for each model. Each rater receives a text prompt and two generated videos from different sampling acceleration methods (in random order). They are then asked to select the video with better visual quality. Five raters evaluate each sample, and the voting results are summarized in Table 5. As one can see, compared to other acceleration methods, the raters strongly prefer the videos generated by our method.

**Table 5: User preference study. The numbers represent the percentage of raters who favor the videos synthesized by our method.**
| Method comparison | Open-Sora 1.2 | Open-Sora-Plan | Latte |
| --- | --- | --- | --- |
| Ours vs. $\Delta$-DiT | 80.67% | 78.00% | 77.33% |
| Ours vs. PAB | 69.33% | 72.67% | 74.00% |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. For CFG-Cache, should we interpret the feature differences in the frequency domain in a similar way as MSE? 

2. In Figure 7b, it seems to me that low-frequency components do not gradually shift into high-freq as the authors mention in L300-301, since the high-freq components differ both at the beginning and near the end. What kind of insights can we gain from this analysis?

3. In the experiments, PAB seems to significantly outperform $\Delta-DiT$, but there are no PAB results for CogVideoX and Vchitect. Also, why is the performance of $\Delta-DiT$ so low? Is it because of the inherent randomness of image diffusion models?
soundness: 3
strengths: 1. The paper clearly demonstrates the motivation and the problem of existing works with illustrative examples (Fig. 3-7). Overall, the limitations of vanilla cache-based acceleration methods with CFG is analyzed very thoroughly.

2. Qualitative results demonstrate notable improvements in visual details and the spatio-temporal consistency of the generated frames.
summary: This paper proposes a training-free feature caching strategy for video diffusion models, named FasterCache, that improves how features are reused for accelerating inference. Instead of directly reusing the adjacent-step features as in previous works, FasterCache interpolates the neighboring features to efficiently synthesize features for the missing steps. In addition, the paper introduces CFG-Cache, which aims to effectively reuse the conditional outputs for computing the unconditional outputs. Experimental results demonstrate notable visual quality and efficiency improvements compared with the existing methods.
weaknesses: 1. Slightly missing justifications for the design choices of the proposed method. For instance, is the linear interpolation strategy sufficient for dynamically building the missing features? Are there any other options that the authors have considered? Using the proposed Dynamic Feature Reuse, how similar are the computed features compared to the existing caching methods? (Please guide me if I missed the descriptions)

2. Limited video-specific contributions. The proposed methodologies seem to applicable to any image diffusion model, of which comparison to previous works is missing, and I could not find any video-related novelties proposed. The notation $t = \{1 ... T\}$ in Equations seem to denote the diffusion sampling steps, and I think there is no notation for the temporal axis? Please correct me if I misunderstood any equations.

3. (Minor) Figure captions are not very descriptive. I would suggest to include the main points (Fig. 2, 8) or where to focus on (qualitative figures).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions:
1. Is the effectiveness of the Dynamic Feature Reuse Strategy because the videos generated by existing video generation methods are relatively smooth, with little difference between adjacent frames, e.g. fixed backgrounds? If so, once the motion of the video is large and the adjacent frames change greatly, then is the proposed method still effective?

2. Negative prompt will affect the quality of video generation, which generally improves it. So will CFG-Cache and the separation of high and low frequencies have an impact on negative prompt setting?

3. For Open-Sora-Plan, although FasterCache achieved an acceleration of 1.68x, the generated video has significant varying in pixel domain compared to the original one (PSNR 23.72). What is the reason for this bias? Blurriness, noise, or changes in semantic level?

4. Will the setting of g value in CFG formula 4 (Line 167) affect the performance of FasterCache?
soundness: 2
strengths: In terms of originality, the proposed FasterCache method is highly innovative. It combines a dynamic feature reuse strategy that considers subtle differences between adjacent timesteps and a CFG - Cache component for handling conditional and unconditional outputs in a novel way. This is a departure from existing methods and reveals new acceleration opportunities through a pioneering investigation of classifier - free guidance (CFG). 

Regarding quality, a thorough analysis of existing cache - based methods and CFG is provided. This includes identifying problems and limitations, and conducting in - depth investigations into feature reuse and CFG outputs, which forms a solid foundation for the method. The experimental design is robust, with testing on multiple video diffusion models and the use of various evaluation metrics. Ablation studies further demonstrate the effectiveness of each component. 

For clarity, the paper has a clear structure, starting with an introduction to the problem and existing solutions, followed by a detailed description of the method in the methodology section. The experimental results are presented clearly, and the discussion and conclusions are well - organized. Complex concepts are explained accessibly, such as through diagrams and detailed descriptions of the key components. 

In terms of significance, the research has practical implications as it can reduce the time and computational resources required for video generation, which is important for applications like video content creation, virtual reality, and augmented reality. It also advances the field by filling gaps in existing research on video diffusion models, addressing limitations of current acceleration methods, and inspiring further studies on improving efficiency and performance.
summary: This paper presents an innovative approach to accelerate video diffusion models without sacrificing quality. By analyzing existing cache - based methods and classifier - free guidance (CFG), the authors identify limitations and redundancies. They introduce FasterCache, which includes a dynamic feature reuse strategy for attention modules and CFG-Cache. The dynamic feature reuse strategy adjusts features across timesteps, maintaining distinction and continuity. CFG-Cache optimizes the reuse of conditional and unconditional outputs. Experimental results on multiple video diffusion models demonstrate significant acceleration while keeping video quality comparable to the baseline, outperforming existing methods.
weaknesses: Hyperparameter Explanation. Some of the hyperparameters used in the method, such as the weighting function are not explained in sufficient detail. Although default values are provided and it is mentioned that they work well for most models, a more in-depth discussion on how these hyperparameters are chosen and their impact on the performance for different datasets and models would make the method more reproducible and understandable.

Uncertainty in complex scenes. The paper mentions that in complex scenes with substantial video motion, the method may occasionally produce degraded results. This may limit the practical applicability of the method once video generation model baselines are imporved and complex video content is common. More discussions refer to the Question 1.

Weak theoretical underpinning: The paper focuses more on the empirical evaluation and practical implementation of the FasterCache method. Although the experimental results are strong, the theoretical foundation behind some of the proposed techniques could be strengthened. For example, a more in-depth analysis of why the dynamic feature reuse strategy and CFG-Cache work as expected in different scenarios from a theoretical perspective would add more credibility to the method.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: - The paper is well-written and easy to read.
- The proposed attention feature map dynamics could be a generic method and applied into all other cache based efficient generation methods.
- the proposed frequency residual method for CFG is novel and seems also a generic method that can be applied into CFG for all diffusion based image/video generation models.
summary: This paper propose a training-free method for efficient video generation. Their method is based on the attention feature map and output redundancy observed in the forward pass of the denoising process. To this end, they propose to (1) use the feature map dynamics to restore the missing details in the previous naive feature map reuse method. (2) use the frequency difference to compensate the cached conditional output in CFG. The proposed method has been validated with several models and showed promising results.
weaknesses: - Distillation based efficient generation methods are less discussed and not added into the comparison. I can understand that this work mainly focused on the cache based method. But:
  - What's the delta of this method compared to distillation based method?
  - Are they compatible to each other? If not, in real world applications, which method we need to use?

- the overall method seems to perform better than all other methods. What's the contribution of each individual method? What the cost of additional computation in each individual method?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. what is the reason of using equation (5) for feature reuse? it seems to be quite heuristic.
2. why in cfg, the similarity of two stages has such a wired characteristics (time-varing low and high frequency similarity)? there should be more analysis and insight.
3. in terms of saving the computation of unconditional stage, how about existing and well-adopted methods such as "On Distillation of Guided Diffusion Models"? 
4. how does your method work with video generation models using flow matching?
soundness: 3
strengths: this paper is relatively well organized and written.
summary: This paper presents a method to accelerate the computation for video diffusion models. Based on the observation that adjacent-time features exhibit subtle differences,  the authors propose a dynamic feature reuse scheme. Based on the observation that the conditional and unconditional generation features have strong similarity in CFG, the authors propose a CFG-Cache scheme to save computations of unconditional stage.
weaknesses: this paper lacks novelty and insight. the proposed method is not solid in terms of logical and theoretical justification. the speedup of computation could be achieved using other methods.

Meta-review: Summary: This work proposes a dynamic feature-caching mechanism for inference-time video diffusion sampling. Unlike straightforward re-use of adjacent-time features, the method interpolates features of neighboring time steps in a weighted manner. Additionally, the authors identify and address a redundancy in classifier-free guidance (CFG), where conditional and unconditional features at the same time step overlap. They propose reusing conditional outputs to compute unconditional outputs, reducing redundancy. Experimental results demonstrate notable improvements in image quality and runtime performance.
Strengths: The main contribution is well-motivated, clearly illustrated, and supported by both qualitative and quantitative analyses. The work is highly relevant for accelerating video sampling, a process often hindered by computational expense.
Weaknesses: The contribution is somewhat narrow, focusing specifically on image/video diffusion sampling. The paper lacks a comparison with distillation-based methods that train smaller networks for similar acceleration goals.
Acceptance Reasoning:The proposed ideas, while simple, are highly relevant for speeding up diffusion sampling processes. The authors have sufficiently addressed most weaknesses raised by reviewers, leading to rating increases from some. Although not all reviewers responded after the rebuttal, the authors provided comprehensive responses to the concerns, addressing them effectively.

Decision: Accept (Poster)