Real-Time Video Generation with Pyramid Attention Broadcast

## Abstract

Abstract We present Pyramid Attention Broadcast (PAB), a real-time , high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference.
PAB demonstrates superior results across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.

## 1 Introduction

Sora (Brooks et al., ) kicks off the door of DiT-based video generation (Peebles & Xie, ). Recent approaches (Ma et al., ; Zheng et al., ; Lab & etc., ) demonstrate their superiority compared to CNN-based methods (Blattmann et al., ; Wang et al., ) especially in generated video quality.
However, this improved quality comes from significant costs, i.e., more memory occupancy, computation, and inference time.
Therefore, exploring an efficient
approach for DiT-based video generation becomes urgent for broader GenAI applications (Kumar & Kapoor, ; Othman, ; Meli et al., ).

Model compression methods employ techniques such as distillation (Crowley et al., ; Hsieh et al., ), pruning (Han et al., ; Ma et al., ), quantization (Banner et al., ; Lin et al., ), and novel architectures (Lin et al., ) to speedup deep learning models and have achieved remarkably success.
Recently, they have also been proven to be effective on diffusion models (Sauer et al., ; Ma et al., ; Chen et al., ).
Nevertheless, these methods usually require additional training with non-negligible computational resources and datasets, which makes model compression prohibitive and impractical especially for large-scale pre-trained models.

Most recently, people revisit the idea of cache (Smith, ; Goodman, ; Albonesi, ) to speedup diffusion models.
Different from model compression methods, model caching methods are training-free.
They alleviate redundancy by caching and reusing partial network outputs, thereby eliminating additional training. Some studies utilize high-level convolutional features for reusing purposes (Ma et al., ) and efficient distributed inference (Li et al., ; Wang et al., ). Similar strategies have also been extended to specific attentions (Zhang et al., ; Wimbauer et al., ), i.e., cross attention, and standard transformers (Chen et al., ).

However, training-free speedup methods for DiT-based video generation still remains unexplored. Besides, previous model caching methods are not directly applicable to video DiTs due to two intrinsic differences:
i) Different architecture. The model architecture has shifted from convolutional networks (Ronneberger et al., ) to transformers (Vaswani et al., ). This transaction makes former techniques that aims at convolutional networks not applicable to video generation anymore.
ii) Different components. Video generation relies on three diverse attention mechanisms: spatial, temporal, and cross attention (Blattmann et al., ; Ma et al., ). Such components lead to more complex dependency and attention interactions, making simple strategies ineffective. They also increase the time consumed by attentions, making attentions more critical than before.

Figure: Figure 2: Comparison of the attention outputs differences between the current and previous diffusion steps. Differences are measured by Mean Square Error (MSE) and averaged across all layers for each diffusion step.
Refer to caption: x2.png

To address these challenges, we take a closer look at attentions in video DiTs and empirically obtain two observation as shown in Figure 2:
(i) The attention differences between adjacent diffusion steps exhibit a U-shaped pattern, with stability in the middle 70% steps, indicating considerable redundancy for attention.
(ii) Within the stable middle segment, different attentions also demonstrate varying degrees of difference. Spatial attention varies the most with high-frequency visual elements, temporal attention shows mid-frequency variations related to movements, and cross-modal attention remains the most stable, linking text with video content (Zhang et al., ).

Based on these observations, we propose Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free method for efficient DiT-based video generation.
Our method mitigates attention redundancy by broadcasting the attention outputs
to subsequent steps, thus eliminating attention computation in the diffusion process.
Specifically, we apply varied broadcast ranges for different attentions in a pyramid style, based on their stability and differences as shown in Figure 2.
We empirically find that such broadcast strategy can also work well to MLP layers and diffusion steps.
Additionally, to enable efficient distributed inference, we propose broadcast sequence parallel, which significantly decreases generation time with much lower communication costs.

In summary, to the best of our knowledge, PAB is the first approach that achieves real-time video generation, reaching up to 20.6 FPS with a 10.5$\times$ acceleration without compromising quality. It consistently delivers excellent and stable speedup across popular open-source video DiTs, including Open-Sora (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), and Latte (Ma et al., ). Notably, as a training-free and generalized approach, PAB has the potential to empower any future video DiTs with real-time capabilities.

## 2 How to Achieve Real-Time Video Generation

### 2.1 Preliminaries

#### Denoising diffusion models.

Diffusion models are inspired by the physical process where particles spread out over time due to random motion, which consists of forward and reverse diffusion processes.
The forward diffusion process gradually adds noise to the data over $T$ steps. Starting with data $\mathbf{x}_{0}$ from a distribution $q(\mathbf{x})$, noise is added at each step:

$$ $\mathbf{x}_{t}=\sqrt{\alpha_{t}}\mathbf{x}_{t-1}+\sqrt{1-\alpha_{t}}\mathbf{z} _{t}\quad\text{for}\quad t=1,\ldots,T,$ (1) $$

where $\alpha_{t}$ controls the noise level and $\mathbf{z}_{t}\sim\mathcal{N}(0,\mathbf{I})$ is Gaussian noise. As $t$ increases, $\mathbf{x}_{t}$ becomes noisier, eventually approximating a normal distribution $\mathcal{N}(0,\mathbf{I})$ when $t=T$.
The reverse diffusion process aims to recover the original data from the noisy version:

$$ $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};\mu_{ \theta}(\mathbf{x}_{t},t),\Sigma_{\theta}(\mathbf{x}_{t},t)),$ (2) $$

where $\mu_{\theta}$ and $\Sigma_{\theta}$ are learned parameters defining the mean and covariance.

Figure: Figure 3: Overview of backbone of current DiT-based video generation models, which compromises Spatial Transformer block and Temporal Transformer block. Cross-attention incorporates information from the text conditioning input.
Refer to caption: x3.png

#### Video generation models.

Early approaches in this domain primarily leveraged GANs (Goodfellow et al., ), VA-VAE (Van Den Oord et al., ), autoregressive Transformer (Rakhimov et al., ) and convolution-based diffusion models (Ho et al., ).
However, the remarkable success of Sora (Brooks et al., ) has demonstrated the great potential of diffusion transformers (DiT) (Peebles & Xie, ) in video generation, which leads to a series of research including Open-Sora (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), CogVideoX (Yang et al., ), and Latte (Ma et al., ).

In this work, we focus on accelerating the DiT-based video generation models.
As illustrated in Figure 3, we present the fundamental architecture of DiT-based video diffusion models.
The backbone is composed of two types of Transformer blocks: spatial and temporal.
Spatial Transformer blocks capture spatial information among tokens that share the same temporal index, while temporal Transformer blocks handle information across different temporal dimensions.
Cross-attention enables the model to incorporate information from the conditioning input at each step, ensuring that the generated output is coherent and aligned with the given context.
Note that cross-attention mechanisms are not included in the temporal blocks of some video generation models (Ma et al., ).

### 2.2 Attention Redundancy in Video DiTs

Figure: Figure 4: a) Visualization of attention differences in Open-Sora. diff i-j represents the difference between step i and step j. b) Comparison of total attention time cost between Stable Video Diffusion (Blattmann et al., ) (U-Net) and Open-Sora (DiT). c) Quantitative analysis of attention differences in Open-Sora, assessed using mean squared error (MSE). The dashed line represents the average value of the corresponding attention difference.
Refer to caption: x4.png

#### Attention’s rising costs.

Video DiTs employ three distinct types of attention: spatial, temporal, and cross-attention. Consequently, the computational cost of attention in these models is significantly higher than in previous methods. As Figure 4(b) illustrates, the proportion of time in total for attention in video DiTs is significantly larger than in CNN approaches, which will further escalate with larger video sizes. This dramatic increase poses a significant challenge to the efficiency of video generation.

#### Unmasking attention patterns.

To accelerate costly attention components, we conduct an in-depth analysis of their behavior. Figure 4(a) depicts the visualized differences in attention outputs across various stages. We observe that for middle segments, the differences are minimal and patterns appear similar. The first few steps show vague patterns, likely due to the initial arrangement of content. In contrast, the final steps exhibit significant differences, presumably as the model sharpens features.

#### Similarity and diversity.

To further investigate this phenomenon, we quantify the differences in attention outputs across all diffusion steps, as illustrated in Figure 4(c). Our analysis reveals that the differences in attention outputs demonstrate low difference for approximately 70% of the diffusion steps in the middle segment. Additionally, the variance in their outputs is also low, but still with slight differences: spatial attention shows the highest variance, followed by temporal and then cross-attention.

### 2.3 Pyramid Attention Broadcast

Figure: Figure 5: Overview of Pyramid Attention Broadcast. Our method (shown on the right side) which sets different broadcast ranges for three attentions based on their differences. The smaller the variation in attention, the broader the broadcast range. During runtime, we broadcast attention results to the next several steps (shown on the left side) to avoid redundant attention computations.
Refer to caption: x5.png

Building on findings above, we propose Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free method to speedup DiT-based video generation by alleviating redundancy in attention computations. As shown in Figure 5, PAB employs a simple yet effective strategy to broadcast the attention output from some diffusion steps to their subsequent steps within the stable middle segment.
Different from previous approaches that reuse attention scores (Li et al., ), we choose to broadcast the entire attention module’s outputs, as we find this method to be equally effective but significantly more efficient. This approach allows us to completely bypass redundant attention computations in those subsequent steps, thereby significantly reducing computational costs. This can be formulated as:

$$ $O_{\mathrm{attn.}}=\{F(X_{t}),{\color[rgb]{0.625,0.625,0.625}\underbrace{Y_{t} ^{*},\cdots,Y_{t}^{*}}_{\rm broadcast\,range}},F(X_{t-n}),{\color[rgb]{ 0.625,0.625,0.625}\underbrace{Y_{t-n}^{*},\cdots,Y_{t-n}^{*}}_{\rm broadcast\, range}},\cdots\}.$ (3) $$

where $O_{\mathrm{attn.}}$ refers to the output of the attention module at all timesteps, $F(X_{t})$ denotes the attentions are calculated at timestep $t$ and $Y_{t}^{*}$ indicates the attentions results are broadcast from timestep $t$.

Furthermore, our research reveals that employing a single strategy across all attention types is still far from optimal speedup, as the feature and difference vary a lot for each attention as shown in Figure 2. To enhance efficiency while preserving quality, we propose to tailor broadcast ranges for each attention type, as depicted in Figure 5. The determination of the broadcast ranges is based on two key factors: the rate of change and the stability of each attention type. Attention mechanisms that exhibit more changes and fluctuations at consecutive step are assigned smaller broadcast ranges for their outputs. This adaptive methodology facilitates more efficient handling of diverse attentions within the model architecture.

### 2.4 Broadcast sequence parallelism

Figure: Figure 6: Comparison between original sequence parallelism and ours. When temporal attention is broadcast, we can avoid all communication.
Refer to caption: x6.png

We introduce broadcast sequence parallel to enhance video generation speed based on Dynamic Sequence Parallelism (DSP) (Zhao et al., ), which leverages PAB’s unique characteristics to improve sequence parallelism.
Sequence parallel methods distributes workload across GPUs, thus reduce generation latency. But they incur significant communication overhead for temporal attention as shown in Figure 6. By broadcasting temporal attention, we eliminate intra-module communications, substantially reducing overhead without quality loss.
This enables more efficient, scalable distributed inference for real-time video generation.

## 3 Experiments

In this section, we present our experimental settings, followed by our results and ablation studies. We then evaluate the scaling capabilities of our approach and visualize the results.

### 3.1 Experimental setup

#### Models.

We select three state-of-the-art open-source DiT-based video generation models including Open-Sora (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), and Latte (Ma et al., ) as our experimental models.

#### Metrics.

Following previous works (Li et al., ; Ma et al., ), we evaluate video quality using the following metrics: VBench (Huang et al., ), Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., ), and Structural Similarity Index Measure (SSIM) (Wang & Bovik, ). VBench evaluates video generation quality, aligning with human perception. PSNR quantifies pixel-level fidelity between outputs, while LPIPS measures perceptual similarity, and SSIM assesses the structural similarity.
The details of evaluation metrics are presented in Appendix A.2.

#### Baselines.

We employ $\Delta$-DiT (Chen et al., ) and T-GATE (Zhang et al., ), which are also cache-based methods as baselines in the evaluation.
We show details in Appendix A.3.

#### Implementation details.

All experiments are carried out on the NVIDIA H100 80GB GPUs with Pytorch. We enable FlashAttention (Dao et al., ) by default for all experiments.

### 3.2 Main Results

**Table 1: Latency and speedup are calculated on one GPU. PABαβγ denotes broadcast ranges of spatial ($\alpha$), temporal ($\beta$), and cross ($\gamma$) attentions. Video generation specifications: Open-Sora (2s, 480p), Open-Sora-Plan (2.7s, 512x512), Latte (2s, 512x512). PSNR, SSIM, and LPIPS are calculated against the original model results. FLOPs indicate floating-point operations per video generation.**
| model | method | VBench (%) $\uparrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$ | FLOPs (T) | latency (s) | speedup |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Open-Sora | original | 79.22 | – | – | – | 3230.24 | 26.54 | – |
| $\Delta$-DiT | 78.21 | 11.91 | 0.5692 | 0.4811 | 3166.47 | 25.87 | 1.03$\times$ |  |
| T-GATE | 77.61 | 15.50 | 0.3495 | 0.6760 | 2818.40 | 22.22 | 1.19$\times$ |  |
| PAB246 | 78.51 | 27.04 | 0.0925 | 0.8847 | 2657.70 | 19.87 | 1.34$\times$ |  |
| PAB357 | 77.64 | 24.50 | 0.1471 | 0.8405 | 2615.15 | 19.35 | 1.37$\times$ |  |
| PAB579 | 76.95 | 23.58 | 0.1743 | 0.8220 | 2558.25 | 18.52 | 1.43$\times$ |  |
| Open-Sora-<br>Plan | original | 80.39 | – | – | – | 12032.40 | 46.49 | – |
| $\Delta$-DiT | 77.55 | 13.85 | 0.5388 | 0.3736 | 12027.72 | 46.08 | 1.01$\times$ |  |
| T-GATE | 80.15 | 18.32 | 0.3066 | 0.6219 | 10663.32 | 39.37 | 1.18$\times$ |  |
| PAB246 | 80.30 | 18.80 | 0.3059 | 0.6550 | 9276.57 | 33.83 | 1.37$\times$ |  |
| PAB357 | 77.54 | 16.40 | 0.4490 | 0.5440 | 8899.32 | 31.61 | 1.47$\times$ |  |
| PAB579 | 71.81 | 15.47 | 0.5499 | 0.4717 | 8551.26 | 29.50 | 1.58$\times$ |  |
| Latte | original | 77.40 | – | – | – | 3439.47 | 11.18 | – |
| $\Delta$-DiT | 52.00 | 8.65 | 0.8513 | 0.1078 | 3437.33 | 10.85 | 1.02$\times$ |  |
| T-GATE | 75.42 | 19.55 | 0.2612 | 0.6927 | 3059.02 | 9.88 | 1.13$\times$ |  |
| PAB235 | 76.32 | 19.71 | 0.2699 | 0.7014 | 2767.22 | 8.91 | 1.25$\times$ |  |
| PAB347 | 73.69 | 18.07 | 0.3517 | 0.6582 | 2648.45 | 8.45 | 1.32$\times$ |  |
| PAB469 | 73.13 | 17.16 | 0.3903 | 0.6421 | 2576.77 | 8.21 | 1.36$\times$ |  |

#### Quality results.

Table 1 presents quality comparisons between our method and two baselines across four metrics and three models. We generate videos based on VBench’s (Huang et al., ) prompts, as other datasets have been shown to suffer from overfitting (Zheng et al., ; Lab & etc., ). We then evaluate VBench (Huang et al., ) for each method independently, and calculate other metrics including PSNR, LPIPS, and SSIM, with respect to the original model results.

Based on the results, we make the following observations:
i) Our method achieves comparable or superior quality results to the two baselines while simultaneously achieving significantly higher acceleration by up to 1.58$\times$ on a single GPU. This demonstrates our method’s ability to improve efficiency with negligible quality loss.
ii) Our method consistently performs well across all three models, which utilize diverse training strategies and noise schedulers, demonstrating its generalizability.

Figure: Figure 7: Speedups. We evaluate the latency and speedup achieved by PAB for single video generation across up to 8 NVIDIA H100 GPUs. The results are presented for three models utilizing broadcast sequence parallelism.
Refer to caption: x7.png

#### Speedups.

Figure 7 illustrates the significant speedup achieved by our method when leveraging multiple GPUs with broadcast sequence parallelism. Our method demonstrates almost linear speedups as the GPU count increases across three different models. Notably, it achieves an impressive 10.60$\times$ speedup when utilizing 8 GPUs. These results highlight the significant reduction in communication overhead and underscore the efficacy of our broadcast sequence parallelism strategy.

### 3.3 Ablation Study

To thoroughly examine the characteristics of our method, we conduct extensive ablation studies. Unless otherwise stated, we apply PAB246 (the higest quality, but less speedup) to Open-Sora for generating 2s 480p videos using a single NVIDIA H100 GPU.

Table: Table 2: Evaluation of components. w/o indicates the broadcast strategy is disabled for that component and $\Delta$ represents the corresponding increased latency.

| broadcast strategy | latency (s) | $\Delta$ | VBench (%) $\uparrow$ |
| --- | --- | --- | --- |
| w/o spatial attn. | 21.74 | +1.87 | 78.45 |
| w/o temporal attn. | 23.95 | +4.08 | 78.98 |
| w/o cross attn. | 20.98 | +1.11 | 78.58 |
| w/o mlp | 20.27 | +0.40 | 78.59 |
| all components | 19.87 | – | 78.51 |

Table: Table 3: Broadcast object comparison. We compare the speedup, memory cost for different broadcasting object. attention outputs refer to the final output of the attention module.

| broadcast object | VBench (%) | latency (s) |
| --- | --- | --- |
| original | 79.22 | 26.54 |
| attention scores | 78.53 | 29.12 |
| attention outputs | 78.51 | 19.87 |

#### Evaluation of components.

As shown in Table 3.3, we compare the contribution of each component in terms of speed and quality. We disable the broadcast strategy for each component individually and measure the VBench scores and increase in latency. While the impact on VBench scores is negligible, each component contributes to the overall speedup.
Spatial and temporal attentions yield the most computational savings, as they address more extensive redundancies compared to other components. Cross attention follows, offering moderate improvements despite its relatively lightweight computation. The mlp shows limited speedup due to its inherently low redundancy.

Figure: Figure 8: Evaluation of attention broadcast ranges. Comparison of latency and video quality across varying attention broadcast ranges in spatial, temporal, and cross attentions.
Refer to caption: x8.png

#### Effect of attention broadcast range.

We conduct a comparative analysis of different broadcast ranges for spatial, temporal, and cross attentions. As illustrated in Figure 8, our findings reveal a clear inverse relationship between broadcast range and video quality. Moreover, we observe that the effect of different broadcast range varies across different attention, suggesting that each type of attention has its own distinct characteristics and requirements for optimal performance.

#### What to broadcast in attention?

While previous works (Treviso et al., ) typically reuse attention scores, we find that broadcasting attention outputs is superior.
Table 3.3 compares the speedup and video quality achieved by broadcasting attention scores versus attention outputs. Our results demonstrate that broadcasting attention outputs maintains similar quality while offering much better efficiency, for two primary reasons:

i) Attention output change rates are low, as the accumulated results after attention aggregation remain similar despite pixel-level changes. This further indicates significant redundancy in attention computations.
ii) Broadcasting attention scores reduces efficiency by precluding the use of optimized kernels such as FlashAttention. It also necessitates additional attention-related computations, including normalization and attention projection, which are avoided when broadcasting outputs.

### 3.4 Scaling ability

To evaluate our method’s scalability, we conduct a series of experiments. In each experiment, we apply PAB246 (the higest quality, but less speedup) to Open-Sora as our baseline configuration, change only the video sizes, parallel method and GPU numbers.

Table: Table 4: Communication and latency comparison of different sequence parallelism methods on 8 NVIDIA H100 GPUs with and without our method. original refers to our method on single GPU. comm. represents the total communication volume required to generate a single 2s 480p video.

| method | w/o PAB | w/ PAB |  |  |
| --- | --- | --- | --- | --- |
| comm. (G) | latency (s) | comm. (G) | latency (s) |  |
| original | – | 97.51 | – | 71.25 |
| Megatron-SP | 184.63 | 17.17 | 104.62 | 14.78 |
| DS-Ulysses | 46.16 | 12.34 | 26.16 | 9.85 |
| DSP | 23.08 | 12.01 | – | – |
| ours | – | – | 13.08 | 9.29 |

Figure: Figure 9: Real-time video generation performance. We evaluate our methods’ speed in frames per second (FPS) using 8 and 16 NVIDIA H100 GPUs for 480p and 720p videos.

#### Scaling to multiple GPUs.

We compare the scaling efficiency with and without our method using 8 GPUs in Table 3.4 for four sequence parallelism methods including Megatron-SP (Korthikanti et al., ), DS-Ulysses (Jacobs et al., ) and DSP (Zhao et al., ). The results demonstrate that: i) PAB significantly reduces communication volume for all sequence parallelism methods. Furthermore, our method achieves the lowest communication cost compared to other techniques, and achieving near-linear scaling on 8 GPUs. With a larger temporal broadcast range, it can yield even greater performance improvements.
ii) Implementing sequence parallelism alone is insufficient for optimal performance because of the significant communication overhead across multiple devices.

Figure: Figure 10: Scaling video size. Validating our method’s acceleration and scaling capabilities on single and multi-GPU setups for generating larger videos.
Refer to caption: x10.png

#### Scaling to larger video size.

Currently, most models are limited to generating short, low-resolution videos. However, the ability to generate longer, higher-quality videos is both inevitable and necessary for future applications. To evaluate our model’s capacity to accelerate processing for larger video sizes, we conducted tests across various video lengths and resolutions, as illustrated in Figure 10. Our results demonstrate that as video size increases, we can deliver stable speedup on a single GPU and better scaling capabilities when extending to multiple GPUs. These findings underscore the efficacy and potential of our method for processing larger video sizes.

#### Real-time video generation.

We evaluate our method’s speed in terms of FPS on 8 and 16 devices. Since in inference, the batch size of diffusion is often 2 because of CFG. Therefore, we split the batch first and apply sequence parallelism to each batch; in this way, PAB can extend to 16 devices with almost linear acceleration. As shown in Figure 9, we can achieve real-time with very high FPS video generation for 480p videos on 8 devices and even for 720p on 16 devices. Note that with acceleration techniques like Torch Compiler, we are able to achieve even better speed.

Figure: Figure 11: Runtime breakdown for generating a 2s 480p video. attn. denotes the time consumed by attention operations alone, while attn. related includes the time for additional operations associated with attention, such as normalization and projection.
Refer to caption: x11.png

#### Runtime breakdown.

To further investigate how our method achieves such significant speedup, we provide a breakdown of the time consumption for various components, as shown in Figure 11. The analysis reveals that the attention calculation itself does not consume a large portion of time because the sequence length for attention will be much shorter if we do attention seperately for each dimension. However, attention-related operations, such as normalization and projection, are considerably more time-consuming than the attention mechanism itself, which mainly contribute to our speedup.

### 3.5 Visualization

As shown in Figure 12, we visualize the video results generated by our method compared to the original model. The generation specifications are the same with those in Table 1, and we employ the highest quality strategy outlined in the table. The visualized results demonstrate that our method maintains the original quality and details.

Figure: Figure 12: Qualitative results. We compare the generation quality between our method and original model. The figures are randomly sampled from the generated video.
Refer to caption: x12.png

## 4 Related Work

### 4.1 Video Generation

Video generation has seen remarkable progress recently, which aims to synthesize visually high-quality and motion-consistent videos.
Previous research can be broadly divided into three key stages: GAN-based, auto-regressive, and diffusion models.
Initially, GANs were extended from image to video generation, aiming to maintain spatiotemporal coherence (Vondrick et al., ; Saito et al., ; Clark et al., ; Kahembwe & Ramamoorthy, ; Wang et al., ).
Despite their success, GAN-based approaches often struggled with mode collapse, which hindered their effectiveness in producing diverse and consistent videos.
To address these limitations, auto-regressive models were introduced, leveraging Transformer architectures to predict video frames sequentially, conditioned on previously generated frames.  (Rakhimov et al., ; Weissenborn et al., ; Yan et al., ; Ge et al., )
Although these models generally yield higher video quality and more stable convergence, they demand substantial computational resources.

The latest advancements in video generation have been driven by diffusion models, which iteratively refine noisy inputs to generate high-fidelity video frames (Ho et al., ; An et al., ; Esser et al., ; Ge et al., ).
While many works focus on conv-based diffusions (Wang et al., ; Ho et al., ; Harvey et al., ; Mei & Patel, ; Singer et al., ; Ho et al., ; Luo et al., ; Wang et al., ; Wu et al., ; Zhang et al., ), researchers begin to explore Transformer-based diffusion models for video generation (Zheng et al., ; Lab & etc., ; Ma et al., ) because of scalability (Peebles & Xie, ) and efficiency.
In this work, we focus on accelerating the DiT-based video generation models.

### 4.2 Diffusion Model Acceleration

Advancements in video diffusion models have demonstrated their potential for high-quality video generation, yet their practical application is often limited by slow inference speeds.
Previous research about speeding up diffusion model inference can be broadly classified into three categories.
First, reducing the sampling time steps has been explored through methods such as DDIM (Song et al., ), which enables fewer sampling steps without compromising generation quality. Other works also explore efficient solver of ODE or SDE (Song et al., ; Jolicoeur-Martineau et al., ; Lu et al., ; Karras et al., ; Lu et al., ), which employs a pseudo numerical method to achieve faster sampling.
Second, researchers aimed at reducing the workload and inference time at each sampling step, including distillation (Salimans & Ho, ; Li et al., ), quantization (Li et al., ; He et al., ; So et al., ; Shang et al., ), distributed inference (Li et al., ).
Third, jointly optimized methods simultaneously optimize network and sampling methods (Li et al., ; Liu et al., ).
Moreover, researchers modify the model structure indirectly by using the cache mechanism to reduce computation.
Cache (Smith, ) in computer systems is a method to temporarily store frequently accessed data from the main memory to improve processing speed and efficiency.
Based on the findings that high-level features usually change minimally between consecutive steps, researchers reuse the high-level features in U-Net structure while updating the low-level ones
(Ma et al., ; Li et al., ; Wimbauer et al., ; So et al., ).
Besides,  (Zhang et al., ) cache the redundant cross-attention in the fidelity-improving stage.

However, previous methods mainly focus on the U-Net structure and image domain.
The most similar work are  (Ma et al., ),  (Chen et al., ),  (Zhang et al., ) and  (Li et al., ).
(Ma et al., ) skip the computation of a large proportion of feedforward layers in DiT models through post-training.
(Zhang et al., ) cache the self-attention in the initial stage and reuses cross-attention in the fidelity-improving phase.
(Chen et al., ) caches feature offsets of DiT blocks.
(Li et al., ) reduce the latency of single-sample generation by running convolution-based diffusion models across multiple devices in parallel while sacrificing quality and efficiency for parallel.
Different from previous works, we aim at real-time DiT-based video generation models using training-free acceleration methods. We utilize pyramid attention and broadcast sequence parallel to accelerate video generation without loss of quality.

## 5 Discussion and Conclusion

In this work, we introduce Pyramid Attention Broadcast (PAB), a novel real-time, high quality, and training-free approach to enhance the efficiency of DiT-based video generation.
PAB reduces redundancy through pyramid-style broadcasting by exploiting the U-shaped attention pattern in the diffusion process.
Moreover, the broadcast sequence parallelization method significantly improves distributed inference efficiency.
Overall, PAB enables real-time, high-quality video generation of up to 720p video, consistently outperforming baseline methods across various models.
We believe that PAB provides a simple yet effective foundation for advancing future research and practical applications in video generation.

#### Limitation.

Our approach shows promise but has some limitations. PAB’s performance may vary depending on the input data’s complexity, especially with dynamic scenes. The fixed broadcast strategy might not work best for all video types and tasks. Also, we only focused on reducing redundancy in attention mechanisms, not other parts of the model like Feed-Forward Networks. Future work could explore ways to make PAB more flexible and effective across different applications, such as developing adaptive strategies and expanding redundancy reduction to other model components.

## Appendix A Experiment setup

### A.1 Models

As we focus on DiT-based video generation, three popular state-of-the-art open-source DiT-based video generation models are selected in the evaluation, including Open-Sora (Zheng et al., ), Open-Sora-Plan (Lab & etc., ), and Latte (Ma et al., ).
Open-Sora-Plan (Lab & etc., ) utilizes CausalVideoVAE to compress visual representations and DiT with the 3D full attention module.
Open-Sora (Zheng et al., ) combines 2D-VAE with 3D-VAE for better video compression and uses an SD-DiT block in the diffusion process.
Latte (Ma et al., ) uses spatial Transformer blocks and temporal Transformer blocks to capture video information in the diffusion process.

### A.2 Metrics

In this work, we evaluate our methods using several established metrics to comprehensively assess video quality and similarity.
On the one hand, we assess video generation quality by the benchmark VBench, which is well aligned with human perceptions.

#### VBench (Huang et al., 2024 )

:
VBench is a benchmark suite designed for evaluating video generative models, which uses a hierarchical approach to break down ’video generation quality’ into various specific, well-defined dimensions.
Specifically, VBench comprises 16 dimensions in video generation, including Subject Consistency, Background Consistency, Temporal Flickering, Motion Smoothness, Dynamic Degree, Aesthetic Quality, Imaging Quality, Object Class, Multiple Objects, Human Action, Color, Spatial Relationship, Scene, Appearance Style, Temporal Style, Overall Consistency.
In experiments, we adopt the VBench evaluation framework and utilize the official code to apply weighted scores to assess generation quality.

On the other hand, we also evaluate the performance of the accelerated video generation model by the following metrics.
We compare the generated videos from the original model (used as the baseline) with those from the accelerated model.
The metrics are computed on each frame of the video and then averaged over all frames to provide a comprehensive assessment.

#### Peak Signal-to-Noise Ratio (PSNR)

: a widely used metric for measuring the quality of reconstruction in image processing. PSNR is defined as:

$$ $\text{PSNR}=10\cdot\log_{10}\left(\frac{R^{2}}{\text{MSE}}\right),$ (4) $$

where $R$ is the maximum possible pixel value of the image and MSE denotes the Mean Squared Error between the reference image and the reconstructed image. Higher PSNR values indicate better quality, as they reflect a lower error between the compared images. For video evaluation, PSNR is computed for each frame and the results are averaged to obtain the overall PSNR for the video. However, PSNR primarily measures pixel-wise fidelity and may not always align with perceived image quality.

#### Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018 )

: a metric designed to capture perceptual similarity between images more effectively than pixel-based measures. LPIPS is based on deep learning models that learn to predict perceptual similarity by training on large datasets. It measures the distance between features extracted from pre-trained deep networks. The LPIPS score is computed as:

$$ $\text{LPIPS}=\sum_{i}\alpha_{i}\cdot\text{Dist}(F_{i}(I_{1}),F_{i}(I_{2})),$ (5) $$

where $F_{i}$ represents the feature maps from different layers of the network, $I_{1}$ and $I_{2}$ are the images being compared, Dist is a distance function (often L2 norm), and $\alpha_{i}$ are weights for each feature layer. Lower LPIPS values indicate higher perceptual similarity between the images, aligning better with human visual perception compared to PSNR. LPIPS is calculated for each frame of the video and averaged across all frames to produce a final score.

#### Structural Similarity Index Measure (SSIM) (Wang & Bovik, 2002 )

: the similarity between two images by considering changes in structural information, luminance, and contrast. SSIM is computed as:

$$ $\text{SSIM}(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}{(\mu_{x}^{ 2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})},$ (6) $$

where $\mu_{x}$ and $\mu_{y}$ are the mean values of image patches, $\sigma_{x}^{2}$ and $\sigma_{y}^{2}$ are the variances, $\sigma_{xy}$ is the covariance, and $C_{1}$ and $C_{2}$ are constants to stabilize the division with weak denominators. SSIM values range from -1 to 1, with 1 indicating perfect structural similarity. It provides a measure of image quality that reflects structural and perceptual differences. For video evaluation, SSIM is calculated for each frame and then averaged over all frames to provide an overall similarity measure.

### A.3 Baselines

We employ $\Delta$-DiT (Chen et al., ) and T-GATE (Zhang et al., ), which are cache-based methods as baselines in the evaluation.

**Table 5: Configuration of $\Delta$-DiT. $b$ represents the gate step of two stages and k is the cache interval. Block range refers to the index of the front blocks that are skipped. ’Block range’ refers to the specific indices of the blocks in the DiT-based video generation model that are skipped during the process. For example, ’Block range’ [0, 2] means that the first three blocks in the model block 0, block 1, and block 2—are skipped.**
| $\Delta$-DiT | Diffusion steps | b | k | Block range |
| --- | --- | --- | --- | --- |
| Open-Sora-Plan | 150 | 148 | 2 | [0, 2] |
| Open-Sora | 30 | 25 | 2 | [0, 5] |
| Latte | 50 | 48 | 2 | [0, 5] |

$\Delta$-DiT (Chen et al., ) uses the offset of hidden states(the deviations between feature maps) rather than the feature maps themselves.
$\Delta$-DiT is applied to the back blocks in the DiT during the early outline generation stage of the diffusion model and on front blocks during the detail generation stage. The stage is bounded by a hyperparameter $b$, and the cache interval is $k$.
Since the source code for $\Delta$-DiT is not publicly available, we implement the baseline based on the methods in the paper.
Additionally, we selected the parameters based on experimental results on video generation models.
We only jump the computation of the front blocks during the Outline Generation stage. The detailed configuration is shown in Table 5.

**Table 6: Configuration of T-GATE. $m$ represents the gate step of the Semantics-Planning Phase and the Fidelity-Improving Phase, and $k$ is the cache interval.**
| T-GATE | Diffusion steps | m | k |
| --- | --- | --- | --- |
| Open-Sora-Plan | 150 | 90 | 6 |
| Open-Sora | 30 | 12 | 2 |
| Latte | 50 | 20 | 2 |

T-GATE (Zhang et al., ) reuses Self-Attention in semantics-planning phase and then skip Cross-Attention in the fidelity-improving phase.
T-GATE segments the diffusion process into two phases: the Semantics-Planning Phase and the Fidelity-Improving Phase.
Suppose $m$ represent the gate step of the transition between phases.
Before gate step $m$, during the Semantics-Planning Phase, cross-attention (CA) remains active continuously, whereas self-attention (SA) is calculated and reused every $k$ steps following an initial warm-up period.
After gate step $m$, cross-attention is replaced by a caching mechanism, with self-attention continuing to function. We present details in Table 6.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please check the detailed comments in the weaknesses part.
soundness: 3
strengths: - The motivation is clear, which relies on one training-free method to speedup the video generation (inference).
- The attention redundancy in video DiTs are extensively studied, such as the attention cost, the attention patterns, the attention similarity and diversity. 
- With the extensive studies on attention redundancy, the authors proposed the pyramid attention broadcast. Moreover, different broadcast ranges for each attention are tailored based on the rate of change and the stability of each attention type.
- Experiments and ablation studies are conducted to demonstrate the effectiveness and efficiency of the proposed PAB metric.
summary: The paper propose one technique, namely pyramid attention broadcast (PAB), which is a training-free for speed-up DiT-based video generation.
The idea is simple, which broadcast the attention results for the spatial, temporal, cross attention, which consumes more computation in video DiTs than that in CNN approaches.
The proposed PAB works well in OpenSora, Open-Sora-Plan, and Latte, demonstrating the effectiveness of the proposed method.
weaknesses: - The proposed PAB is simple and effective. However, the optimal solution for PAB, such as how to determine the optimal broadcast ranges are not specified. According to the Figure. 8, the larger broadcast range, the low latency and low video generation quality. Such studies seed to be obvious to the authors. For the specific video generation model, such as open-sora, open-sora-plan, latte, how can we set the broadcast range. Please provide more discussions.
- The authors claimed that the broadcast the attention outputs other than attention scores. In my opinion, the effect seems to be the same or very similar for broadcasting the attention scores outputs. Please provide more explanation.
- One important questions is that the authors reveal that the attention outputs (spatial, temporal, cross attention) are redundancy. As such, they can broadcast the attention outputs with one training-free to speedup the video generation. In my opinion, such studies also means that the existing video generation model, such as open-sora, open-sora-plan, latte, learns a lot of redundancy information, which results in the redundancy attention outputs. Therefore, can PAB helps to learn effectively of the video generation model. Please make more discussions.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The broadcast sequence parallelism in this work relies on DSP, so it’s unclear how much of the latency reduction is directly attributable to PAB alone. In Table 4 or in the main text, it would be helpful to further clarify PAB’s independent contribution to latency improvements.
soundness: 3
strengths: The method’s effectiveness is demonstrated on multi-GPU setups, achieving real-time video generation (e.g., within 2 seconds) on an 8-card H100 configuration. This efficiency is also validated across multiple open-source Video DiT models.
summary: This paper proposes Pyramid Attention Broadcast (PAB), a method for DiT-based video generation that leverages a pyramid-shaped broadcast design. This design is inspired by two key observations: (1) the attention output differences are minimal in the middle 70% of the diffusion steps, and (2) spatial, temporal, and cross-attention differences decrease in a hierarchical, pyramid-like manner. Based on these insights, PAB sets different broadcast ranges for each attention type. The method is further extended to distributed settings, supporting multi-GPU parallelism (e.g., 8-GPU, 16-GPU) to further reduce generation latency. Experiments on VBench and WebVid using three types of DiT models demonstrate that PAB effectively reduces latency and accelerates video generation.
weaknesses: 1. In Figure 4, the quantitative analysis of attention differences uses 30 inference steps based on Open-So ra, but the visualized attention differences cover 50 steps, creating inconsistency. Is this visualization based on Latte? Additionally, it’s unclear whether the attention feature difference plot is derived from an entire dataset analysis or a single video generation process.

2. Some comparisons appear to be unfair. For example, in Figure 7, the original generation using a single device is compared to an 8-device setup, leading to a claimed 10.5x speedup. This comparison may be misleading and should be clarified.

3. Key implementation details should be presented in the main paper. For instance, the most crucial setting, **PAB_246**, which defines the spatial, temporal, and cross-attention broadcast ranges, should be clearly explained rather than shown in supplementary materials.

4. Broadcasting attention is not entirely new. For instance, TGATE, an image generation acceleration approach, also uses iterative caching and reusing of self-attention (SA) for acceleration. This reduces the novelty of the proposed approach.

5. The qualitative results focus primarily on natural, static scenes. In real-world applications, generating videos with complex, dynamic actions, such as those involving people or animals, is more impactful. The qualitative evaluation scope is therefore too limited to showcase the method’s full potential in diverse scenarios.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Overall, I find this to be a well-crafted paper. Nevertheless, I am curious if there might be a more suitable metric for assessing redundancy. Additionally, I wonder if it's possible to dynamically select layer-wise broadcasting in conjunction with distillation-based methods to facilitate real-time video generation.
soundness: 4
strengths: 1. Figure 2 vividly illustrates the motivation, and the results appear highly plausible, with the spatial attention difference being the most significant, followed by the temporal attention difference, and then the cross attention difference.

2. The proposed broadcasting method appears to be straightforward, potent, and adaptable.

3. Broadcasting the entire sequence in parallel significantly enhances inference speed. It seems that sequence parallelism is exceptionally effective for accelerating the process.
summary: This paper proposes Pyramid Attention Broadcast (PAB), which is a real-time, high-quality, and training-free approach for DiT-based video generation. Motivated by the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. This paper broadcast attention outputs to subsequent steps in a pyramid style. PAB demonstrates up to 10.5x speedup across three models compared with baselines, achieving real-time generation for 720p vdeos.
weaknesses: * Why opt for the Mean Squared Error (MSE) metric to assess redundancy, and does the MSE metric accurately reflect the nuances of redundancy?

* It appears that utilizing broadcasting may lead to an increase in communication time. How can we quantify this additional overhead?

* Is it feasible to apply the PAB (Parallel Attention Broadcasting) technique to T2I (Text-to-Image) models, such as FLUX? Given that current T2I models are becoming increasingly large, there is a pressing need to accelerate models that exceed 5 billion parameters.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1)In Table 1, the PAB range is different for different models. Besides, the diffusion timesteps in the Appendix are different, such as Table 6 in the Appendix? Does this mean that these hyperparameters need to be determined experimentally? Will the hyperparameters used for the same model be very different when trained with different data?
 2)When generating long (greater than 1 minute), complex texture, high dynamic range videos, will acceleration lead to more quality loss. Can you give some quantitative results on how video quality metrics change as video length and complexity increase
 3)In Figure 11, why does the attention-related operation take up such a large proportion of the time? Are there other operations besides normalization and projection? Can you give a more detailed breakdown of the attention-related operations？
soundness: 3
strengths: 1)This paper is easy to follow and the code is open source.
2)he acceleration effect is obvious from the qualitative and quantitative experimental results, 
3)The ablation experiment is thorough
summary: This paper proposes an accelerated algorithm for video generation that is training-free. They observe the redundancy in attention calculations and adopts different acceleration strategies for different attention mechanisms. This paper gives detailed quantitative and qualitative results.
weaknesses: 1)This is essentially a cache technology, and the idea has been studied before, such as T-GATE and DeepCache.
2)The generalization of this method is questionable in scenarios with long video duration, complex textures, and drastic dynamic changes. For more details, please refer to Question 2 below.

Meta-review: (a) Scientific Claims and Findings

The paper identifies significant redundancy in attention modules across diffusion steps in DiT-based video generation models. Based on this observation, the authors propose Pyramid Attention Broadcast (PAB), a method that reuses attention outputs across steps. This approach reduces computational overhead without sacrificing video quality.

(b) Strengths
* Clear Motivation and Analysis: The paper provides a clear motivation for addressing redundancy in attention modules and includes extensive empirical studies on attention redundancy within DiT models.
* Clear Speedup: The paper shows convincing evidence of PAB’s ability to reduce inference time.
* Extensive Evaluation: The authors conduct thorough ablation studies and evaluate the proposed method across multiple models to validate the effectiveness of the method.

(c) Weaknesses
* Lack of Clarity on Approach Details: The paper does not provide sufficient implementation details and does not explain the process for determining hyperparameters, such as the optimal broadcast range and diffusion steps.
* Fairness of Comparisons: The claim of a 10x speedup is not valid as it compares between single-GPU and multi-GPU setups.
* Novelty Concerns: The core idea of caching and reusing attention outputs has been explored in prior works, such as T-GATE and DeepCache.
* Limited Qualitative Results: The qualitative results are limited to relatively static and simple domains, raising concerns about the method’s generalizability to more dynamic and complex scenarios.

(d) Reasons for Accept / Reject

Despite its weaknesses, the paper is well-presented, and the authors provide a thorough empirical study to support their claims. The speedup introduced by PAB is well justified by empirical results, demonstrating a meaningful reduction in computation time. However, the contribution is somewhat constrained due to two key factors: the actual speedup achieved (approximately 30%) is far from authors' original 10x claim, and the novelty of the PAB is limited given it's similarity with existing approaches.

Decision: Accept (Poster)