Real-Time Video Generation with Pyramid Attention Broadcast
Authors: Xuanlei Zhao
Xiaolong Jin
Kai Wang
Yang You
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Acceleration
DiT
Video Generation
Efficient
Real-Time
Parallelism
Sequence Parallelism
Primary area: generative models

Abstract: We present Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates up to 10.5x speedup across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.

Published as a conference paper at ICLR 2025
REAL-TIME VIDEO GENERATION WITH PYRAMID ATTENTION BROADCAST
Xuanlei Zhao'r, Xiaolong Jin Kai Wang Yang You' National University of Singapore Code: NUS-HPC-AILab//ideees
Purdue University
ABSTRACT
We present Pyramid Attention Broadeast (PAB), real-time, high quality and training-free approach for DiT-hased video generation. Our method is founded the observation that attention difference in the diffusion process exhibits U-shaped pattern, indicating significant redundancy. We mitigate this by broadcast- ing attention outputs subsequent steps pyramid style. applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates up to 10.5 speedup across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve robust baseline and facilitate future research and application for video generation.
Open-Sora
Open- Sprn-SoraPPlan
Latte
original, 1 GPU
latency: 97.5s, FPS: 2.0
latency: 139.5s, FPS: 1.6
latency: 80.5s, FPS: 0.6
outs, GPUs
latency: 9.2s, FPS: 21.3 (10.5x Faster)
latency: 16.5s FPS: 134 844x Faster)
latency: 9.2s (87x Faster)
FPS: 138
scene in forested area. The first frame The second frame The third frame The video is time-lapse, capturing the transition from day to night, with the lake and forest serving coustant backdrop. The style of the video is na Naturalistic, emphasizing the beauty
prmpp: serene sight
night sky and the peace ness the forest.
Figure 1: Results and speed comparison of our and original methods. PAB can significantly boost generation speed while preserving original quality. Latency is measured on 8 H100 GPUs. Video gen- cration specifications: Open-Sore (2s, 480p), Open-Sora- Sora-Plan (2.7s 512x512), Latte (2s, 512x512).
"equal contribution 1 'equal correspondence (xuanlei, kai. wai. youy) @ comp.nus..edu.sg jin509 purdue.cdu

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Sora (Brooks et al., 2024) kicks off the door of DiT--ased video generation (Peebles & Xie, 2023). Recent approaches (Ma et al., 2024a; Zheng et al., 2024; Lab & etc., 2024) demonstrate their superiority compared to CNN-based methods (Blattmann et al., 2023; Wang et al., 2023a) especially in generated video quality. However, this improved quality comes from significant costs, i.e., more memory occupancy, computation, and inference time. Therefore, exploring an efficient approach for DiT-based video generation becomes urgent for broader GenAl applications (Kumar & Kapoor, 2023; Othman, 2023; Meli et al., 2024).
Model compression methods employ techniques such distillation (Crowley al., 2018; Hsieh et al., 2023), pruning (Han et al., 2015; Ma et al., 2023), quantization (Banner et al., 2019; Lin et al., 2024), and novel architectures (Lin al., 2024) speedup deep learning models and have achieved remarkably success. Recently, they have also been proven to be effective on diffusion models (Sauer et al., 2023; Ma et al., 2024b; Chen et al., 2024b). Nevertheless, these methods usually require additional training with considerable computational resources and datasets, which makes model compression prohibitive and impractical especially for large-scale pre-trained models.
Most recently, researchers revisit the idea of cache (Smith, 1982; Goodman, 1983; Albonesi, 1999) to speedup diffusion models. Different from model compression methods, model caching methods are training-free. They alleviate redundancy by caching and reusing partial network outputs, thereby eliminating additional training. Some studies utilize high-level convolutional features for reusing purposes (Ma et al., 2024c) and efficient distributed inference (Li et al., 2024; Wang et al., 2024). Similar strategies have also been extended to specific attentions (Zhang et al., 2024; Wimbauer et al., 2024), i..., cross attention, and standard transformers (Chen et al., 2024c).
However, training-free frree speedup methods for DiT-based video generation still remains unexplored. Besides, previous model caching methods are not directly applicable to video DiTs due to two intrinsic differences: i) Different architecture. The model architecture has shifted from convolutional networks (Ronneberger et al., 2015) to transformers ( (Vaswani al., 2017). This transaction makes former techniques that aims at convolutional networks not applicable to video generation anymore. ii) Different components. Video generation relies on three diverse attention mechanisms: spatial, temporal, and cross attention (Blattmann et al., 2023; Ma et al., 2024a). Such components lead to more complex dependency and attention interactions, making simple strategies ineffective. They also increase the time consumed by attentions, making attentions more critical than before.
To address these challenges, we take a closer look at attentions in video DiTs and empirically obtain two observations as shown in Figure 2: (i) The attention out- put differences between adjacent diffusion steps exhibit U-shaped pattern, with sta-
stable attention segment
(current vs. previous step)
70%
attention difference
attention difference ranking: spatial temporal cross
bility in the middle 70% steps, indicating considerable redundancy for attention. (ii) Within the stable middle segment, differ- ent attention types also demonstrate vari- ous degrees of difference. Spatial atten- tion changes the most with high-frequency visual elements, temporal attention shows mid- fedfrrquency variations related to move- ments, and cross-modal attention remains the most stable, linking text with video con- tent (Zhang et al., 2024).
spatial attention
temporal attention
cross attention
diffusion step
Comparison of the attention outputs differ- ences between the current and previous diffusion steps. Differences are measured by Mean Square Error (MSE) and averaged across all layers for each diffusion step.
Figure 2:
Based on these observations, we propose Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free method for efficient DiT-based video generation. Our method mitigates attention redundancy by broadcasting the attention outputs to subsequent steps, thus eliminating attention computation in the diffusion process. Specifically, we apply various broadcast ranges for different attentions in a pyramid style, based on their stability and differences as shown in Figure 2. We empirically find that such broadcast strategy can also work MI.P layers. Additionally, to
2

---

Published as a conference paper at ICLR 2025
enable efficient distributed inference, we propose broadcast sequence parallel, which significantly decreases generation time with much lower communication costs.
In summary, to the best of our knowledge, PAB is the first approach that achieves real-time video generation, reaching up to 35.6 FPS with a 10.5 x acceleration without compromising quality. consistently delivers excellent and stable speedup across popular open-source video DiTs, including Open-Sora (Zheng et al., 2024), Notably, as training-free and generalized approach, PAB has the potential empower any future video DiTs with real-time capabilities.
(Lab & etc., 2024), and Latte (Ma et al., 2024a).
2
HOW TO ACHIEVE TIME VIDEO GENERATION
2.1
PRELIMINARIES
Denoising diffusion models. Diffusion models are inspired by the physical process where particles spread out over time due to random motion, which consists forward and reverse diffusion processes. The forward diffusion process gradually adds noise to the data over I steps. Starting with data Xo from a distribution g(x), noise is added at each step:
X = Va + v1 a,Z, for t
where or controls the noise level and Z, ~ (N(,,) I) is Gaussian noise. As 1 increases, x, becomes noisier, eventually approximating normal distribution (O00,) I) when / T. The reverse diffusion process aims to recover the original data from the noisy version:
xx xx) = (x (x,, t). Eg (x,, t)),
(2)
where μıl and Eg are learned parameters defining the mean and covariance.
Video generation models. The remarkable success of Sora (Brooks et al., 2024) has demonstrated the great potential of diffusion transformers (DiT) (Peebles & Xie, 2023) in video generation, which leads to series of research including Open- Sora (Zheng al., 2024), Open-Sora- Sora-Plan (Lab & etc., 2024), and Latte (Ma et al., 2024a).
video
temporal
FFN
cross attn.
temporal attn.
text embed
In this work, we focus on accelerating the DiT-based video generation models. As illustrated in Figure 3, we present the fundamental architecture of video DiTs. Different from tran- sitional transformers, the model is composed of two types of transformer blocks: spatial and temporal. Spatial transformer blocks capture spatial information among tokens that share the same temporal index, while temporal transformer blocks han- dle information across different temporal dimensions. Cross- attention enables the model to incorporate information from the conditioning input each step, ensuring that the generated output is coherent and aligned with the given context. Note that cross-attention mechanisms are not included the temporal blocks of some video generation models (Ma et al., 2024a).
Nx
FFN
Sreds
cross attn.
spatial attn.
noise
Figure 3: Overview of DiT-based video generation models, which compromises spatial and temporal transformer block. Cross attention incorporates information from text.
2.2 ATTENTION REDUNDANCY VIDEO DITs
Attention's rising costs. Video DiTs employ three distinct types of attentions: spatial, temporal, and cross attention. Consequently, the computational cost of attention in these models is significantly higher than in previous methods. As Figure 4(b) illustrates, the proportion of time for total attention in video DiTs is significantly larger than in CNN approaches, which will further increases with larger video sizes. This dramatic increase poses a significant challenge to the efficiency of video generation.
Unmasking attention patterns. To accelerate costly attention components, conduct in-depth analysis of their behavior. Figure 4(a) shows the visualized differences in attention outputs across various stages. We observe that for middle segments, the differences are minimal and patterns appear
3

---

Published as a conference paper at ICLR 2025
mocel
Open Soca
70.08R
29,92% 52.31%
Statle Video Diffosio
uage tecooduse
total atin. cost companson
604
spolle
ifbertie diflnmnce
prompt pan upward of blading oekc inchor fireplace.
"Mage COO
diff 49-50
diff 34-35
diff 24-25
diff 14-15
diff 0-1
difluiion step
a) visualized examples attention difference
quantitative attn. difference
Figure 4: a) Visualization of attention differences in Latte. diff represents the difference between step and s. Comparison of total attention time cost between Stable Video Diffusion (Blattmann et al., 2023) (U-Net) and Open-Sora (DiT). c) Quantitative analysis of attention differences in Open- Sora, assessed using squared error (MSE). The dashed line represents the average value of the corresponding attention difference.
similar. The first few steps show vague patterns, likely due to the initial arrangement of content. In contrast, the final steps exhibit significant differences, presumably the model sharpens features.
Similarity and diversity. To further investigate this phenomenon, we quantify the differences in at- tention outputs across all diffusion steps, as illustrated Figure 4(c). Our analysis reveals that the dif- ferences in attention outputs demonstrate low difference for approximately 70% of the diffusion steps the middle segment. Additionally, the variance their outputs also low, but still with slight dif- ferences: spatial attention shows the highest variance, followed by temporal and then cross-attention.
2.3 PYRAMID ATTENTION BROADCAST
Zt-R
attention difference
spatial attn.
spatial atin.
opatial ottn. temporal
broadcast
temporal attn.
temporal
temporai
broadcest
cross attn.
cross
cross ottn.
cross ottn.
cross attn.
broadcast range
broadcast
Figure 5: Overview of Pyramid Attention Broadcast. Our method (shown on the right side) which sets different broadcast ranges for three attentions based their differences. The smaller the variation in attention, the longer the broadcast range. During runtime, we broadcast attention results to the next several steps (shown the left side) to avoid redundant attention computations.
Building on the findings above, we propose Pyramid Attention Broadcast (PAB), real-time, high quality and training-free method to speedup DiT-based video generation by alleviating redundancy in attention computations. As shown in Figure 5, PAB employs a simple yet effective strategy to broadcast the attention output from some diffusion steps to their subsequent steps within the stable middle segment of diffusion process. Different from previous approaches that reuse attention scores (Treviso et al., 2021), we choose to broadcast the entire attention module's outputs, as we find this method to be equally effective but significantly more efficient. This approach allows to completely bypass redundant attention computations in those subsequent steps, thereby significantly reducing computational costs. This can be formulated
4

---

Published as a conference paper at ICLR 2025
Outtn. = (F(x)) Your F((,n)) Yimi" V
(3)
broadenst raagg
broadcast range
where Oattn. refers to the output of the attention module at all timesteps, F(X;) denotes the attentions are calculated at timestep and Y indicates the attentions results are broadcast from timestep I. We also apply similar strategy mlp modules depicted in Appendix A.2.2.
Furthermore, our research reveals that a single strategy across all attention types is still far from optimal, each attention vary lot shown in Figure 2 and 4(c). To improve efficiency while preserving quality, we propose to tailor different broadcast ranges for each attention, as depicted in Figure 5. The determination of the broadcast ranges based two key factors: the rate of change and the stability each attention type. Attention types that exhibit more changes and fluctuations consecutive step are assigned smaller broadcast ranges for their outputs. This adaptive strategy enables more efficient handling of diverse attentions within the model architecture.
2.4 BROADCAST SEQUENCE PARALLELISM
We introduce broadcast sequence par- allel, which leverages PAB's unique characteristics to improve distributed inference speed. Sequence parallel methods (Jacobs et al., 2023; Zhao al., 2024) distributes workload across GPUs, thus reducing genera- tion latency. But they incur significant communication overhead for tempo- ral attention as shown in Figure By broadcasting temporal attention, naturally eliminate extra communications, substantially reducing overhead without quality loss, which enables more efficient, scalable distributed inference for real-time video generation.
communication
temporal attn.
communication
a) original sequence parallelism
communication
temporal attn.
communication
+
broadcast
b) broadcast sequence parallelism
Figure 6: Comparison between original sequence parallelism and ours. When temporal attention is broadcast, we can avoid all communication.
3 EXPERIMENTS
In this section, we present our experimental settings, followed by our results and ablation studies. We then evaluate the scaling capabilities of our approach and visualize the results.
3.1 EXPERIMENTAL SETUP
Models. We select three sate-of-the-arr open-source DiT-based video generation models including Open-Sora-vl.2 (Zheng et al., 2024), Soen---oa-llan-v...1 (Lab & etc., 2024), and Latte-1.0 (Ma al., 2024a) as our experimental models.
Metrics. Following previous works (Li et al., 2024; Ma et al., 2024a), evaluate video quality using the following metrics: VBench (Huang et al., 2024), Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018), and Structural Similarity Index Measure (SSIM) (Wang & Bovik, 2002). VBench evaluates video generation quality, aligning with human perception. PSNR quantifies pixel-level fidelity between outputs, while LPIPS measures perceptual similarity, and SSIM assesses the structural similarity. The details of evaluation metrics are presented Appendix A.4.
Baselines. We employ A-DiT (Chen al., 2024c) and T-GATE (Zhang et al., 2024), which are both frree caching methods to accelerate DiTs. We show details in Appendix A.3.
Implementation details. All experiments are carried out on the NVIDIA H100 80GB GPUs with Pytorch. We enable Attenion (Dao et al., 2022) by default for all experiments.
5

---

Published as a conference paper at ICLR 2025
model
method
VBench f
PSNR 1
LPIPS 4
SSIM f
FLOPs (T)
latency (8)
speedup
original
79.22
3230.24
26.54
D-DiT
78.21
11.91
0.5692
0.4811
3166.47
25.87
1.03
T-GATE
77.61
15.50
0.3495
0.6760
2818.40
22.22
1.19x
Open-Sora
24B
78.51
27.04
0.0925
0.8847
2657.70
19.87
1.34x
Pa
77.64
24.50
0.1471
0.8405
2615.15
19.35
1.37
PA
76.95
23.58
0.1743
0.8220
2558.25
18.52
1.43x
original
80.39
12032.40
46,49
D-DiT
77.55
13.85
0.5388
0.3736
12027.72
46.08
1.01
Open-Sora Plan
T-GATE
80.15
18.32
0.3066
0.6219
10663.32
39.37
1.18x
PAB246
80,30
18.80
0,3059
0.6550
9276.57
33.83
1.37
357
77.54
16.40
0.4490
0.5440
8899.32
31,61
1.47
71.81
15.47
0.5499
0.4717
8551.26
29.50
1.58x
original
77.40
3439.47
11.18
DiT
52.00
8.65
0.8513
0.1078
3437.33
10.85
1.02x
T-GATE
75.42
19.55
0.2612
0.6927
3059.02
9.88
1.13x
Latte
PAB
76.32
19.71
0.2699
0.7014
2767.22
8.91
1.25
P4
73.69
18.07
0.3517
0.6582
2648.45
8.45
1.32x
PAB.69
73.13
17.16
0.3903
0.6421
2576.77
8.21
1.36x
Table I: Quality results on single GPU. PAB..y denotes broadcast ranges of spatial (a), temporal (3), and cross (7) attentions. Video generation specifications: Open-Sora (2s, 480p), Open-Sora-Plan (2.7s, 512x512), Latte (2s, 512x512). PSNR. SSIM, and LPIPS are calculated against the original model results. FLOPs indicate floating pointt operations per video generation.
3.2
MAIN RESULTS
Quality results. Table presents quality comparisons between our method and baselines across four metries and three models. We generate videos based on VBench's (Huang et al., 2024) prompts. Then evaluate VBench for each method, and calculate other metrics including PSNR, LPIPS, and SSIM with respect to the original results. PAB, denotes broadcast ranges of spatial (a),, temporal (3), and cross (7) attentions. More experiments other datasets can be found in Appendix B.
Based on the results, we make the following observations: i) Our method achieves superior quality results compared with two baselines while simultaneously achieving significantly higher acceleration by up to 1.58x on single GPU. This demonstrates our method's 's ability to improve efficiency with negligible quality loss. ii) Our method consistently performs well across all three models, which utilize diverse training strategies and noise schedulers, demonstrating its generalizability.
original (1 device)
ours device)
ours (2 devices)
ours (4 devices)
ours (8 devices)
Open-Sora
Open-Sora- Plan
Latle
153
97.57 51
139-47
80.52
1997
1.38x
709
1.35x
latency (8)
75.25
3,22x
101.27
2.53x
7.46x
6.08x
4.79k
8.43x
4.72%
8.70x
05.05
30.31
215
387
177
17.06
9.29
8.0s (204 frames), 480p
9.2s (221 frames), 512x612
16.0s (128 framnes), 512x512
Figure 7: Speedups. We evaluate the latency and speedup achieved by PABaaa/AABzz (the strategy with best quality, but less speedup) for single video generation across up to NVIDIA H100 GPUs. The results are presented for three models utilizing broadcast sequence parallelism. The multiple GPUs' speedup is compared with single GPU's speed.
Speedups. Figure/ illustrates the significant speedup achieved by method when leveraging multiple GPUs with broadcast sequence parallelism. Our method demonstrates almost linear speedups the GPU number increases across three different models. Notably, it achieves impressive 10.50x

---

Published as a conference paper at ICLR 2025
speedup when utilizing 8 GPUs. These results highlight the significant reduction in communication overhead and underscore the efficacy of our broadcast sequence parallelism strategy.
3.3 ABLATION STUDY
To thoroughly examine the characteristics of our method, we conduct extensive ablation studies. Unless otherwise stated, we apply PAB24s (the best quality, but less speedup) to Open-Sora for generating 2s 480p videos using single NVIDIA H100 GPU.
Table 2: Evaluation components. w/o indicates that the broadcast strategy is disabled only for the corresponding component.
Table 3: Broadcast object comparison. We compare the speedup and effect for attention map and attention outputs. attention outputs refer to the final output of attention module. attention scores denotes attention score map.
broadcast strategy
latency (8)
VBench
w/o spatial alin.
21.74
1.87
78.45
1.006
w/o temporal attn.
23.95
1.008
78.98
00.47
broadcast object
VBench
latency
w/o cross attn.
20.98
1.11
78.58
10.07
original
79.22
26.54
w/o mlp all components
20.27
0.44
78.59
10.08
attention scores
78.53
29.12
19.87
78.51
attention outputs
78.51
19.87
Evaluation of components. As shown in Table 2, we compare the contribution of each component terms speed and quality. We disable the broadcast strategy for each component individually and measure the VBench scores and increase in latency. While the impacts VBench scores are negligible, all components contribute to the overall speedup. Spatial and temporal attentions yield the most computational savings, as they address more extensive redundancies compared to other components. Cross attention follows, offering moderate improvements despite its relatively lightweight computation. The mlp shows limited speedup due to its inherently low redundancy.
spatial attention.
temporal attention
attention
27.
11
lalency
VBench score
28.
latency (8)
20.
latoncy (6)
25
VBBetchh score latenny (8)
-722
24.0
77 VBerch
22
22.81
21:94
25.72
25.06
25:85
-700
VBench
23.5
22.
752
21
785
broadcast range
Figure 8: Evaluation of attention broadcast ranges. Comparison of latency and video quality across varying attention broadcast ranges spatial, temporal, and cross attentions.
Effect of attention broadcast range. We conduct a comparative analysis of different broadcast ranges for spatial, temporal, and cross attentions. As illustrated in Figure 8, our findings reveal clear inverse relationship between broadcast range and video quality. Morcover, we observe that the effect of different broadcast range varies across different attention, suggesting that each type of attention has its own distinct characteristics and requirements for optimal performance.
What to broadcast in aftention? While previous works (Treviso al., 2021) typically reuse attention scores, we find that broadcasting attention outputs is superior. Table 3 compares the speedup and video quality achieved by broadcasting attention scores versus attention outputs. Our results demonstrate that broadcasting attention outputs maintains similar quality while offering much better efficiency, for two primary reasons:
i) Attention output change rates are low, as the accumulated results after attention aggregation remain similar despite pixel-level changes. This further indicates significant redundancy in attention computations. ii) Broadcasting attention scores prevents the use of efficient attention kernels such as FlashAttention (Dao et al., 2022). It also requires complete attention-relate computations, including attention calculation and linear projection, which avoided when broadcasting outputs.
7

---

Published as a conference paper at ICLR 2025
Table 4: Communication and latency of different sequence parallel methods on 8 NVIDIA H100 GPUs. original refers to our method single GPU. comm. represents the total communication volume to generate 8s 480p video.
ours (8 devices)
ours (15 devicos)
35.56
16:22
FPS
20.57
w/o PAB
w/ PAB
method
comm. (G)
latency (s)
comm. (G)
latency (8)
4.0s (102 frames). 480p 4.0s (102 frames)) 720p
original
97.51
73.25
Figure 9: Real-time video generation performance. We evaluate our methods' speed in frames per second (FPS) using and 16 NVIDIA H100 GPUs for 480p and 720p videos.
Megatron-SP
184.63
17.17
104.62
14.78
DS- DS-Ulysses
46.16
12.34
26.16
9.85
DSP
23.08
12.01
ours
13.08
9.29
or orginaal devize)
ours (1 device)
ours devices)
auts (4 devices)
ours (8 devices)
50.99
97.51
85.79
100154
1,20x
Latency
3,21x
75.25
3.22x
5.00K
128.81
5.8kx
9.0x
10.5%
10.00t
10.50k
15.34
30.21
20:12
303
6:0
14:04
9:2
6:29
15.15
4.06 (102 frames), 480p
8.0s (204 frames),
2.0s (51 frames), 720p
2.0s (51 frames), 1080p
Figure 10: Scaling video size. Validating our method's 's acceleration and scaling capabilities on single and multi-GPU setups for generating larger videos.
3.4 SCALING ABILITY
To evaluate our method's scalability, we conduct a series of experiments. In each experiment, apply PAB245 (the best quality, but less speedup) to Open-Sora as our baseline configuration, change only the video sizes, parallel method and GPU numbers.
Scaling to multiple GPUs. We compare the scaling efficiency with and without our method using 8 GPUs Table for four sequence parallelism methods including Megatron-SY (Korthikanti et al., 2023), DS- (Jacobs et al., 2023) and DSP (Zhao et al., 2024). Our broadcast sequence parallel is implemented based on DSP, and is also adaptable to other methods. The results demonstrate that: i) PAB significantly reduces communication volume for all sequence parallelism methods. Furthermore, our method achieves the lowest communication cost compared to other techniques, and achieving liar-r scaling 8 GPUs. With a larger temporal broadcast range, it can yield even greater performance improvements. ii) Applying sequence parallelism alone insufficient for optimal performance because of the significant communication overhead across multiple devices.
Scaling to larger video size. Currently, most models are limited to generating short, low-resolution videos. However, the ability to generate longer, higher-quality videos both inevitable and necessary for future applications. To evaluate sizes, we conducted tests across various video lengths and resolutions, as illustrated in Figure 10. Our results demonstrate that video size increases, better scaling capabilities when extending to multiple GPUs. These findings underscore the efficacy and potential of our method for processing larger video sizes.
model's 's capacity to accelerate processing for larger video
deliver stable speedup single GPU and
Real-time video generation. We evaluate our method's speed in terms of FPS 8 and 16 devices. Since in inference, the batch size of diffusion is often 2 because of CFG. Therefore, we split the batch first and apply sequence parallelism to each batch: in this way, PAB can extend to 16 devices with almost linear acceleration. As shown in Figure 9, we can achieve real-time with very high FPS video generation for 480p videos 8 devices and even for 720p 16 devices. Note that with acceleration techniques like Torch Compiler (Ansel et al., 2024), we are able to achieve even better speed.
Runtime breakdown. To further investigate how our method achieves such significant speedup, we provide breakdown of the time consumption for various components, shown in Figure 1. The analysis reveals that the attention calculation itself does not consume a large portion of time because the sequence length for attention will be much shorter if we do attention separately for each dimension.
8

---

Published as a conference paper at ICLR 2025
spotial attn. rolated
temporal attn temporal releted
cross atth.
mp
other
related
org-
ours
latency (s)
Figure 11: Runtime breakdown for generating 2s 480p video. atm. denotes the time consumed by
attention operations alone, while atin. related includes the time for additional operations associated with attention, such as normalization and projection.
However, iitention-related operations, such as normalization and projection, are considerably more time-consuming than the attention mechanism itself, which mainly contribute speedup.
3.5
VISUALIZATION
As shown in Figure 12, we visualize the video results generated by our method compared to the original model. we employ the highest quality strategy for each model. The visualized results demonstrate that our method maintains the original quality and details.
Open-
Open- Sora- Plan
Latte
Oricinal
Ours
Orgiiall
Our
Original
Ours
wilb baragoodd. It
3leading
pounst Sunuear landsapp a nounin lake. svall ruit aooda SeT on le wallr NEVOS. mecting and suclight Renuch the cloads naves, In Da bodiquaed of Da acunia da Rpp.
poonst lorsza popular dia, scspyooall Is Detap baleed sone plats with Linelé. dosee 21:00 Eail.
prnnpe wlow gat spoord Hentey osk Indoor Freplace
poonpe sooN Billing ove malizle louses and Ines wister laadscape agalast right do. chrsteas Satidty and colebrazzn coscagt
Figure 12: Qualitative results. We compare the generation quality between method and original model. The figures are randomly sampled from the generated video.

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
4.1 VIDEO GENERATION
Early approaches of video generation primarily leveraged GANs (Goodfellow al., 2014), VA- VAE (Van Den Oord et al., 2017), autoregressive Transformer (Rakhimov et al., 2020) and convolution- based diffusion models (Ho et al., 2022b). Recently Video generation has seen remarkable progress driven by diffusion models, which iteratively refine noisy inputs to generate high-fidelity video frames (Ho et al., 2022b; An et al., 2023; Esser et al., 2023; Chen et al., 2024a). While many works focus on conv-based diffusions (Ho et al., 2020; Harvey et al., 2022; Singer et al., 2022; Ho et al., 2022a; Luo et al., 2023; Wang et al., 2023b; Zhang et al., 2023) and achieve good results, rescarchers begin to explore Transformer-basee diffusion models for video generation (Zheng et al., 2024; Lab & etc., 2024; Ma et al., 2024a) because of scalability and efficiency (Peebles & Xie, 2023).
4.2 DIFFUSION MODEL ACCELERATION
Scheduler: Reducing the sampling steps with schedulers has been explored through methods such as DDIM (Song et al., 2020), which enables fewer sampling steps without compromising generation quality. Other works also explore efficient solver of ODE or SDE (Song et al., 2021; Jolicoeur- Martineau al., 2021; Lu et al., 2022; Karras al., 2022; Lu et al., 2023).
Compression. Researchers aimed reducing the workload and inference time at each sampling step, including distillation (Salimans & Ho, 2022; Li et al., 2023d), quantization (Li et al., 2023c; He et al., 2023; So et al., 2023a; Shang al., 2023) and joint optimization (Li et al., 2023a; Liu et al., 2023). However, these methods demand significant resources and data for training, making them impractical especially for large-scale pre-trained models.
Caching. Recently, researchers revisited the concept of caching (Smith, 1982) in video generation for training-free frree acceleration. Some works (Ma et al.. 2024c; Li et al., 2023b; Wimbauer et al., 2024; So et al., 2023b) reuse high-level features in U-Net structures while only updating low-level features based on the observation. However, these convolution-base to video DiTs. For transformer architectures, T-GATE (Zhang et al., 2024) introduce caching different attention at different stages, while A-DiT (Chen et al., 2024c) propose to cache feature offsets of DiT blocks. Nevertheless, neither approach effectively addresses the unique attention features present in video DiTs, resulting in suboptimal performance.
methods can not directly apply
Parallelism. Sequence parallelism techniques (Korthikanti et al., 2023; Jacobs et al., 2023; Zhao al., 2024) have been proposed to reduce generation latency through distributed inference. However, these methods introduce additional communication costs. To address this issue, some works (Li al., 2024; Wang et al., 2024) leverage convolutional features reduce communication overhead. Nevertheless, these approaches are still limited to convolutions.
5 DISCUSSION AND CONCLUSION
In this work, we introduce Pyramid Attention Broadcast (PAB), a real-time, high quality, and training- free approach to enhance the efficiency of DiT-based video generation. PAB reduces attention redundancy through pyramid-style broadcasting by exploiting the U-shaped attention pattern in the diffusion process. Moreover, our broadcast sequence parallel significantly improves distributed inference efficiency. Overall, PAB achieves up to 10.5x speedup with negligible quality loss and consistently outperforms baselines across various models. We believe that PAB provides a simple yet effective foundation for advancing future research and practical applications in video generation.
Limitation. PAB's performance may vary depending on the input data's 's complexity, especially with dynamic scenes. The fixed broadcast strategy might not work best for all video types and tasks. Also, we only focused on reducing redundancy in attention, not other parts of the model.
Future works. Our work opens several promising avenues for future research. One key direction is extending to more video models with diverse architectures. Another area is the redundancy in MLPs remains under-explored and warrants further investigation. Furthermore, our findings suggest potential for developing more efficient attention for video generation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
D.H. Albonesi. Selective cache ways: on-demand cache resource allocation. In MICRO-32, 1999.
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent- Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Videe Generation, 2023.
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky,
Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vitito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang. Yinghai Lu, C. K. Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark-Albert Saroufim, Marcos Yukio Siraichi, Helen Suk, Shunting Zhang, Michael Suo, Phil Tillet, Xu Zhao, Eikan Wang, Keren Zhou, Richard Zou, Xiaodong Wang, Ajit Mathews, William Wen, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 2024. URL https:/ //apps::/ap... semant eschola arr org/ Corpus ID: 6977478.
Max Bain, Arsha Nagrani, Güil Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. IEEE International Conference on Computer Vision, 2021.
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid- rapid-deployment. NeurlPS, 2019.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXhv:2311.15127 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators, 2024. URL https: https:://pppeee
meode aar-asswwrllde s mum mumuloor..
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song. Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videolln-online: Online video large language model for streaming video. In CVPR, 2024a.
Lei Chen, Yuan Meng, Chen Tang, Xinzhu Ma, Jingyan Jiang. Xin Wang. Zhi Wang, and Wenwu Zhu. Q-dit: Accurate post-training quantization for diffusion transformers. arXiv:2406. 17343, 2024b.
Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. delta-dit: A training-free acceleration method tailored for diffusion transformers. arXiv:2406.01 125, 2024c.
Elliot J Crowley, Gavin Gray, and Amos J Storkey, Moonshine: Distilling with cheap convolutions. NeurlPS, 2018
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. NeurIPS, 2022.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and Content-Guidee Video Synthesis with Diffusion Models. In CVPR, 2023.
Ian Goodfellow, Jean Abadet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
James R. Goodman. Using cache memory to reduce processor-memory traffic. SIGARCH Compur. Archit. News, 11(3):124-31, 1983. ISSN 0163-5964. doi: 10.1145/1066611.8164477 URL 801
fttps:///on http://o..ooo/g/////

---

Published as a conference paper at ICLR 2025
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv:1510.00149, 2015.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible Diffusion Modeling of Long Videos, 2022.
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. PTQD: Accurate Post-Training Quantization for Diffusion Models, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen Video: High Definition Video Generation with Diffusion Models, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. NeurIPS, 2022b.
Cheng-Yo Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-stepl outperforming larger language models with less training data and smaller model sizes. arXiv:2305.02301, 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In CVPR, 2024.
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhan- dari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv:2309. htt.:23399.45009 /4509, 2023.
Alexia aalicoeur-Martines Ke Li, Rémi Gotta Go Fast When Generating Data with Score-Based Models, 2021.
Tal Kachman, and Ioannis Mitliagkas.
Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion- Based Generative Models, 2022.
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. MLSys, 2023.
Madhav Kumar and Anuj Kapoor. Generative and personalized video advertisements. Available at SSRN 4814118, 2023.
PKU-Yuan Lab and Tuzhan etc. Open-sora-plan, April 2024. URL httptt://dooi.orr/// 5281/zenodo 10948109.
Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. AutoDiffusion: Training- Free Optimization of Time Steps and Architectures for Automated Diffusion Model Accelcration, 2023a.
Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In CVPR, 2024.
Senmao Li, Taihang Hu, Fahad Shahbaz Khan, Linxuan Li, Shiqi Yang, Yaxing Wang, Ming-Ming Cheng, and Jian Yang. Faster diffusion: Rethinking the role of unet encoder in diffusion models. arXiv:23/2.09668. 2023b.
Yanjing Li. Sheng Xu, Xianbin Cao, Xiao Sun, and Baochang Zhang. Q-DM: An Efficient Low-bit Quantized Diffusion Model. In NeurlPS, 2023c.
Yanyu Li, Huan Wang. Qing Jin. Ju Hu, Pavlo Chemerys. Yun Fu, Yanzhi Wang. Sergey Tulyakov, and Jian Ren. SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds, 2023d.
12

---

Published as a conference paper at ICLR 2025
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang. Chuang Gan, and Song Han. Awq: Activation-wware weight quantization for on-device Ilm compression and acceleration. MLSys, 2024.
Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang, and Yu Wang. OMS-DPM: Optimizing the Model Schedule for Diffusion Probabilistic Models, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast
ODE Solver for Diffusion Probabilistic Model Sampling Around 10 Steps, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models, 2023.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang. Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Notice of Removal: VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation. In CVPR, 2023.
Xin Ma. Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv:2401.03045 2024a.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. NeurIPS, 2023.
Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cache: Accelerating diffusion transformer via layer caching, arXiv:2406.01733, 2024b.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. In CVPR, 2024c.
K Meli, J Taouki, and D Pantazatos. Empowering educators with generative ai: The genai education frontier initiative. EDULEAAR224 Proceedings, pp. 4289-4299, IATED, 2024.
Imran Othman. Ai video editor: A conceptual review in generative arts. In ICCM, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In CVPR, 2023.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent Video Transformer. 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv:2311.17042, 2023.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training Quantization on Diffusion Models, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A- A-Vidoo Text-to-Video Generation without Vedeo Data, 2022.
Alan Jay Smith. Cache memories. ACM Computing Surveys, 14(3)473-530, 1982.
Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal Dynamic Quantization for Diffusion Models, 2023a.
Junhyuk So, Jungwon Lec, and Eunhyeok Park. Frdiff: Feature reuse for exquisite zero-shot acceleration diffusion models. arXiv:2312.03517, 2023b.
Jiaming Song, Chenlin Meng. and Stefano Ermon.
Denoising diffusion implicit models.
arXivv2010.02500. 2020.
13

---

Published as a conference paper at ICLR 2025
Yang Song, Jascha Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Basee Generative Modeling through Stochastic Differential Equations, 2021.
Marcos Treviso, António Góis, Patrick Fernandes, Erick Fonseca, and André FT Martins. Predicting attention sparsity in transformers. arXiv:2109.124 12188, 2021.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. NeurIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Jiannan Wang, Jiarui Fang, Aoyu Li, and PengCheng Yang. Pipefusion: Displaced patch pipeline parallelism for inference diffusion transformer models. arXiv:2405. 2024.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv:2308.00 06571, 2023a.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang. Yinan He, Jiashuo Yu, Peiqing Yang, Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Qiao, and Ziwei Liu. LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models, 2023b.
Zhou Wang and Alan C Bovik. A universal image quality index. IEEE signal processing letters, 2002.
Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, al. Cache me if you can: Accelerating diffusion models through block caching. In CVPR, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018.
Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross-attention makes inference cumbersome in text-to-image diffusion models. arXiv:2404.02747, 2024.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. ControlVideo: Training-free Controllable Text-to-Vides Video Generation, 2023.
Xuanlei Zhao, Shenggan Cheng, Zangwei Zheng, Zheming Yang, Ziming Liu, and Yang You. Dsp: Dynamic sequence parallelism for multi-dimensional transformers. arXiv:2403. 10266, 2024.
Zangwei Zheng. Xiangyu Peng, Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024, URL https: /ig t hub. com hatm///ccattech//pee.t Sora.
14

---

Published as a conference paper at ICLR 2025
Real-Time Video Generation with Pyramid Attention Broadcast
Appendix
We organize our appendix as follows:
Experimental Settings:
Section A.I: Models
Section A.2: PAB generation settings
Section A.2.1: Attention
Section A.2.2: MLP
Section A.3: Baselines generation settings
Section A.4: Metries
Additional Experimental Results and Findings:
Section B.1: Additional quantitative results
Section B.2: Findings for MLP broadcast
Section B.3: Results for long, complex and dynamic scenes
Section B.4: Breakdown of PAB's contribution with multiple GPUs
Section B.5: Breakdown of time cost within attention module
Section B.6: Workflow comparison for broadcasting different objects
Section B.8: Extension to Text-to-Image model
Section B.7: Various metrics for evaluating redundancy
A EXPERIMENT SETTINGS
A.1 MODELS
As focus DiT-based video generation, three popular state-of-the-art open-source DiT-based video generation models are selected in the evaluation, including Open-Sora (Zheng et al., 2024), Open-Sora-Plan (Lab & etc., 2024), and Latte (Ma et al., 2024a). Open-Sora-Plan (Lab & etc., 2024) utilizes Vidsol VAE to compress visual representations and DiT with the 3D full attention module. Open-Sora (Zheng et al., 2024) combines 2D-VAE with 3D-VAE for better video compression and uses an SD-DiT block in the diffusion process. Latte (Ma et al., 2024a) uses spatial Transformer blocks and temporal Transformer blocks to capture video information in the diffusion process. The inference configs of three models are shown in Table 5, which strictly follow the official settings.
Table 5: The inference config of three models.
scheduler
steps
Open-Sora
RFLOW
30
Sora--laaa
PSNR
150
Latte
DDIM
50
A.2 PAB GENERATION SETTINGS
A.2.1 ATTENTION
In Table we demonstrate the detailed settings of attention broadcast in experiments.
15

---

Published as a conference paper at ICLR 2025
Table 6: The attention broadcast configuration of PAB. diffusion timesteps represents the start and end diffusion timestep of the broadcast, where 1000 is the beginning and 0 is the end.
broadcast range
model
method
diffiusion timesteps
spatial
temporal
cross
PAB
Open-Sora
PAB
[930-450]
PABs79
PAB2s6
2
6
Open- Open-Sora-Plan
PAB
[850-100]
PAB570
203
Latte
PAB
3
7
[800- 1000
A.2.2 MLP
As demonstrated in Section 2.3 and Figure 4(c), the attention outputs exhibit low difference across approximately 70% of the diffusion steps within the middle segment. Spatial attention shows the highest variance, followed by temporal attention and, finally, cross-attention. Empirically, we perform similar analysis on the MI.P module to investigate whether it also involves redundant computations during the diffusion process.
In our current evaluation experiments, we select the skippable MLP modules for each model through empirical analysis in Appendix B.2. We show our detailed configuration for MLP modules in Table
Table 7: The MLP broadcast configuration of PAB. diffusion timesteps represents the starting diffusion timestep of the broadcast and the Block indicates the index of the broadeast block.
model
diffusion timesteps
block
breadeast
Open-Sora
[864, 788, 676]
2,
2
690, 642,
Open-Sora-Plan Plan
618, 594, 570, 546, 522, 498, 474, 450, 426]
[0, 1, 2,
2
Latte
[720, 640, 560, 480, 400)
[0, 1, 2,
2
A.3 BASELINES GENERATION SETTINGS
We employ A-DiT (Chen et al., 2024c) and T-GATE (Zhang et al., 2024), which are cache-based methods as baselines in the evaluation.
Table Configuration of A-DIT. b represents the gate step of two stages and k is the cache interval. Block range refers to the index of the front blocks that are skipped. Block range refers to the specific indices of the blocks the DiT-based video generation model that are skipped during the process. For example, Block range [0. 2] means that the first three blocks in the model block O, block 1, and block 2 are skipped.
D-DIT
dittusion steps
b
k
block range
Open-Sora
30
25
[0, 5]
Open-Sora-Plan
150
148
2
[0, L]
50
[0
A-DiT (Chen et al., 2024c) uses the offset of hidden states (the deviations between feature maps) rather than the feature maps themselves. A-DiT is applied to the back blocks in the DiT during the early outline generation stage of the diffusion model and on front blocks during the detail generation stage. The stage is bounded by a hyperparameter b, and the cache interval is k. Since the source code for A-DiT is not publicly available, we implement the baseline based on the methods in the paper. Additionally, we selected the parameters based experimental results video generation models.
16

---

Published as a conference paper at ICLR 2025
We only jump the computation of the front blocks during the Outline Generation stage. The detailed configuration shown in Table 8.
Table 9: Configuration of T-GATE. m represents the gate step of the Semantics-Planninn Phase and the Fidelity-Improving Phase, and l is the cache interval.
T-GATE
diffusion steps
k
Open-Sora
30
12
2
Open-Sora-Plan
150
Latle
50
2
T-GATE (Zhang al., 2024) reuses self-attention in semantics-planning phase and then skip cross-attention in the
phase. T-GATE segments the diffusion process into two phases: the semantics- planning phase and the fidelity-imrroiing phase. Suppose m represent the gate step of the transition between phases. Before gate step m, during the Semantics-Planning Phase, cross-attention (CA) remains active continuously, whereas self-attention (SA) is calculated and reused every & steps following an initial warm-up period. After gate step m, cross-attention is replaced by a caching mechanism, with self-attention continuing to function. We present details in Table 9.
A.4 METRICS
In this work, we evaluate our methods using several established metrics to comprehensively assess video quality and similarity. On the one hand, we assess video generation quality by the benchmark VBench, which is well aligned with human perceptions.
VBench. VBench (Huang al., 2024) a benchmark suite designed for evaluating video generative models, which uses hierarchical approach break down 'video generation quality' into various specific, well-defined dimensions. Specifically, VBench comprises 16 dimensions in video generation, including Subject Consistency, Background Consistency, Temporal Flickering, Motion Smoothness, Dynamic Degree, Aesthetic Quality, Imaging Quality, Object Class, Multiple Objects, Human Action, Color, Spatial Relationship. Scene, Appearance Style, Temporal Style, Overall Consistency. In experiments, we adopt the VBench evaluation framework and utilize the official code apply weighted scores assess generation quality.
On the other hand, we also evaluate the performance of the accelerated video generation model by the following metries. We compare the generated videos from the original model (used the baseline) with those from the accelerated model. The metrics are computed on each frame of the video and then averaged over all frames to provide a comprehensive assessment.
Peak Signal-to-Noise Ratio (PSNR). PSNR is widely used metric for measuring the quality of reconstruction in image processing. It is defined as:
R² MSE
PSNR 10 logio
where the maximum possible pixel value the image and MSE denotes the Mean Squared Error between the reference image and the reconstructed image. Higher PSNR values indicate better quality, as they reflect a lower error between the compared images. For video evaluation, PSNR computed for each frame and the results are averaged to obtain the overall PSNR for the video. However, PSNR primarily measures pixel-wise fidelity and may not always align with perceived image quality.
Learned Perceptual Image Patch Similarity (LPIPS). LPIPS (Zhang et al., 2018) (118) is a metric
designed to capture perceptual similarity between images more effectively than pixel-base measures. It is based on deep learning models that learn to predict perceptual similarity by training large datasets. measures the distance between features extracted from pre-trained deep networks. The LPIPS score is computed as:
17

---

Published as a conference paper at ICLR 2025
LPIPS Σ as (1))
(5)
where F; represents the feature maps from different layers of the network, I1 and It are the images being compared, Dist is distance function (often L2 norm), and are weights for each feature layer. Lower LPIPS values indicate higher perceptual similarity between the images, aligning better with human visual perception compared to PSNR. LPIPS is calculated for each frame of the video and averaged across all frames to produce a final score.
Structural Similarity Index Measure (SSIM), SSIM (Wang & Bovik, 2002) evaluate the similarity between two images by considering changes in structural information, luminance, and contrast. SSIM is computed as:
SSIM v) = (2 + C1) + C2) (eg 12 Ci)(o + og C2)
where Mr and My are the mean values of image patches, o and of are the variances, Gay the covariance, and C₁ and C2 are constants to stabilize the division with weak denominators. SSIM values range from to with indicating perfect structural similarity. provides a measure of image quality that reflects structural and perceptual differences. For vidco evaluation, SSIM is calculated for each frame and then averaged over all frames to provide an overall similarity measure.
B ADDITIONAL EXPERIMENTAL RESULTS AND FINDINGS
B. ADDITIONAL QUANTITATIVE RESULTS.
PSNR 1
LPIPS J.
SSIM 1
model
method
w/ g.t.
w/ org.
w/
w/ org.
w/
w/
original
8.62
0,7582
0.3506
0.4676
T GATE
8.38
14.22
0.7658
0.3951
0.381
0.6286
Open-Sora
PAB24n
8.69
26.53
0.7682
0.1001
0.3606
0,8635
PABast
8.79
24.12
0.7719
0.1597
0.3695
0.8133
PABa78
8.84
22.48
0.7821
0.2129
0.3741
0.7745
original
0.2619
7.88
12.26
0.7719
0.5572
0,1884
Open-Sora-
T-GATE
8.39
13.60
0.7734
0.4750
0,2436
0.4544
Plan
PAB248
8.65
19.84
0.7653
0.2575
0,2759
0.6847
PAB PABast
8.87
17.39
0.7637
0.3814
0.2766
0.5767
PAB57a
0.4831
original
8.83
0.7670
0.3008
A-DiT
7.09
9.64
0.8071
0.7787
0.0741
0.1567
T-GATE
9.27
19.13
0.7655
0.2585
0.3202
0.6416
PAB535
0.7743
0.2667
0.3742
0.6461
FAB
10.38
17.49
0,7775
0.3577
0.4032
0.5813
P5B450
10.60
16.76
0.7832
0.3934
0.4190
0.5619
Table 10: Quality results on webvid. Latency and speedup are calculated on one GPU. PABast
denotes broadcast ranges of spatial (a), temporal (3), and cross (7) attentions. Video generation specifications: Open-Sora (2s, 480p), Open-Sora-Plan (2.7s, 512x512), Latte (2s, 512x512), w/ g.f. indicates evaluating the metrics based on the ground videos, while w/ org. means with the original methods' outputs.
In Section 3.2, we present results only based on Vbench prompts. To further evaluate the efficacy of our method, we expand our analysis using a subset 1000 videos from WebVid (Bain et al., 2021), 1), large-scale text- video dataset sourced from stock footage websites. We apply PAB this subset, assessing its performance across three models and four metrics. The results of this additional experimentation are summarized in Table 10.
18

---

Published as a conference paper at ICLR 2025
B.2 FINDINGS FOR MLP BROADCAST.
VGt
115
612
119
600
318
1.00
122
282
151
(a) Average MSE of spatial MLP
(b) Average MSE of temporal MLP
3.51
Becco
Block
Decch
3.50
Block
0.00
1e
3.55
Block
Timmetepp
Timesiop
(c) MSE different layers of spatial MLP
(d) MSE different layers of temporal MLP
Figure 13: Quantitative analysis of ML. module differences in Latte by mean squared error (MSE) of the MLP output across continuous time steps. In Figures (a) and (c), we present the results for the spatial MLP, while Figures (b) and (d) illustrate the outcomes for the temporal MLP. Additionally, in Figure (a) and (b), we show the average MSE across all layers. In Figures (c) and (d), we select the block 0, 8, 16, and 24 to illustrate the characteristics of MLPs across different layers.
Auerage MSE
fuerage MSE
0:12
MSE
2.00
6.04
0.00
0:0
Timestoe
Timestop
(a) Average MSE of spatial MLP
(b) Average MSE of temporal MLP
1.75
Block
0.50
Blok
153
Book
6.0
Bock
Blcck 24
Bock 24
3.5
6.15
359
6.0
540
700
643
300
704
800
Timestep
Timeddep
(c) MSE of different layers of spatial ML.
(d) MSE of different layers of temporal ML.P
Figure 14: Quantitative analysis of MLP module differences in Open-Sora by mean squared error
(MSE) of the MLP output across continuous time steps. Figures (a) and (c) present the results for the spatial MLP, while Figures (b) and (d) show the outcomes for the temporal MLP. Figures (a) and (b) display average MSE across all layers, and Figures (c) and (d) examine block 0, 8, 16, and to showcase the MLP characteristics across different layers.
We present quantitative analysis of the FFN output differences in Latte, Open-Sora, and Open-Sora- -oraa-lla,,, using mean squared error (MSE) as the evaluation metric in Figure 13, 14 and 15.
We observe that during the intermediate stages of diffusion, the MSE exhibits periodic spiking pattern, where local maxima occurs at specific timesteps, followed by consistently low values in subsequent timesteps. Therefore, retain the MI.P output at the peak and reuse it during the
19

---

Published as a conference paper at ICLR 2025
Auerage HSE
Avenge
3.2
Auoruge RSC
C9915
Timestop
Timesiop
(a) Average MSE of spatial MLP
(b) Average MSE of temporal MLP
Biccc
0.002
Block
3.2
3.3
3.5%
1.9
920
1.3
735
970
145
414
414
710
732
7.5
Timostop
Timestop
(c) MSE of different layers of spatial MLP
(d) MSE of different layers of temporal MLP
Figure 15: Quantitative analysis of MLP module differences in Open-Sora-Plan by mean squared error (MSE) of the MLP output across continuous time steps. Figures (a) and (c), the results for the spatial MLP are shown, while Figures (b) and (d) show the results for the temporal MLP. Figures (a) and (b) display the average MSE across all layers, and Figures (c) and (d) highlight block 8, 16, and 24 to illustrate the MLP behavior across different layers.
following low-value timesteps. Additionally, by analyzing the FFN modules across different blocks in Figure, we found that the output differences in the lower layers' MLPs are relatively small, while those in the upper layers' MLPs are significantly larger. Based on these findings, we empirically selected MLP modules to broadcast and corresponding broadcast ranges for each model, including Latte, Open-Sora, and Open-Sora-Plan.
B.3 RESULTS FOR LONG, COMPLEX AND DYNAMIC SCENES.
In this section, we evaluate the quantitative and qualitative results for PAB when dealing with long, complex and dynamic scenes.
Table 11: Quantitative results of Open-Sora (16s 480p) on subset dimensions of Vbench datasets for long, complex and dynamic scenes.
method
human
overall consistency
imaging quality
aesthetic quality
dynarnic degree
motioo
subject consistency
total
original
92.67
73.65
61.40
56.59
21.07
96.43
90.26
74.54
PAR24c
91.33
73.43
60.18
56.24
19.91
97.05
90.08
73.98
PA PABast
89.33
72.53
58.17
54.86
19.45
96.40
88, 38.35
72.58
PAB
54.63
88.32
Quantitative results. For model settings, we specifically use Open-Sora to generate videos of 16 seconds duration. This longer duration purposefully challenges our method with more complex and dynamic scenes. Open-Sora is the only model used other models are restricted to fixed short lengths.
For dataset, from VBench's comprehensive 16-dimensional evaluation metrics, strategically select categories that best assess complex and dynamic scenes. Total performance scores are calculated based exclusively these 7 categories provide focused evaluation of complex and dynamic capabilities.
As shown in Table 11, specifically test method under the most challenging conditions by using the longest videos and selecting the more complex tasks the dataset. The results show that PAB performs consistently well, with PAB246 showing comparable performance.
20

---

Published as a conference paper at ICLR 2025
What's particularly encouraging is that PAB maintains good scores even in the most difficult dimensions we tested, like human action and dynamic degree. This shows that our model stays reliable even under demanding conditions.
Qualitative results. As shown in Figure 16, our method demonstrates robust performance in processing dynamic, complex scenes while maintaining high-quality output.
Original
Ours
Original
Ours
Original
Ours
paper,
dancin Square.
balkgroodd,
proppt Sewing machine, dld leving modizee wotking
prompt Yoda playing gulter the sago.
prompt: harso naming to hend in lind.
prompt thes is makeup the mamng
prompt Inon flyng sky.
prompt minoorcccle sovving down sui..
Figure 16: Qualitative results of Open-Sora (16s 480p) on subset dimensions of Vbench datasets for long, complex and dynamic scenes.
B.4 BREAKDOWN OF PAB'S CONTRIBUTION WITH MULTIPLE GPUS
As shown in Table B.4, we evaluate the independent contribution of PAB from computation and communication and come with the following conclusions:
With PAB's computation speedup (save computation by attention broadcast), the latency is further reduced by 24% compared with DSP only.
With PAB's communication speedup (can save communication cost when temporal attention is skipped), the latency can be further reduced by 5.0% compared with computation speedup only.
Since we only evaluate based on PAB216 (better quality but less speedup). PAB is able achieve more speedup if using more aggressive strategies.
Table 12: Breakdown PAB's contribution with multiple GPUs using Open-Sora (8s 480p).
method
latency (s)
original gpu)
96.90
DSP gpus)
11.53
DSP PAR (with computation speedup) (8 gpus)
9.29
DSP + PAB (with computation and communication specdup) (8 gpus)
8.85
B.5 BREAKDOWN TIME COST WITHIN ATTENTION MODULE
As shown in Figure 1, the attention operation takes only small proportion of time in attention module for 2s 480p Open-Sora. In this section, we further investigate what the main cost in attention module.
As shown in Table 13;14;15, our findings show that attention operation isn't actually the main thing slowing down the model. The real bottleneck comes from other parts specifically layernorm and positional embedding. Even though these operations have fewer calculations and FLOPs, they run much slower in practice. Because modern GPUs are built to handle big matrix calculations super efficiently, but they struggle with operations that work on one element at a time, which is exactly what LorerNorm and positional embedding do.
21

---

Published as a conference paper at ICLR 2025
Table 13: Breakdown of time cost in spatial attention.
time
layemorml
mask
modulate
okv proj
layernomm2
proj
attn
reshape
absolute (ms)
1.132
0.149
0.616 9.3%
0.473 7.2%
2.160
0.176
1.595
0.312
normalized
17.1%
2.2%
32.7%
2.7%
24.1%
4.7%
Table 14: Breakdown of time cost in temporal attention.
time
layernorm1
mask
modulate
gky proj
layermom22
pos emb
o pro)
attn
reshape
absolute (ms)
1.126
0.150
0.616
0.477
2.154
2.610
0.176
0.896
0.314
normalized
13.1%
25.3%
Table 15: Breakdown of time cost in cross attention.
qkv proj
proro
reshape
absolute (ms)
0.771 34.8%
0.362
0.176
0.912
normalized
16.2%
7.9%
41.1%
B.6 WORKFLOW COMPARISON FOR BROADCASTING DIFFERENT OBJECTS
In Table 3, we demonstrate the efficiency of broadcasting different objects. In this section, we further demonstrate why there will be such difference:
Broadcasting attention outputs enables us bypass all intermediate computations within the attention module (including layer normalization, positional embedding, and qkvo projections) while maintaining compatibility with efficient attention kernels such FlashAttention (we enable FlashAttention in all experiments by default to be closer to real-world usage).
But broadcasting attention scores still requires partial computation in the attention module (c.g., attention calculation and linear projection). Its performance may even degrade below baseline due to incompatibility with FlashAttention.
To be more clear, here are the workflows in attention module for different broadcast strategies:
original:
x q, u= proj(x) + g, pos emb(layer norm(q, k)) - attn(g k v) + proj(o)
attention score (cannot use FlashAttention because we need attention score explicitly):
x 0 = pr) + o attn(brraaccuut.scorr,,, v) o proj(o)
attention outputs:
broadcast outputs
B.7 VARIOUS METRICS FOR EVALUATING REDUNDANCY
We evaluate different metrics for measuring redundancy shown in Figure 17.
22

---

Published as a conference paper at ICLR 2025
attention difference (MSE)
0.1783
spatial temporal
attention difference (L1)
0.2291
spatial temporal
0,1505
0,2057
cross
cross
0.1189
0.1580
0.1273
0.0783
0.1018
0.0594
0.0764
0.1155
0.028
0.0000
0.0000
20
20
15
26
diffusion step
diffusion step
(a) MSE for redundancy
(b) LI for redundancy
attention difference (mutual info)
0,0008
0106000
attention difference (cos)
0.8589
6.08811
08871
0.05307
0.8722
033733
0.96605
0.047-3
08074
spatial
spetial
0.42632
0.04040
temporal
cross
0.7177
0.2476
temporal
cross
20
diffusion step
diffusion step
(c) Mutual Information for redundancy
(d) Cosine similarity for redundancy
Figure 17: Various metrics for evaluating redundancy.
B.8 EXTENSION TO TO-YTTO-MAGE MODEL
PAB also has the potential to extend to Text-to- Irragee--maee model like FLUX. In this section, we demonstrate speedup, qualitative and quantitative results FLUX.
Original
Ours
Original
Oure
Original
Ours
making
up
prompt. A fow loys noal cornic bendo tnptlerr usbidde
pronpe chiint linccaa stlei andla vink
prompt waling smarad healy
Figure 18: Qualitative results of FLUX with PAB55.
Table 16: Speedup of FLUX. PAB denotes broadcast ranges of spatial (a: and cross (t) attentions.
method
latency (8)
original PABes
13.8
7.8
23

---

Published as a conference paper at ICLR 2025
As shown in Table 16, achieve x.77 speedup compared with original method. We choose PABss because it offers best balance between speedup and image quality. Note that this is only ran on single GPU.
As shown in Figure 18, we visualize the quantitative results on FLUX. Our method can achieve comparable results compared with baseline.
24


Reviews:
[{"review_id": "6TtRQySjli", "reviewer": "Reviewer_R3bt", "summary": "The paper propose one technique, namely pyramid attention broadcast (PAB), which is a training-free for speed-up DiT-based video generation.\nThe idea is simple, which broadcast the attention results for the spatial, temporal, cross attention, which consumes more computation in video DiTs than that in CNN approaches.\nThe proposed PAB works well in OpenSora, Open-Sora-Plan, and Latte, demonstrating the effectiveness of the proposed method.", "questions": "Please check the detailed comments in the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation is clear, which relies on one training-free method to speedup the video generation (inference).\n- The attention redundancy in video DiTs are extensively studied, such as the attention cost, the attention patterns, the attention similarity and diversity. \n- With the extensive studies on attention redundancy, the authors proposed the pyramid attention broadcast. Moreover, different broadcast ranges for each attention are tailored based on the rate of change and the stability of each attention type.\n- Experiments and ablation studies are conducted to demonstrate the effectiveness and efficiency of the proposed PAB metric.", "weaknesses": "- The proposed PAB is simple and effective. However, the optimal solution for PAB, such as how to determine the optimal broadcast ranges are not specified. According to the Figure. 8, the larger broadcast range, the low latency and low video generation quality. Such studies seed to be obvious to the authors. For the specific video generation model, such as open-sora, open-sora-plan, latte, how can we set the broadcast range. Please provide more discussions.\n- The authors claimed that the broadcast the attention outputs other than attention scores. In my opinion, the effect seems to be the same or very similar for broadcasting the attention scores outputs. Please provide more explanation.\n- One important questions is that the authors reveal that the attention outputs (spatial, temporal, cross attention) are redundancy. As such, they can broadcast the attention outputs with one training-free to speedup the video generation. In my opinion, such studies also means that the existing video generation model, such as open-sora, open-sora-plan, latte, learns a lot of redundancy information, which results in the redundancy attention outputs. Therefore, can PAB helps to learn effectively of the video generation model. Please make more discussions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pVitDPevzL", "reviewer": "Reviewer_PKxB", "summary": "This paper proposes Pyramid Attention Broadcast (PAB), a method for DiT-based video generation that leverages a pyramid-shaped broadcast design. This design is inspired by two key observations: (1) the attention output differences are minimal in the middle 70% of the diffusion steps, and (2) spatial, temporal, and cross-attention differences decrease in a hierarchical, pyramid-like manner. Based on these insights, PAB sets different broadcast ranges for each attention type. The method is further extended to distributed settings, supporting multi-GPU parallelism (e.g., 8-GPU, 16-GPU) to further reduce generation latency. Experiments on VBench and WebVid using three types of DiT models demonstrate that PAB effectively reduces latency and accelerates video generation.", "questions": "The broadcast sequence parallelism in this work relies on DSP, so it’s unclear how much of the latency reduction is directly attributable to PAB alone. In Table 4 or in the main text, it would be helpful to further clarify PAB’s independent contribution to latency improvements.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method’s effectiveness is demonstrated on multi-GPU setups, achieving real-time video generation (e.g., within 2 seconds) on an 8-card H100 configuration. This efficiency is also validated across multiple open-source Video DiT models.", "weaknesses": "1. In Figure 4, the quantitative analysis of attention differences uses 30 inference steps based on Open-So ra, but the visualized attention differences cover 50 steps, creating inconsistency. Is this visualization based on Latte? Additionally, it’s unclear whether the attention feature difference plot is derived from an entire dataset analysis or a single video generation process.\n\n2. Some comparisons appear to be unfair. For example, in Figure 7, the original generation using a single device is compared to an 8-device setup, leading to a claimed 10.5x speedup. This comparison may be misleading and should be clarified.\n\n3. Key implementation details should be presented in the main paper. For instance, the most crucial setting, **PAB_246**, which defines the spatial, temporal, and cross-attention broadcast ranges, should be clearly explained rather than shown in supplementary materials.\n\n4. Broadcasting attention is not entirely new. For instance, TGATE, an image generation acceleration approach, also uses iterative caching and reusing of self-attention (SA) for acceleration. This reduces the novelty of the proposed approach.\n\n5. The qualitative results focus primarily on natural, static scenes. In real-world applications, generating videos with complex, dynamic actions, such as those involving people or animals, is more impactful. The qualitative evaluation scope is therefore too limited to showcase the method’s full potential in diverse scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r8cQ0Iu80h", "reviewer": "Reviewer_6TRe", "summary": "This paper proposes Pyramid Attention Broadcast (PAB), which is a real-time, high-quality, and training-free approach for DiT-based video generation. Motivated by the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. This paper broadcast attention outputs to subsequent steps in a pyramid style. PAB demonstrates up to 10.5x speedup across three models compared with baselines, achieving real-time generation for 720p vdeos.", "questions": "Overall, I find this to be a well-crafted paper. Nevertheless, I am curious if there might be a more suitable metric for assessing redundancy. Additionally, I wonder if it's possible to dynamically select layer-wise broadcasting in conjunction with distillation-based methods to facilitate real-time video generation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Figure 2 vividly illustrates the motivation, and the results appear highly plausible, with the spatial attention difference being the most significant, followed by the temporal attention difference, and then the cross attention difference.\n\n2. The proposed broadcasting method appears to be straightforward, potent, and adaptable.\n\n3. Broadcasting the entire sequence in parallel significantly enhances inference speed. It seems that sequence parallelism is exceptionally effective for accelerating the process.", "weaknesses": "* Why opt for the Mean Squared Error (MSE) metric to assess redundancy, and does the MSE metric accurately reflect the nuances of redundancy?\n\n* It appears that utilizing broadcasting may lead to an increase in communication time. How can we quantify this additional overhead?\n\n* Is it feasible to apply the PAB (Parallel Attention Broadcasting) technique to T2I (Text-to-Image) models, such as FLUX? Given that current T2I models are becoming increasingly large, there is a pressing need to accelerate models that exceed 5 billion parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jJIvMyi6Od", "reviewer": "Reviewer_4GeG", "summary": "This paper proposes an accelerated algorithm for video generation that is training-free. They observe the redundancy in attention calculations and adopts different acceleration strategies for different attention mechanisms. This paper gives detailed quantitative and qualitative results.", "questions": "1)In Table 1, the PAB range is different for different models. Besides, the diffusion timesteps in the Appendix are different, such as Table 6 in the Appendix? Does this mean that these hyperparameters need to be determined experimentally? Will the hyperparameters used for the same model be very different when trained with different data?\n 2)When generating long (greater than 1 minute), complex texture, high dynamic range videos, will acceleration lead to more quality loss. Can you give some quantitative results on how video quality metrics change as video length and complexity increase\n 3)In Figure 11, why does the attention-related operation take up such a large proportion of the time? Are there other operations besides normalization and projection? Can you give a more detailed breakdown of the attention-related operations？", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)This paper is easy to follow and the code is open source.\n2)he acceleration effect is obvious from the qualitative and quantitative experimental results, \n3)The ablation experiment is thorough", "weaknesses": "1)This is essentially a cache technology, and the idea has been studied before, such as T-GATE and DeepCache.\n2)The generalization of this method is questionable in scenarios with long video duration, complex textures, and drastic dynamic changes. For more details, please refer to Question 2 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)