Video-Infinity: Distributed Long Video Generation

## Abstract

Abstract Diffusion models have recently
achieved remarkable results
for video generation.
Despite the encouraging performances,
the generated videos are typically
constrained to a small number of frames,
resulting in clips lasting merely a few seconds.
The primary challenges in producing longer videos
include the substantial memory requirements
and the extended processing time required on a single GPU.
A straightforward solution would be to
split the workload across multiple GPUs,
which, however, leads to two issues:
(1) ensuring all GPUs communicate
effectively to share timing and context information,
and (2) modifying existing video diffusion models,
which are usually trained on short sequences,
to create longer videos without additional training.
To tackle these, in this paper we introduce Video-Infinity,
a distributed inference pipeline that
enables parallel processing across multiple GPUs
for long-form video generation.
Specifically,
we propose two coherent mechanisms: Clip parallelism and Dual-scope attention . Clip parallelism optimizes the gathering and sharing of context information across GPUs
which minimizes communication overhead,
while Dual-scope attention modulates the temporal self-attention to balance local and global contexts efficiently across the devices.
Together, the two mechanisms join forces
to distribute the workload
and enable the fast generation of long videos.
Under an 8 × \times × Nvidia 6000 Ada GPU (48G) setup, our method generates videos up to 2,300 frames in approximately 5 minutes,
enabling long video generation at a speed 100 times faster than the prior methods.

## 1 Introduction

A long-standing pursuit of human being is to replicate the dynamic world we live in, in the digital system.
Traditionally dominated by physics and graphics, this effort has recently been enhanced by the emergence of data-driven generative models [, , , ], which can create highly realistic images and videos indistinguishable from reality.
However, these models typically produce very short video segments, with most limited to 16-24 frames [, , ].
Some models extend to 60 or 120 frames [, ], but compromise heavily on resolution and visual quality.

Generating long video poses substantial challenges, primarily due to the extensive resource demands for model training and inference.
Current models, constrained by available resources, are often trained on brief clips, making it difficult to sustain quality over longer sequences.
Moreover, generating a minute-long video
in one go can overwhelm GPU memory, making the task seem elusive.

Existing solutions, including autoregressive, hierarchical, and short-to-long methods, offer partial remedies but have significant limitations.
Autoregressive methods [, ] produce frames sequentially, dependent on preceding ones.
Hierarchical methods  [, , ] create keyframes first, then fill in transitional frames.
Furthermore, some approaches treat a long video as multiple overlapping short video clips [, ].
These methods are not end-to-end; they often miss global continuity, require extensive computation, especially in regions of overlap, and struggle with consistency across segments.

To bridge these gaps, we introduce a novel framework for distributed long video generation, termed Video-Infinity.
On the high level, it work in a divide-and-conquer principle. It breaks down the task of long video generation into smaller, manageable segments.
These segments are distributed across multiple GPUs, allowing for parallel processing. All clients should work collaboratively to ensure the final video is coherent in semantics.

This setup, while straightforward, faces two principal challenges: ensuring effective communication among all GPUs to share
and contextual information, and adapting existing models—typically trained on shorter sequences—to generate longer videos without requiring additional training.

To overcome these challenges, we introduce two synergistic mechanisms: *Clip parallelism* and *Dual-scope attention*.
Clip parallelism enables efficient collaboration among multiple GPUs
by splitting contextual information into three parts.
It uses an interleaved communication strategy
to complete the sharing in three steps.
Building on the capabilities of Clip parallelism, Dual-scope attentionmeticulously adjusts the temporal self-attention mechanisms to achieve an optimal balance between local and global contexts across devices. This balance allows a model trained on short clips to be extended to long video generation with overall coherence.

Even more exciting, by leveraging both strategies, Video-Infinity reduces memory overhead from a quadratic to a linear scale. With the power of multiple device parallelism and sufficient VRAM, our system can generate videos of any, potentially even infinite length.

As a results, our method significantly extends the maximum length of videos that can be generated and accelerates the speed of long video generation.
Specifficly, on an 8 $\times$ Nvidia 6000 Ada (48G) setup, our method manages to generate videos up to 2300 frames in just 5 minutes.
Our contributions are summarized as follows:
(1) We are the first to address long video generation using distributed parallel computation, enhancing scalability and reducing generation times.
(2) We introduce two interconnected mechanisms: Clip parallelism, which optimizes context information sharing across GPUs, and Dual-scope attention, which adjusts temporal self-attention to ensure video coherence across devices.
(3) Our experiments show that, compared to the existing ultra-long text-to-video method Streaming T2V [], our approach can be up to 100 times.

## 2 Related works

### 2.1 Diffusion models

Diffusion models have gained significant attention in recent years due to their impressive ability to generate high-quality media.
Originally introduced for image synthesis, models like Denoising Diffusion Probabilistic Models (DDPM) [] and Latent Diffusion Models (LDM) [] have demonstrated state-of-the-art performance in image generation.
These models progressively denoise a Gaussian noise distribution by learning a sequence of reverse transformations.
Beyond images [, ], diffusion models have also shown promise in audio [, , ] and 3D generation [, ].
Adaptations of diffusion models for video generation incorporate temporal modules to capture the sequential nature of video frames.
For instance, Video Diffusion Models (VDM) [] and Flexible Diffusion Model (FDM) [] effectively extend diffusion frameworks to video data, overcoming challenges like temporal consistency and quality degradation.
More recent models such as AnimateDiff [], ModelScope [], and VideoCrafter [, ] can now produce video clips with better dynamics and improved visual quality.

### 2.2 Techniques for long video generation

Streaming T2V [] introduces a method that relies on a conditional attention module to ensure smooth transitions between video segments and a scene-preserving mechanism for content consistency.
However, this method requires training and is not end-to-end, posing limitations on its practicality.
FreeNoise [] utilizes rescheduled noise sequences and window-based temporal attention to improve video continuity. Despite these innovations, the rescheduled noise contributes to limited dynamics in the generated videos, and the overlapping attention windows introduce additional computational overhead.
NUWA-XL [] from the NUWA series employs a “coarse-to-fine” autoregressive approach, using a global diffusion model to generate keyframes and local models to fill the intermediate frames.
Although promising, NUWA-XL has been trained only within a narrow domain and has not yet made its models and code available, limiting both its evaluation and reproducibility.
Gen-L-Video [] adapts short video diffusion models to handle long videos conditioned on multiple texts without requiring additional training.
This approach cleverly uses latent overlaps to extend video length, which is a common strategy among recent methodologies.
SEINE [] leverages a random-mask diffusion method to automate the generation of transition videos between scenes, guided by textual descriptions.
Like other models, SEINE employs an autoregressive approach and requires image conditioning to facilitate the generation process.

### 2.3 Distributed diffusion

Recently, to reduce the latency of each denoising step in diffusion models, various distributed parallel methods have been applied to image diffusion models.
ParaDiGMS [] utilizes step-based parallelism, where each denoising step is executed on a different GPU device in parallel.
However, this approach tends to waste much computation.
Another method, DistriFusion [], employs a technique of dividing images into patches, allowing different patches to be denoised on separate GPUs.
This approach ensures synchronization among patches and achieves minimal computational waste.
However, it is designed specifically for image diffusion and requires significant communication overhead and specialized hardware support to achieve low latency.

## 3 Preliminaries

Diffusion Models in Video Generation

The process of generating videos using diffusion models involves progressively denoising the latent representation, denoted as $x_{t}$, where $t$ ranges from 0 to $T$.
The initial noisy video latent is represented by a random noise tensor $x_{T}$.
With each denoising step, $x_{t}$ is updated to a clearer latent $x_{t-1}$.
This iterative process continues until $x_{T}$ is denoised to $x_{0}$, which is then fed into a decoder to generate the final video.
The key aspect of updating $x_{t}$ to $x_{t-1}$ is computing the noisy prediction $\epsilon_{t}$, given by:

$$ $\epsilon_{t}=\mathcal{E}_{\theta}(x_{t}),$ (1) $$

where $\mathcal{E}_{\theta}$ represents the diffusion model.

The diffusion model $\mathcal{E}_{\theta}$ can be implemented using various architectures, such as U-Net [, , , , ] or DiT [, , ].
These diffusion models are generally composed of several similar layers.
More specifically, the initial random noise tensor is written as $x_{T}\in\mathbb{R}^{F\times H\times W\times C}$, where $F$ represents the number of frames, $H$ and $W$ denote the height and width of each frame, respectively, and $C$ is the number of channels.

The latent tensor $v$ in each layer generally maintains a consistent shape, $v\in\mathbb{R}^{F\times H^{\prime}\times W^{\prime}\times C^{\prime}}$, where $F$ remains constant across layers.
The dimensions $H^{\prime}$, $W^{\prime}$, and $C^{\prime}$ can vary due to the down-sampling and up-sampling operations of the U-Net architecture.

These layers in the diffusion model $\mathcal{E}_{\theta}$ are usually composed of two main types of modules: spatial and temporal.
The spatial modules receive slices of the latent $v$ shaped $v\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C^{\prime}}$ (a single frame), representing tokens for each video frame in the latent space.
They independently process spatial features within each frame.
The temporal modules receive elongated strips of the latent tensor $v$ shaped $v\in\mathbb{R}^{F\times C^{\prime}}$, representing tokens containing temporal information across frames at specific spatial locations.
They capture temporal dependencies between frames at each location.

## 4 Distributed Long Video Generation

Figure: Figure 2: (a) Pipeline of Video-Infinity: The latent tensor is split into clips and distributed to different devices. The diffusion model predicts noise in parallel with communication, and the noises are concatenated to produce the final output. (b) Illustration of Clip parallelism: In each layer of the video diffusion module, spatial modules operate independently, whereas temporal modules synchronize context elements $c^{i}_{\text{pre}}$, $c^{i}_{\text{post}}$, and $c^{i}_{\text{global}}$. Peer-to-peer and collaborative communications are employed.
Refer to caption: x2.png

At the core of our pipeline, Video-Infinity segments the video latent into chunks, which are then distributed across multiple devices.
An overview of our method is shown in Figure 3, where we divide the video latent along the temporal dimension. Such partitioning allows for parallel denoising on different devices, each handling non-overlapping frames.
To facilitate this, we propose Clip parallelism, detailed in in Section 4.1, a mechanism that efficiently synchronizes temporal information across devices. Additionally, we incorporate Dual-scope attention in Section 4.2, which modulates temporal attention to ensure training-free long video coherence.

Formally, Video-Infinity splits the noisy latent $x_{T}\in\mathbb{R}^{F\times H\times W\times C}$ into $N$ sub-latent clips $x_{T}^{i}\in\mathbb{R}^{F_{\text{clip}}\times H\times W\times C}$, where $i\in[1,N]$, $F_{\text{clip}}={F}/{N}$ represents the number of frames in each clip, and $N$ represents the total number of clips. This structured segmentation facilitates an even load distribution across $N$ devices. Additionally, the spatial modules of video diffusion models operate independently across frames, which eliminates the need for inter-device communication and maintains consistency in the outputs across different devices.

### 4.1 Clip parallelism for video diffusion

To ensure coherence among clips distributed on different devices, we propose Clip parallelism, shown in Figure 3. It parallelizes the temporal layers for video diffusion models and enables efficient inter-device communication.

Parallelized temporal modules.
In the standard diffusion model, a temporal module aggregates features across frames, which could be simplified as

$$ $v_{\text{out}}=\text{temporal}\left(v_{\text{in}}\right),$ (2) $$

where $v_{\text{in}}\in\mathbb{R}^{F\times*\times C^{\prime}}$ is the input feature of this temporal layer.

However, Video-Infinity distributes input feature tensors $v_{\text{in}}$ across multiple devices, dividing them into several clips $v_{\text{in}}^{i}\in\mathbb{R}^{{F}_{\text{clip}}\times*\times C^{\prime}}$, each placed on device(i). To facilitate distributed inference *without modifying the original structure* of the temporal modules, we redefine the temporal operation. This modified operation now considers not only the current clip, but also adjacent clips and global semantics. Conceptually, the parallelized temporal modules are defined as follows:

$$ $\displaystyle v_{\text{out}}^{i}$ $\displaystyle=\text{temporal}_{\text{Parallel}}\left(v_{\text{in}}^{i},c^{i} \right),$ (3) $\displaystyle c^{i}$ $\displaystyle=\left\{c^{i}_{\text{pre}},c^{i}_{\text{post}},c_{\text{global}}\right\}$ (4) $$

where $c^{i}$ stands for the temporal information that enriches each device’s computation by incorporating inter-device context.
Each $c^{i}$ includes temporal information from the preceding device(i-1) via $c^{i}_{\text{pre}}$, and from the succeeding device(i+1) via $c^{i}_{\text{post}}$.
Furthermore, $c_{\text{global}}$ is a selective aggregate of inputs from all devices, optimizing global information coherence and reducing overhead.

The output for each device, $v_{\text{out}}^{i}$, reflects localized computations augmented by these contextual inputs.
The complete output of the layer, $v_{\text{out}}$, is obtained by concatenating the outputs from all devices:

$$ $v_{\text{out}}=\text{Concat}\left(\left\{v_{\text{out}}^{i}|i\in\left[1,N \right]\right\}\right)$ (5) $$

This concatenation provides a holistic view of the processed features, maintaining temporal coherence across the distributed system. Further details on how these temporal modules integrate context will be discussed in Section 4.2.

Figure: Figure 3: Tree different stages in the communication process of Clip parallelism
Refer to caption: x3.png

Three-round context communication. Redefining the temporal modules necessitates efficient communication of the context components $c^{i}=\left\{c^{i}_{\text{pre}},c^{i}_{\text{post}},c_{\text{global}}\right\}$. This is achieved through a three-stage synchronization process, where each stage addresses a specific part of the context, as illustrated in Figure 3.

In the first stage, T1, each device(i) broadcasts its global context $c^{i}_{\text{global}}$ with all other devices through an all_gather() operation. This operation ensures that every device receives the global context, maintaining global consistency across the entire video.

The subsequent stages, T2 and T3 focus on exchanging neighboring contexts. Due to connection limits(^1^11Only one device can communicate with another at a time.), we employ an interleaved strategy. In T2, odd-numbered nodes send their $c^{i+1}_{\text{pre}}$ to their subsequent device(i+1), and even-numbered nodes send their $c^{i-1}_{\text{post}}$ to device(i-1).
In T3, this pattern reverses—odd-numbered devices receive context from their predecessors, and even-numbered devices from their successors. This approach prevents bottlenecks, optimizes channel usage, and minimizes deadlock risks.

Finally, all nodes complete context synchronization, ensuring that each device has the full context needed to perform its computations. More details can be found in the pseudocode in Appendix A.2.

### 4.2 Putting each module in parallel

Building upon the Clip parallelism, this section details that how information is synchronized in each temporal module. A key technique here is the *Dual-scope attention*, which facilitates training-free long video generation and reduces the communication cost.

There are typically three temporal modules in video diffusion models: attention module [] Attention(), convolution module [] Conv(), and group normalization module [] GroupNorm(). We have tailored these modules to integrate into Clip parallelism, enabling distributed processing across multiple devices for efficient and coherent video content synchronization.

DualScope attention.
Applying attention in parallel inference incurs new challenges.
The original attention module require simultaneous access to all input tokens [].
To adopt it under Clip parallelism, it necessitates aggregating tokens across devices, resulting in tremendous communication costs.
Additionally, those attention trained on shorter video clips often degrade in quality when applied to longer sequences.

To address these issues, we introduce the *DualScope attention* module. It revises the computation of K-V pairs to incorporate both local and global contexts into the attention. For each query token from frame $a$, its corresponding keys and values are computed from tokens in the frame set $\mathcal{A}^{a}=\mathcal{N}^{a}\cup\mathcal{G}$:

- •
Local Context ($\mathcal{N}^{a}$). This includes the $|\mathcal{N}^{a}|$ neighboring frames of $a$, from which the keys and values are derived to capture the local context. This local setup is typically achieved through window attention, focusing on the nearby frames to enhance the temporal coherence.
- •
Global Context ($\mathcal{G}$). In contrast, the global context consists of frames uniformly sampled from videos across all devices. This context provides keys and values from a broader range, giving the model access to long-range information

In practice, the keys $K$ and values $V$ are constructed by concatenating the tokens from both contexts $K=\text{Concat}(K_{\text{local}},K_{\text{global}})$ and $V=\text{Concat}(V_{\text{local}},V_{\text{global}})$, where $K_{\text{local}}$ and $Q_{\text{local}}$ is derived from $\mathcal{N}^{a}$ and $K_{\text{global}}$ and $Q_{\text{global}}$ from $\mathcal{G}$. We find that this modified key-value computation can be easily incorporated into existing temporal attention without additional training, enhancing the coherence of long videos.

In the implementation of Clip parallelism, above reformulated attention largely reduce the communication overhead. Comparing to gathering all tokens of length $F$, we only synchronize constant number of tokens. Specifically, we set $|c^{i}_{\text{pre}}|=|c^{i}_{\text{post}}|=\frac{|\mathcal{N}^{a}|}{2}$ and $|c_{\text{global}}|=|\mathcal{G}|$, with both $|\mathcal{N}^{a}|$ and $|\mathcal{G}|$ configured to 16. This reduces data synchronization demands while still capturing essential local and global information.

Convolution module.
The temporal convolution module Conv() applies convolution along the temporal dimension to its input $v_{\text{in}}^{i}\in\mathbb{R}^{F_{\text{clip}}\times C^{\prime}}$.
In Clip parallelism, the context $c^{i}$ of the Conv() includes $c^{i}_{\text{pre}}$ and $c^{i}_{\text{post}}$. They are padded to the original sequences.
Specifically, $c^{i}_{\text{pre}}$ consists of the last $n$ frames of $v_{\text{in}}^{i-1}$, and $c^{i}_{\text{post}}$ consists of the first $n$ frames of $v_{\text{in}}^{i+1}$, where $n$ is the receptive field size of the convolution.

Group normalization.
In video diffusion model, group normalization is applied to the input tensor $v_{\text{in}}^{i}\in\mathbb{R}^{F_{\text{clip}}\times H\times W\times C^{
\prime}}$ to maintain consistent feature scaling across different frames.

In Clip parallelism, each device first computes the group mean $\mu^{i}$ of its respective video clip. These means are aggregated to compute the global mean $\bar{\mu}=\frac{\sum_{i=1}^{N}\mu^{i}}{N}$, where N is the number of devices.
Subsequently, using $\bar{\mu}$, each device computes its standard deviation $\bar{\sigma}^{i}$, which is shared to calculate the global standard deviation $\bar{\sigma}$.
The global mean $\bar{\mu}$ and global standard deviation $\bar{\sigma}$, serving as $c_{\text{global}}$, are used for normalization (^2^22Note that simply averaging the individual standard deviations $\sigma^{i}$ does not yield the true global standard deviation $\bar{\sigma}$.).

## 5 Experiments

### 5.1 Setups

Base model.
In the experiments, the text to video model VideoCrafter2 [] (320 x 512) is selected as the base model of our method.
VideoCrafter2, which was trained on 16-frame videos, excels at generating video clips that are both consistent and of high quality. It is also the highest-scoring open-source video generation model under the VBench [] evaluation, achieving the top total score.

Metrics evaluation.
VBench [] is utilized as a comprehensive video evaluation tool, featuring a broad array of metrics across various video dimensions. For each method, videos are generated using the prompts provided by VBench for evaluation.
The metrics measured encompass all the indicators under the Video Quality category in VBench, including subject consistency, background consistency, temporal flickering, motion smoothness, dynamic degree, aesthetic quality and imaging quality.
Given that VBench’s evaluation is typically performed on video clips of 16 frames, we have modified the evaluation method for videos longer than 16 frames: we randomly sample five 16-frame clips from each video to evaluate separately, and then calculate the average score of these assessments.

Baslines.
Our approach is benchmarked against several other methods:

- •
FreeNoise []:
We chose FreeNoise as a baseline because it is also a training-free method that can base the VideoCrafter2 [] model, which also serves as our base model, to generate long videos.
It employs a rescheduling technique for the initialization noise and incorporates Window-based Attention Fusion to generate longer videos.
- •
Streaming T2V []:
To assess our method’s effectiveness in generating longer videos, StreamingT2V was chosen as our baseline.
Streaming T2V involves training a new model that uses an auto-regressive approach to produce long-form videos.
Like our approach, it also has the capability to generate videos exceeding 1000 frames.
OpenSora V1.1 [], a video diffusion model based on DiT [], supports up to 120 frames, can generate videos at various resolutions, and has been specifically trained on longer video sequences to enhance its extended video generation capabilities.

Dual-scope attentionsetting.
In the implementation of the Dual-scope attention, the number of neighboring frames $\mathcal{N}^{i}$ is set to 16, with 8 frames coming from the preceding clip and 8 frames from the subsequent clip.
The number of global frames, $\mathcal{G}$, is set to 16.
To balance consistency and dynamics during the denoising process, the weights of frames in $\mathcal{G}$ and $\mathcal{N}^{i}$ are dynamically adjusted.
Specifically, the weight of $\mathcal{G}$ increases by 10 for timesteps $t$ greater than 800, whereas the weight of $\mathcal{N}^{i}$ increases by 10 for timesteps $t$ less than or equal to 800.

Implementation details.
By default, all parameters of the diffusion are kept consistent with the original inference settings of VideoCrafter2 [], with the number of denoising steps set to 30.
Our experiments are conducted on 8 $\times$ Nvidia 6000 Ada (with 48G memory) .
To implement the temporal module in Clip parallelism, we utilized the torch.distributed tool package, employing Nvidia’s NCCL as the backend to facilitate efficient inter-GPU communication.
Additionally, all fps conditions are set to 24, and the resolution is set to $512\times 320$. Note that the resolution for Streaming T2V cannot be modified; thus, videos are generated at its default resolution ($256\times 256$ for preview videos and $720\times 720$ for final videos).

**Table 1: Comparison of maximum frames and generation times for different methods.**
| Method | Max | Time Coast (second) |  |
| --- | --- | --- | --- |
|  | Frames | 128 frames | 1024 frames |
| Ours | 2300 | 21 | 131 |
| ST2V (preview) | - | 277 | 2,196 |
| ST2V (final cut) | - | 1730 | 13,726 |
| FreeNoise [] | 128 | 201 | $\times$ |
| Open-Sora v1.1 [] | 120 | 234 | $\times$ |

### 5.2 Main results

Capacity and efficiency.

We evaluated the capabilities of our method on an 8 $\times$ Nvidia 6000 Ada (48G) setup.
Our approach successfully generated videos of 2300 frames at a resolution of 512 $\times$ 320, equivalent to a duration of 95 seconds at 24 frames per second.
Remarkably, the entire computation process took approximately 5 minutes (312s), benefiting from efficient communication and the leveraging of multi-GPU parallel processing.

Table 1 presents the capacities for long video generation of various methods, all measured under the same device specifications.
To ensure comparability, we standardized the resolution of the videos generated by all methods to 512x320.
For StreamingT2V, we provide two sets of data: one for generating preview videos at 256x256 resolution, and another for final videos produced at a resolution of 720x720.
The results demonstrate that our method is the most capable within the end-to-end category, generating the longest videos of up to 2300 frames — 8.2 times more than OpenSora V1.1.
Additionally, our method consistently produces the final videos in the shortest time, both for short videos of 128 frames and long videos of 1024 frames.
Notably, in the generation of 1024-frame videos, our method is over 100 times faster than StreamingT2V, the only baseline method capable of producing videos of this length.
Even when compared to the speed of generating smaller, lower-resolution preview videos by StreamingT2V, our method is 16 times faster.

Video quality.
We compared the videos generated by our method with those produced by FreeNoise [] and StreamingT2V [] for long video generation.
Figure 4 visualizes some frames from videos generated by different methods using the same prompt.
Additionally, Table 2 displays the quality of the videos produced by these methods, evaluated across various metrics in VBench [].

Figure: Figure 4: Comparison of frame images from sample videos generated by different methods.
Refer to caption: x4.png

Figure 4 shows that while the StreamingT2V [] method generates long videos with sufficient dynamism, they lack consistency between the beginning and end.
Conversely, videos generated by FreeNoise [] maintain consistency in object placement throughout but exhibit minimal variation in visuals.
For example, as shown in Figure 4, the video of the person playing the guitar maintains a single pose with only minimal movement.
Similarly, the dog on the left remains intently focused on the camera, with no changes in the position of its ears, nose, or body.
OpenSora V1.1 [] failed to generate the first video and the second video’s background was not smooth.
In contrast, our method not only ensures better consistency but also features more significant motion in the generated videos.

**Table 2: Evaluation metrics: Comparison of performance metrics for various video generation methods as benchmarked by VBench. Bold values represent the best performance within each group.**
| Method | Number of | Subject | Background | Temporal | Motion | Dynamic | Aesthetic | Imaging | Overall |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Frames | consistency | consistency | flickering | smoothness | degree | quality | quality | Score |
| V.C.2 | 16 | 96.85% | 98.22% | 98.41% | 97.73% | 42.50% | 63.13% | 67.22% | 80.58% |
| FreeNoise | 64 | 94.16% | 96.63% | 98.37% | 97.04% | 44.44% | 60.53% | 67.44% | 79.80% |
| OpenSora v1.1 | 64 | 86.18% | 95.83% | 98.47% | 97.27% | 73.61% | 51.69% | 50.61% | 79.09% |
| Ours | 64 | 97.77% | 93.90% | 97.77% | 96.84% | 81.94% | 59.38% | 67.90% | 85.07% |
| ST2V | 192 | 75.02% | 87.93% | 95.96% | 94.71% | 80.56% | 48.08% | 57.85% | 77.16% |
| Ours | 192 | 91.32% | 92.93% | 97.40% | 95.55% | 77.78% | 55.49% | 66.93% | 82.49% |

Table 2 reveals that our method, when compared to our base model VideoCrafter 2 [], experiences a slight decrease in most metrics except for the metric of dynamic.
In the generation of 64-frame videos, the performance of our method shows mixed results compared to other methods, with both advantages and disadvantages noted.
However, our average metric scores are higher than those of both FreeNoise and OpenSora V1.1.
In the generation of longer 192-frame videos, our method outperforms StreamingT2V, the only other method capable of producing videos of this length, across the majority of evaluated metrics.

### 5.3 Ablation

Figure: Figure 5: Visualization of ablation studies on temporal module communication and context effects in video generation. Top panel: Ablation of communication between the ResLayer module and the Attention module, showcasing two adjacent frames from the video sequence generated on different GPUs. Bottom panel: Effects of ablating different contexts within the Attention module, displaying frames from videos generated post-ablation.
Refer to caption: x5.png

As mentioned in Section 4.2, three types of temporal modules (Conv(), GroupNorm(), and Attention()) are adapted to synchronize context in Clip parallelism.
To demonstrate the effectiveness of context synchronization by these modules, we conducted ablation experiments and visualized in Figure 5 the impact of removing certain parts of the context synchronization on the quality of the generated videos.
We performed ablation on the communication between the temporal Attention() module and the temporal ResNet() module in the video diffusion model, where the ResNet module includes temporal Conv() and temporal GroupNorm() as submodules.
Subsequently, we conducted ablations on the global context $c_{\text{global}}$and the local context $c^{i}_{\text{pre}},c^{i}_{\text{post}}$within the Attention() module.

Removing local context. From the top panel of Figure 5, it can be observed that the absence of synchronized information from the ResNet() leads to discrepancies in detail between the last frame on device(1) (frame 23) and the first frame on device(2) (frame 24), which are highlighted in red.
These discrepancies, such as differences in the color of the clothes of the person behind the robot and the shape of the parts in the robot’s hands on the table, do not appear in the original inference.
When context of the Attention() module is absent, frame 23 and frame 24 become markedly different images, illustrating a significant discontinuity between video segments generated on adjacent devices.
These observations suggest that synchronization in both ResNet() and Attention() modules is crucial for preserving visual coherence and continuity across video frames generated on different devices.

Removing global context. The bottom panel of Figure 5 demonstrates that when synchronization of the global context is absent, content consistency within the video is difficult to maintain.
For example, in frames 12 and 16, the horizon remains high, but in frames beyond 20, there is a noticeable rise in the horizon.
Furthermore, when the local context synchronization is removed, although the content across different device clips remains consistent, the lack of shared context in the transition areas leads to anomalies.
For instance, the content of snow in frame 22 abruptly transitions to a dog, highlighted in red.
These examples highlight the importance of global and local context synchronization for video generation.

## 6 Conclusion

We presented Video-Infinity, a distributed inference pipeline that leverages multiple GPUs for long-form video generation.
We present two mechanisms, Clip parallelism and *Dual-scope attention*, to addressed key challenges associated with distributed video generation.
Clip parallelism reduces communication overhead by optimizing the exchange of context information, while *Dual-scope attention* modified self-attention to ensure coherence across devices.
Together, these innovations enable the rapid generation of videos up to 2,300 frames long, vastly improving generation speeds compared to existing methods.
This approach not only extends the practical utility of diffusion models for video production but also sets a new benchmark for efficiency in long-form video generation.

## 7 Limitation

To fully harness the potential of our method, it relies on the availability of multiple GPUs.
Additionally, our approach does not effectively handle video generation involving scene transitions.

## Appendix A Appendix

### A.1 Communication overhead

Table 3 demonstrates the additional time overhead caused by communication between different temporal modules.
The experiments were conducted on multiple Nvidia A5000 GPUs, with two settings: a dual-GPU configuration and an eight-GPU configuration.

**Table 3: Effect of Synchronization on Inference Time**
| Sync | Inference Time (s) |  |
| --- | --- | --- |
|  | 2$\times$GPU | 8 $\times$ GPU |
| Plain | 145.4 | 149.5 |
| + Conv() | 152.9 ($5.1\%\uparrow$) | 157.1 ($5.1\%\uparrow$) |
| + GroupNorm() | 158.3 ($8.9\%\uparrow$) | 160.1 ($7.1\%\uparrow$) |
| + Attention() | 170.7 ($17.4\%\uparrow$) | 180.2 ($20.5\%\uparrow$) |
| Full Sync | 182.3 ($25.3\%\uparrow$) | 192.3 ($28.6\%\uparrow$) |

### A.2 Communication Algorithm

Figure: Algorithm 1 Distributed Temporal Module Communication

## Appendix B Gallery

More videos are available in the supplementary materials and at the following link.
[Link](https://ubiquitous-lobe-604.notion.site/Videos-Gallery-57dd5b23506f483d9f28dc27547b877a)

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Has “Video-Infinity” been tested on different types of video content, such as fast-moving scenes or varying lighting conditions, which might challenge the coherence of frame transitions? How robust is the model across these diverse scenarios?
soundness: 3
strengths: 1. This work brings incremental novelty by adapting distributed parallelism specifically for long-form video generation. It introduces a dual-scope attention mechanism to balance local and global temporal interactions, ensuring coherence across extended sequences. The clip parallelism approach further enables efficient processing of video clips across GPUs, effectively handling the unique scalability and memory demands of video data. These adaptations, including optimizations for temporal continuity, showcase Video-Infinity’s tailored application of distributed inference to the distinct challenges of generating coherent long videos.

2. Speed up performance is great. The proposed Clip parallelism and Dual-scope attention mechanisms optimize inter-device communication and memory requirements, leading to faster processing times and scalability for generating extended video sequences. It could reduce the inference time by up to 52%.
summary: The paper proposes Video-Infinity, a distributed inference pipeline designed for long-form video generation using diffusion models. The framework leverages two main mechanisms: Clip parallelism, which distributes video segments across multiple GPUs to improve processing efficiency, and Dual-scope attention, which balances local and global temporal contexts across devices. Together, these components enable Video-Infinity to generate lengthy, coherent videos with reduced memory overhead. On an 8 × Nvidia 6000 Ada GPU setup, the framework can produce videos up to 2,300 frames in approximately 5 minutes.
weaknesses: 1. Performance. In the Table 2 under 64 frames settings, although the proposed work got the highest overall score, it did not showed dominating better results than other baselines.

2. Results on longer context. This work claims capability to generate longer video clips, while it only shows results for a maximum of 192 frames in Table 2. Since it emphasis the long video generation ability, I would suggest putting more quantitive results on longer video. 

3. Results on memory usage comparison. This work lacks of comparison of reduced memory overhead to demonstrate the efficiency of the method.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The paper focuses on the video diffusion model with the temporal self-attention layer. However, DiT-based architectures are popular in current work, and they tend to use the 3D attention layer instead of the temporal self-attention layer (e.g., CogVideo-5B[2]).
How does the *Dual-scope Attention* perform on the 3D attention layer?
2. How is the video quality compared with FIFO-Diffusion[2]?
3. Would it be possible to extend Video-Infinity to OpenSora for the efficiency comparison?


[3] CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. Zhuoyi Yang, Jiayan Teng, et al, arxiv 2408.06072. 2024.
soundness: 3
strengths: 1. The paper is well-written and easy to follow.
1. It is a training-free inference pipeline while extending the baseline model generation capacity.
2. ***Dual-scope Attention*** provides a new view of gathering the global and local context for high-fidelity long video generation. The generation results are impressive. It might provide insight into the training scheme or new architecture design.
summary: This paper presents Video-Infinity, a distributed inference pipeline for long-video generation using diffusion models. It introduces two techniques for the main challenges in long-video generation. Clip parallelism divides a long clip generation task into several short clips to address the high GPU memory usage. Dual-scope attention gathers local and global context for temporal self-attention to generate a consistent long video. It compares with FreeNoise, StreamingT2V, and OpenSora 1.1V.
weaknesses: 1. The novelty of **Clip Parallelism** is limited. The paper merely migrates the DistriFusion[1] to the video diffusion model, where DistriFusion splits a large image into patches while this paper splits a long video into short clips. The distributed modules are similar to the sparse operations in DistriFusion[1], except for extending the sparse 2D convolution to the 1D/3D temporal convolution with different padding schemes. Also, the *GroupNorm* modification is similar. Moreover, the DistriFusion[1] further introduced *Corrected asynchronous GroupNorm*, which is more efficient than the paper's implementation since the asynchronous communication can be pipelined into the computation.
1. The paper didn't compare the video quality with FIFO-Diffusion[2], which also focused on long-video generation. It is difficult to demonstrate the proposed method's advantage over the SOTA work.
1. In the comparison of efficiency, comparing Open-Sora v1.1 and the proposed method is unfair because they use different model architectures (Spatial-Temporal DiT vs. VideoCrafterV2).
1. There are several typos. (e.g., GPU index in Fig.1, captions in Fig.3

[1] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models. Muyang Li, Tianle Cai, et al. CVPR 2024

[2] FIFO-Diffusion: Generating Infinite Videos from Text without Training. Jihwan Kim and Junoh Kang and Jinyoung Choi and Bohyung Han, NeurIPS 2024

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Please refer to Weaknesses.
soundness: 2
strengths: 1. The empirical results are persuasive, with Video-Infinity achieving a 10x improvement over comparable methods like FIFO-Diffusion and being significantly faster than alternatives like Streaming T2V.

2. The paper is well-organized, clearly outlining the technical details, methodology, and communication strategies.
summary: The paper introduces Video-Infinity, a distributed inference framework designed to enable efficient generation of long videos using diffusion models. It tries to address a challenge in video generation: the resource-intensive nature of long-form content, which often restricts video length and quality due to memory and computation limits. The proposed approach leverages two main innovations: Clip parallelism, which optimizes the distribution and synchronization of context information across multiple GPUs, and Dual-scope attention, which balances local and global temporal self-attention to maintain semantic coherence without requiring additional model training.
weaknesses: 1. It looks like this work adopt the idea from DistriFusion [1]. While the authors claim to tackle a more challenging problem, the dimensionality of frames, from a technical standpoint, is actually much simpler to manage compared to the problems addressed in DistriFusion. 

2. How does this method impact frame-to-frame continuity? I noticed that many of the generated videos in the Supplementary Material exhibit noticeable continuity issues. The authors do not seem to have adequately addressed this problem. Additionally, many other generated long videos can only display repetitive motions and clips.

3. The evaluation lacks comprehensiveness, as the authors have only demonstrated their method on a single model, VideoCrafter2. It remains unclear whether the approach is effective across a broader range of model architectures. For instance, how well does this method generalize to new architectures like DiT? Additionally, what is the performance impact on these models? 

4. It's more of an engineering work, the novelty contribution of this work is not good enough. 

I'm sure it needs dedicated effort for applying this method on every new model architecture.

[1] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models, CVPR'24

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How the synchronization latency affects the continuity and quality of the video, especially in dynamic scenes? Are there mechanisms in place to mitigate negative impacts if synchronization is delayed?

2. How does the proposed model perform with high-motion sequences or videos requiring rapid scene changes, and what are the limitations of the current approach in handling such dynamics?  

3. The paper focuses on generating videos from a limited set of prompts and scenarios. How well does the approach generalize to a wider variety of video content or more complex scenes?
soundness: 3
strengths: 1. The integration of Clip parallelism and Dual-scope attention is a novel approach that effectively addresses the scalability and efficiency challenges in video generation.

2. The paper demonstrated ability to generate longer videos much faster than current methods, achieving substantial reductions in generation time.

3. Experiments are conducted to validate the performance, showcasing significant improvements over other methods in terms of speed and video length capabilities.
summary: The paper presents Video-Infinity, a novel framework for generating long-form videos using distributed diffusion models across multiple GPUs. This approach aims to reduces the inference time and resource demands typically associated with long video generation. The paper proposed two methods: Clip parallelism and Dual-scope attention, which optimize inter-GPU communication and temporal attention across frames, respectively. The methodology enables the generation of videos up to 2,300 frames in just 5 minutes,  faster than existing methods.
weaknesses: 1. The method of synchronizing context across GPUs, crucial for maintaining temporal coherence, is not discussed detail. 

2. While the framework improves efficiency, there is not much discussion on how these gains impact the qualitative aspects of the videos, such as resolution, realism, particularly under complex scene dynamics.