FIFO-Diffusion: Generating Infinite Videos from Text without Training
Authors: Jihwan Kim
Junoh Kang
Jinyoung Choi
Bohyung Han
Venue: NeurIPS 2024 poster (2024)
Keywords: generative models
diffusion
long video generation
tuning-free
Primary area: diffusion_based_models
TL;DR: Infinitely long video generation technique without training based on pretrained diffusion models

Abstract: We propose a novel inference technique based on a pretrained diffusion model for text-conditional video generation. Our approach, called FIFO-Diffusion, is conceptually capable of generating infinitely long videos without additional training. This is achieved by iteratively performing diagonal denoising, which simultaneously processes a series of consecutive frames with increasing noise levels in a queue; our method dequeues a fully denoised frame at the head while enqueuing a new random noise frame at the tail. However, diagonal denoising is a double-edged sword as the frames near the tail can take advantage of cleaner frames by forward reference but such a strategy induces the discrepancy between training and inference. Hence, we introduce latent partitioning to reduce the training-inference gap and lookahead denoising to leverage the benefit of forward referencing. Practically, FIFO-Diffusion consumes a constant amount of memory regardless of the target video length given a baseline model, while well-suited for parallel inference on multiple GPUs. We have demonstrated the promising results and effectiveness of the proposed methods on existing text-to-video generation baselines. Generated video examples and source codes are available at our project page.

FIFO-Diffusion: Generating Infinite Videos from Text without Training
Jihwan Kim*l
Junoh Kang" 1 Jinyoung Choi
Bohyung Han 1.2
Computer Vision Laboratory, EEC & IPAI, Seoul National University Ukjb26720,jumol junoh.kang, fan0.chosi. bhhan Gsnaancssn...k. ac.
2500
5000
7500
10000
"A spectacular fireworks display over Sydney Harbour, 4K, high resolution."
2500
5000
7500
10D00
(b) "An astronaut walking the moon's surface, high-quality, 4K resolution."
2500
2000
7500
10:00
(c) "A colony of penguins waddling Antarctic ice sheet, 4K, ultra HD."
Figure I: Illustration of 10K-frame long videos generated by FIFO-Diffusion based on a pretrained text-conditional video generation model, VideoCrafter2 [3]. The number at the top-left corner of each image indicates the frame index. The results clearly show that FIFO-Diffusion can generate extremely long videos effectively based on the model trained short clips (16 frames) without quality degradation while preserving the dynamics and semantics of scenes.
Abstract
We propose a novel inference technique based on a pretrained diffusion model for text-conditional video generation. Our approach, called OFFO-Diffsiion, concep- tually capable of generating infinitely long videos without additional training. This is achieved by iteratively performing diagonal denoising, which simultaneously processes a series of consecutive frames with increasing noise levels in a queue; method dequeues fully denoised frame the head while enqueuing random noise frame at the tail. However, diagonal denoising a double-edged sword as the frames near the tail can take advantage of cleaner frames by forward reference but such strategy induces the discrepancy between training and infer- ence. Hence, introduce latent partitioning to reduce the training-inference gap and lookahead denoising to leverage the benefit of forward referencing. Practically, FIFO-Diffusion consumes a constant amount of memory regardless the target video length given a baseline model, while well-suited for parallel inference on multiple GPUs. We have demonstrated the promising results and effectiveness of the proposed methods on existing text-to-video generation baselines. Generated video examples and source codes are available at our project page
"indicates equal contribution. https: http//njiiiaaa..oi//ea github. http:/ooojocts/tFoiiiiffff.ee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Diffusion probabilistic models have achieved remarkable success in generating high-quality images [8 23 18). On top of the success in the image domain, there has been rapid progress in the generation of videos p 22 37 31. Despite the progress, long video generation still lags behind compared to image generation. One reason is that video diffusion models (VDMs) often consider video as single 4D tensor with an additional axis corresponding to time, which prevents the models from generating videos at scale. An intuitive approach to generating long video is autoregressive generation, which iteratively predicts a future frame given the previous ones. However, in contrast to the transformer- based models [00 28), diffusion-based models cannot directly adopt the autoregressive generation strategy due to the heavy computational costs incurred by iterative denoising steps for a single frame generation. Instead, several recent works (9 7 29 [2 4| adopt a chunked autoregressive generation strategy, which predicts several frames in parallel conditioned on few preceding ones, consequently reducing computational burden. While these approaches are computationally tractable, they often leads to temporal inconsistency and discontinuous motion, especially between the chunks predicted separately, because the model captures a limited temporal context available in the last few few-only one or two in practice-frames
To address the limitation, we propose a novel inference technique, IIFO-DDifusion, which realizes arbitrarily long video generation without training based on a pretrained video generation model for short clips. Our approach effectively alleviates the limitations of the chunked autoregressive method by enabling every frame to refer to sufficient number of preceding frames. Our approach generates frames through diagonal denoising (Section 3.1 in a fiusstit- manner using a queue, which contains sequence of frames with different- monotonically mncreasing-noin levels over time. At each step, a completely denoised frame at the head is popped out from the queue while a new random noise image is pushed back at the tail. Diagonal denoising offers both advantage and disadvantage; noisier frames benefit from referring to cleaner ones while the model may suffer from training- -inference gap because video models generally trained to denoise frames with the same noise level. To overcome this trade-off and embrace the advantage of diagonal denoising, we propose latent partitioning (Section (.2) and lookahead denoising (Section (.3). Latent partitioning reduces training-inference gap by narrowing the range of noise levels in to-be-denoised frames and enables inference with finer steps. Lookahead denoising allows to-be-denoised frames to reference cleaner frames, thereby performing more accurate noise prediction. Furthermore, both latent partitioning and lookahead denoising offer parallelizability on multiple GPUs.
Our main contributions are summarized below.
We propose OFF--Difusioon through diagonal denoising, which is a training-free video generation technique for VDMs pretrained on short clips. Our approach denoises images with different noise levels for seamless video generation, enabling us to generate arbitrarily long videos.
We introduce latent partitioning and lookahead denoising, which respectively reduce the training-inference gap incurred by diagonal denoising and allow the reference to less noisy frames for denoising, improving generation quality.
FIFO-Diffusion requires a constant amount of memory regardless of the length of the generated videos given a baseline model. It is straightforward to run FIFO-Diffusion in parallel on multiple GPUs
Our experiments on four strong baselines, based on the U-Net 19 or DiT [6 architectures. show that degradation on quality over time.
generates extremely long videos including natural motion without
2 Vixttoo-Vieeo Diffusion Models
We summarize the basic idea of conditoniiional video generation techniques based on diffusion models. They consist of a few key components: an encoder Enc(-), decoder Dec(.), and a noise prediction network (- They learn the distribution of videos corresponding to text conditions, and the video is denoted by u E fx x3, where f is the number of frames and // W indicates the image resolution. The encoder projects each frame onto the latent image space and the decoder reconstructs the frame from the latent. A video latent 2g EEn(v) = =I=ji..:; =d] E / xhxwxe obtained by concatenating projected frames and the latent diffusion model is trained to denoise its perturbed version. 2,. For noise ~ I),, diffusion time step ~ U([L,.., T)), and a text

---

Timesteps T2
14
Fo
1
Dequeued
1+1
Frames
1+2
4+3
1+4
Input
Output
Enqueued
Diagonal denoising
Figure 2: Illustration of diagonal denoising with f = The frames surrounded by solid lines are model inputs while frames surrounded by dotted line are their denoised version. After denoising, the fully denoised instance at the top-right corner is dequeued while random noise is enqueued.
condition the model is trained minimize the following loss:
Ev,,, [Z); c, -
where the perturbed latent, 2; 8;20 dre, is obtained using predefined constants {s) and {o:)/-o- with the constraints 8q = 1, 0o and Ør // > 1.
Following time step schedule, 0 = To < rI < < Ts = T, initialized by a diffusion scheduler,
the model generates video by iteratively denoising [= i...; z/51 NN I) over S steps using sampler (--) such as the DDIM sampler. Each denoising step is expressed as
[2]] i.... 24, ,I {[21 :.... 2 Iroi..: 71), c;
(2)
where 2 denotes the latent of the 5h frame at time step Tt-
3 FIFO-Diffusion
This section discusses how FIFO-Diffusion generates long videos consisting of N frames using pretrained model only for f frames (f < N). The proposed approach iteratively employs diagonal denoising (Section 3.1] over predefined number of frames with different levels of noise. Our method also incorporates latent partitioning (Section 3.2) and lookahead denoising (Section 3.3)
to improve
the output quality of FIFO- FFFO-Diffusio based on diagonal denoising.
3.1 Diagonal denoising
Diagonal denoising processes series of consecutive frames with increasing noise levels depicted Figure To specific, for a time step schedule = To < TI <.. < Ty = T, each denoising step defined as
-., (112), :..:; 2 [ri....: T eg)-
Note that the latents along the diagonal, [z]] :...; = are stored a queue, Q. and diagonal denoising jointly considers the latents with different noise levels of (T.;...:T/) in contrast to the standard method specified in Equation 2 Algorithm in Appendix
C Cillutrrates how diagonal denoising FIFO- Diffusion works. After each denoising step with | i..;; 2,,1. the foremost frame is dequeued as it arrives at the noise level model generates frames in inst-in-first-our manner.
0, and the new latent at noise level Ty is enqueued. As result, the
Additionally, the initial diagonal latents :z! to initiate the diagonal denoising can be gener- ated from / random noises at time step Ty. similar to the the process described above. Notably, our approach does not require pregenerated videos or additional training for the initial latent construction. The detailed algorithm is presented in Algorithm in Appendix

---

Video frames
Video frames
Inference time
Inference time
(a) Chunked autoregressive
(b) FIFO-Diffusion
Figure 3: Comparison between the chunked autoregressive methods and FIFO-Diffusion proposed for long video generation. The random noises (black) are iteratively denoised to image latents (white) by the models. The red boxes indicate the denoising network in the pretrained base model while the green boxes denote the prediction network obtained by additional training.
FIFO-Diffusin takes f frames as input, regardless of the target video length, and generates an arbitrary number of frames by producing one frame per iteration using a sliding window approach. Note that generating N (>> f) frames for video requires (()) memory in each step (see Table 2
which is independent of N.
Diagonal denoising allows to generate consistent videos by sequentially propagating context to later frames. Figure
3 illustrates the conceptual difference between chunked autoregressive and FIFO-Diffusion. The former often struggles to maintain long-term context across chunks since their conditioning conlll the last generated frame lacks contextual information propagated from previous frames. In contrast, diagonal denoising progresses through the frame sequence with a stride of I, allowing each frame to reference a sufficient number of preceding frames during generation. This approach enables the model naturally extend the local consistency of a few frames to longer sequences. Additionally, FIFO-Diffusion requires no subnetworks or extra training, depending solely base model. This distinguishes it from existing autoregressive methods, which often require an additional prediction model or fine-tuning for masked frame outpainting.
methods 9 7 29
3.2 Latent partitioning
Although diagonal denoising enables infinitely long video generation, it introduces training- inference gap, as the model trained to denoise all frames at uniform noise levels. To address this, we aim to reduce noise level differences in the input latents by extending the queue length times (from f to n.f with n > 1), partitioning it into n blocks, and processing each block indepen- dently. Note that the extended queue length increases the number of inference steps. Algorithm In Appendix queue Q has diagonal latents [z__ i...; 1. We partition Q into n blocks, [Roi...: 2n-1]. of equal size f, then each block 2x contains the latents at time steps тą = [TKf+ij...; T(++1))] Next, we
C provides the procedure of FIFO-Diffusion with latent partitioning. Let
apply diagonal denoising to each block in divide-andd-onquer manner (See Figure (a)). At
O,...; n 1, each denoising step updates the queue as follows:
e₂ - ((2x. Tk. C: ca).
(4)
Latent partitioning offers three key advantages for diagonal denoising. First, it significantly reduces the maximum noise level difference between the latents from |o_., - to max; (0 [0 T The effectiveness of latent partitioning is supported theoretically and empirically by Theorem 3.3 and
Table respectively. Second, latent partitioning improves throughput of inference by processing partitioned blocks in parallel multiple GPUs (see Table Z). Last, it allows the diffusion process to leverage large number of inference steps, n.f (n 2 2), reducing discretization error during inference. We now show in Theorem 3.3 that the gap incurred by diagonal denoising is bounded by the maximum noise level difference, which implies that the error can be reduced by narrowing the noise level differences of model inputs.
Definition 3.1. We define z,! = [=t; 2/1, where z is the latent of the 1hh frame at time step t (noise level of a, ct for constant c). 2, ydm satisfies the following ODE from 11
dz) = c- cc-E(())dm t. 1)dt,
(5)
for 1 = [1:...; 1] and E(.) is the scaled score function -o V 1z

---

++7
-+4
inputs
denoising
instances
(a) Latent partitioning
++7
F3
2
++-
I+J
7%
1-1
(b) Lookahead denoising
Figure 4: Illustration of latent partitioning and lookahead denoising where f = 4 and n = 2. (a) Latent partitioning divides the diffusion process into parts to reduce the maximum noise level difference. (b) Lookahead denoising on (a) enables all frames to be denoised with an adequate number former frames at the expense of two times more computation than (a).
Lemma 3.2. Ef)) is bounded, then
||
21,1 = O(tt ss) for vi.
Proof.
Refer Appendix A.1
Theorem 3.3. Assume the system satisfies the following two hypotheses:
(Hypothesis 1) c(-) is bounded.
(Hypothesis 2) The diffusion model co is K K-Lipschitz continuous.
Then, for diagonal latents dia [z] j...; z4,1 and corresponding time steps dlag [TI i...; Tz]. |z digg c(2), T; 1)'ll (z) To 1) c(2)2 Ti 1) +01o,, on l) (6)
where the E.).)' and e(-)* are ith element of (.).) and e(.), and T1 <_. < TJ. In other words, the error introduced by diagonal denoising is bounded by the noise level difference.
Proof. The left-hand side of Equation (6) is bounded as:
||ea (z diay mingg e(z,".. T, 1)'1l
I| vdm To 11) + ||eo (z., To -1) -((z).. S llea ding diaey' Eg (z,,
vdm
vddm : -1)1l,,
by triangle inequality. Then, the first term the right-hand side satisfics the following inequality: | diag -diagss Es (z
vdm
1)'ll < [2[(2 ding dhiug (2 1)ll
<
(1
x Ir - Tel) = 0( onl),
#=1
which is from the Lipshitz continuity and Lemma (Hypothesis 2) in Appendix A.2
3.2 Furthermore, provide justification for
3.3 Lookahead denoising
Although our diagonal denoising introduces training-inferent inferrenceeeeeeeee gap, advantageous in another respect because noisier frames benefit from observing cleaner ones, leading to more accurate denois- ing. As empirical evidence, Tigure3]show the relative MSE losses in noise prediction of diagonal denoising with respect to the original denoising strategy. The formal definition of the relative MSE is given by
(x - ydiagg - E(2)dm Ta 1)
|oo (2))d T; 1)¹ c <(z)dm
vdm T; 1)
5

---

As depicted in Figure 4 (b), we estimate noise only for the benefited later half of the frames. In other words, we perform diagonal denois- ing with stride of f" = 1 11 updating only
1.20
VDM
1.2
1.15-
1.2 12-10
the last f' frames to ensure that each frame is denoised with reference
rclative MSE
sufficient number--
1,05
at least f' of of clearer frames. Precisely, for O...., 2n I, each denoising step updates the queue as
1.00
0.95-
c + (Ck, Tk, C; co) +1:f. +1:
(8)
0.85
17
25
-4
49
Algorithm in Appendix Goutlines the detailed procedure of LIFO-Diffusion with lookahead de- noising. We illustrate the effectiveness of looka- head denoising with the red line in Figure [ Except for few early time steps, lookahead denoising enhances the baseline models noise prediction performance, nearly eliminating the training-inferenee gap described in Section 3.2 Note that, this approach requires twice the computa- tion of the original diagonal denoising since we only update the half of the queue each step. However, the concerns about the additional computational overhead are easily addressed via parallelization in the same manner as latent partitioning (see Table(2)
timestep
T]
The relative MSE losses of the noise prediction of x²,, (see Equation 7 when 4. 'VDM' indicates the original denoising strategy reference line. 'LP' and 'LD' denote latent partitioning and lookahead denoising, respectively.
Figure 5:
4 Experiment
This section presents the examples generated by existing long video generation methods including FIFO-Diffusion, and evaluates their performance qualitatively and quantitatively. We also perform the ablation study verify the benefit of latent partitioning and lookahead denoising introduced in FIFO-Diffusion.
4.1 Implementation details
We implement FIFO-Diffusion based on existing open-source text-to-video diffusion models trained on short video clips, including three U-Net-based models, VideoCrafterl 2, VideoCrafter2 [3], and zeroscope as well as DiT-based model, Open-Sora Plar We employ the DDIM sampling 24 with 77 (0.5, 1}. Appendix B provides more details about our implementations.
For quantitative evaluation, we measure FVD12s (27) and IS [2T] scores using Latte 13 as a base model, which is DiT-based video model trained on UCF-101 101 26 We generate 2,048 videos with 128 frames each to calculate FVD12s. and randomly sample
f6-frame clip from each video to measure IS score, following evaluation guidelines in 233. To calculate computational cost, we adopt VideoCrafter2 as the baseline model, using DDPM scheduler with 64 inference steps on A6000 GPUs.
4.2 Qualitative results
We first evaluate the performance of the proposed approach qualitatively. Siguurealllstatteee examples of long videos (longer than 10K frames) generated by FIFO-Diffusion based on VideoCrafter2. It demonstrates the ability of FIFO-Diffusion to generate significantly longer videos than the target length of pretrained baseline models-16 16 frames in this case. The individual frames exhibit outstand- ing visual quality with no perceptual quality degradation even in the later part of the videos while preserving semantic information across all frames. Figure (a) and (b) present the generated videos with natural motion of scenes and cameras; the consistency of motion is effectively maintained by referencing earlier frames through the generation process.
Furthermore, Figure/o[c illustrates that FIFO-Diffusion can generate videos with extensive motion driven by sequence of changing prompts. The capability to generate multiple motions and seamless transitions between scenes highlight the practicality of our method. Please refer to Appendices and E for more examples and our project pagel for video demos, in comparisons with the videos from other baselines.
576

---

50
240
(a) 'a serene winter scene a forest. The forest is blanketed in thick layer snow, which..."
GO
120
180
240
(h) "A vibrant underwater scene of a scuba diver exploring a shipwreck, 2K, photorealistic..
20
40
80
8D
100
120
220
240
260
280
300
(c) "A tiger walking + standing - resting on the grassland, photorealistic, 4k, high definition"
Figure 6: Illustrations of long videos generated by FIFO-Diffusion based on (a) Open-Sora Plan and (b) VideoCrafter2, as well as (c) multiple prompts based on VideoCrafter2. The number on the top-left corner of each frame indicates the frame index.
Nriee
120
180
240
LaVie+SHINE Gen-L Gen-L-VViden Video
0
60
120
160
240
180
240
"An astronaut floating in space, high quality, 4K resolution."
Figure 7: Sample videos generated by (first) FIFO-DIffusion on VideoCrafter2, (second) FreeNoise on VideoCrafter2, (third) Gen-L--Vides VideoCrafter2, and (last) La SEINE. The number on the top-left corner of each frame indicates the frame index.

---

Table Comparisons of FVD12s and IS scores on UCF-101. FIFO-Diffusion with latent partitioning and lookahead denoising utilizes Latte [I3 as its baseline, where the number of partitions is four (n = 4). The FVD and scores of the other algorithms are obtained from their respective papers,
and PVDM 35 denotes PVDM-L. (400-400s).
FVD (4)
IS (t)
StyleGAN-V
1773.4
23.94=0.73
VIDM 14
1531.9
PVDM B5
648.4
74.40=1.25
FI Diffusion (ours)
596.64
74.44-1.17
Table 2: Memory usages and inference times of long video generation methods. FIFO-Diffusion utilizes latent partitioning with 4 and lookahead denoising.
Memory usage [MB] (4)
Inference time [s/frame] (L)
Method
128 frames
256 frames
512 frames
FreeNoise 17
26.163
44,683
memory
6.09
Gen-L Video
30
10,913
10,937
10,965
22.07
FIFO-Diffusion (I GPU)
11,245
11,245
11,245
12.37
FIFO-Diffusion (8 GPUs)
13,496
13,496
13,496
1.84
Figure 7 compares the results from FIFO- Diffusion with two training-free techniques,
FIFO-Diffusion Draw
FreeNoise
FreeNoise [17 and Gen-L-Video [30 based on VideoCrafter2, as well as training- rraseing-based chunked autoregressive method, LaVie [32
Overall Preference
60.2%
14.50%
25.25%
Plausibiliity of Motion
55
17:30%
20.70%
SEINE [ Note that the chunked autoregres- sive method requires two models: LaVie for T2V and SEINE for 12V. We observe that our method significantly outperforms the others in terms of motion smoothness, frame quality, and scene diversity.
Magnitude
of Mation
57.9%
18 49%
28.50%
Fidelity to Text
02.2%
23.78%
24.00%
Quality
18.02%
Among the training-free methods, Gen-L-Video often produces videos with blurred background while FreeNoise struggles to generate dynamic scenes The videos from LaVie + SEINE grad- ually degrade and diverge from text prompts due error accumulation in their autoregressive generation processes. Additionally, they often exhibit discontinuities between adjacent chunks, only the last frame of each chunk is employed to transfer contextual information to the next. Figures[T8] and and 19 in Appendix Appendix(Fjprovide pooovvdd further examples comparing these methods.
25%
50%
75%
The results of user study between FIFO- Diffusion and FreeNoise for five criteria.
Figure 8:
We also conduct user study to evaluate the long video generation performance of FIFO-Diffusion compared to an existing approach, FreeNoise. As shown in Figure g users expressed a strong preference for FiFO-Diffusion across all criteria, particularly those related to motion. Given that motion is one of the most defining characteristics of vidcos as opposed to images, the strong performance of FIFO-Diffusion in these criteria promising and highlights its potential generate more natural, dynamic videos. Details about the user study are provided in Appendix B.]
4.3 Quantitative results
We compare FIFO-Diffusion with the baselines trained for long video generation 23 terms of the FVD12s 2s and IS scores. As shown in Table our approach outperforms all the compared methods including PVDM-L (400-400s) (35), which employs a chunked autoregressive generation strategy. Note that PVDM-L (400-400s) iteratively generates 16 frames conditioned on the previous outputs over 400 diffusion steps while our approach only requires 64 inference steps (with lookahead denoising) without need for additional training.
14 35
in
WWe provide qauantitative evaluation the magnitude of motion in Appendix

---

20
40
80
DD
80
DD+LP4LD DD+LP
80
"A scenic cruise ship journey at sunset, ultra HD."
Figure 9: Ablation study. DD, LP, and LD signifies diagonal denoising, latent partitioning, and lookahead denoising, respectively. The number on the top-left corner of each frame indicates the frame index.
Table 3: Relative MSE losses of ablations. 'LP' and 'LD' denote latent partitioning and lookahead denosing, respectively.
of partitions
without I.D
with L.D
without LP with LP with LP
1.09
1.01
2
1.04
0.99
4
1.02
0.98
4.4 Computational cost
To evaluate computational efficiency, we assess memory usage and inference time per frame for training-frree, long video generation methods. As shown in Table 2 FIFO-Diffusion generates videos of arbitrary lengths with constant memory allocation, while FreeNoise requires memory proportional the target video length. Although Gen-L- Video maintains nearly constant memory usage, it exhibits the slowest inference speed due to redundant computations. Notably, leverages parallel computation; while incorporating lookahead denoising increases computational demand, utilizing multiple GPUs for parallel processing significantly reduces sampling time.
4.5 Ablation study
We conduct ablation study to analyze the effect of latent partitioning and lookahead denoising on the performance of FIFO-Diffusion. Sigureeishov that latent partitioning significantly improves both visual quality and temporal consistency of the generated videos. Moreover, lookahead denoising further refines the quality generated videos by facilitating temporal coherency and reducing flickering effects. The videos our project
demonstrate the benefit of EIFO-Diffusion. Additionally. Table 3 compares the relative MSE loss (see Equation (7) averaged over all time steps across different ablation settings. The results show that latent partitioning effectively reduces the training-inferent gap caused by diagonal denoising as the number of partitions increases. Furthermore, lookahead denoising enhances the model's noise prediction accuracy, achieving low relative MSE losses (below 1.0) when used in conjunction with latent partitioning.
5 Related work
This section discusses existing diffusion-based generative models for videos including long video generation techniques.
5.1 Video diffusion models
Diffusion models, originally developed for high-quality image synthesis, have become prominent approach in video generation 2 9 22 37 3T. VDM [9] modifies the structure of U-Net [19] and proposes 3D U-Net architecture to incorporate temporal information for denoising. On the
httpp:://jihhwa www.ann.iitub..... wttphhhb..oooroojee/////////./.. wttpprooeect///II---D

---

other hand, Make-A-Vides [22 employs a ID temporal convolution layer following 2D spatial convolutional layer to approximate 3D convolution. This design enables the model to capture visual-textual relationships by training spatial layers with image-text pairs before incorporating temporal context through 1D temporal layers. Recently, [16] introduce a transformer architecture, known as DIT, for diffusion models. Additionally, several open-sourced text-to-video models have emerged [31 2 32 3) trained on large-scale text-image and text-video datasets.
5.2 Long video generation
Long video generation approaches typically involve training models to predict future frames sequen- tially 29
6 I 4]. or generate a set of frames in a hierarchical manner [7] 34). For instance, Video LDM and MCVD 29] employ autoregressive techniques sequentially predict frames given several preceding ones, while FDM 66] and SEINE 44 generalize masked learning strategies for both prediction and interpolation. Autoregressive methods are capable of producing indefinitely long videos in theory, but they often suffer from quality degradation due to error accumulation and limited temporal consistency across frames. Alternatively, NUWA-XI. [34] adopts hierarchical approach, where a global diffusion model generates sparse key frames with local diffusion models filling in frames using the key frames as references. However, this hierarchical setup requires batch processing.
making it unsuitable for generating infinitely long videos.
There are few training- -free long video generation techniques. Gen-L-Video 30 treats a video as overlapped short clips and introduces temporal co-denoising, which averages multiple predictions for one frame. FreeNoise 17 employs window-band based attention fusion to sidestep the limited attention scope issue and proposes local noise shuffle units for the initialization of long video. FreeNoise requires memory proportional the video length for the computation of cross, limiting its scalability for generating infinitely long videos.
5.3 Diffusion models with latents of different noise levels
Recent studies have adopted diffusion models for sequence generation by leveraging sliding window approach with temporally varying noise levels [36] 20 These methods train diffusion models from scratch to accommodate latents with different noise levels, addressing tasks such as motion generation [36] and video prediction (20). However, training diffusion models from scratch introduces significant computational costs, especially for text-to-video generation tasks. In contrast, our approach is training-free inference technique based the standard diffusion models, trained on latents with uniform noise, for sequence generation within the sliding window framework. While (2 is implemented with nested loop to deal with two different axes corresponding to video frame index and diffusion time step, FIFO-Diffusion combines these two dimensions using a ID queue, improving efficiency with single loop.
6 Conclusion
We introduced
novel inference algorithm that enables the generation of infinitely long videos from text without tuning video diffusion models pretrained on short clips. Our approach achieves this by introducing diagonal denoising, which processes latents with increasing noise levels using queue in a first-nn-irst-oou fashion. While diagonal denoising presents a trade-off, we addressed its limitations with latent partitioning and leveraged its strengths with lookahead denoising, Together, these techniques allow FIFO-Diffusion to generate high-quality, long videos that maintain strong scene consistency and expressive dynamic motion. Although latent partitioning reduces the -inhiing-ifeeencee gap of diagonal denoising, the gap persists due to changes in the model's, 's input distribution. However, we believe that this gap could be addressed by integrating the diagonal denoising paradigm into the training phase, and the benefits of as well. We leave this integration as future work; aligning the training and inference environments can significantly enhance JIFOODiifusion's 's performance.
remains for training
A Acknowledgements
This work was partly supported by LG AI Research, and the Institute of Information & communi- cations Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2022-11220959 (No.2022-0-00959, (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making); NO.RS-2021- 11211343, Artificial Intelligence Graduate School Program (Scoul National University)].
10

---

References
[1] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[2] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXww:3310.19512. 2023.
[3] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. VideoCrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprini arXiv:2401.00044 2024.
[4] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. arXiv preprini urXiv:2310.20700 2023.
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurlPS, 2021.
[6] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In NeurIPS, 2022.
[7] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221, 2022.
[8] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[9] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In NeurIPS, 2022.
[10] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. CogVideo: Large-scale pretraining for text-to-video generation via transformers. In ICLR, 2023.
[11] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022.
[12] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In CVPR, 2023.
[13] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.0888 2024.
[14] Kangfu Mei and Vishal M. Patel. Vidm: Video implicit diffusion models. In AAAI, 2023.
[15] OpenAl. GPT-4 technical report. arXiv preprint 2333022303.08774 2023.
[16] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[17] Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. FreeNoise: Tuning-free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310. 169, 2023.
[18] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[19] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[20] David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. In ICML, 2024.

---

[21] Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, 2016.
[22] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Vides Text-to-video generation without text-video data. In ICLR, 2022.
[23] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In CVPR, 2022.
[24] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[25] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICI.R, 2021.
[26] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
[27] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprini arXiv: /8/2.1171, 2018.
[28] Ruben Villegas, Mohammad Babacizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. In ICLR, 2023.
[29] Vikram Voleti, Alexia Jolicoeur and Christopher Pal. MCVD: Masked conditional video diffusion for prediction, generation, and interpolation. In NeurIPS, 2022.
[30] Fu- Yun Wang. Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen- L-Video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. 18264, 2023.
[31] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. ModelScope text-to-video technical report. arXiv preprini arXiv:2308.06:: 2023.
[32] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang. Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiging Yang, Yuwei Guo, Tianxing Wu, Chenyang Si, Yum- ing Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin. Yu Qiao, and Ziwei Liu. LaVie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15103, 2023.
[33] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR- MSR-VTT: A large video description dataset for bridging video and language. In CVPR, 2016.
[34] Shengming Yin, Chenfei WU, Huan Yang. Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang. Jianlong Fu, Gong Ming, Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan. NUWA-XL: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv: 12346, 2023.
[35] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In CVPR, 2023.
[36] Zihan Zhang, Richard Liu, Kfir Aberman, and Rana Hanocka. Tedi: Temporally-entangled diffusion for long-term motion synthesis. In SIGGRAPH, 2024.
[37] Daquan Zhou, Weimin Wang, Hanshu Yan. Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magic Video: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211111018 2022.
12

---

Appendix
A Details for Lemma 3.2 and Theorem 3.3
A.1 Proof of Lemma 3.2
Lemma 3.2 If <({) is bounded, then
|| 2i1l O(lt s|) for any i.
Proof. Since e(-) is bounded, there exists some M > 0 satisfying ||{(-)] S M.
||
||z ydm اسm' 211l
c. c(2 vdm dull
c. d.Ile(zu u 1)lldul
M s/.
A.2 Justification (Hypothesis 2) of Theorem 3.3
We provide justification for the hypothesis, which the diffusion model is K-Lipschitz continuous. At inference, we can consider = E [0, BJ xexbxw and [min:: mmax] where a >> since is pixel values and we inference for such In appendix B.3 of [11, e(z, or) is given as the following:
c(2,0)= = V. N.N,,,
E,
where ,1, 12,-.. Yn are data points. Note that (N yr, I is twice differentiable and continuous, and N I2) Lipschitz continuous. Since (-) estimates c(-), assuming Lipschitz continuity can be justified.
for Be O. Therefore, the differential function of e(2, σ)) is bounded and
13

---

B Implementation details
We provide the implementation details of the experiments in Table We use VideoCrafter [2]. VideoCrafter2 B], zeroscope" Open-Sora Plar LaVie [32]. and SEINE 4 pre-trained models. zeroscope, VideoCrafter, and Open-Sora Plan are under CC BY-NC 4.0, Apache License 2.0, and MIT License, respectively. Except for automated results, all prompts used in experiments generated by ChatGPT-4 [15]. We empirically choose n 4 for the number of partitions in latent partitioning and lookahead denoising. Also, stochasticity y, introduced by DDIM [24], is chosen achieve good results from the baseline video generation models.
randomly
Table 4: Implementation details regarding experiments
Experiment
Model
Sampling Method
1
Prompts
Frames Resolution
MSR loss
VideoCrafterl
FIFO-Diffusion
0.5
200
320 x 512
(Figure 5 and Table
zeroscope
24
FIFO-Diffusion
0.5
320 x 576
VideoCrafter1
16
FIFO-Difusion
4
0.5
100
320 x 512
VideoCrafter2
16
FIFO- Diff-fffusoon
100~10K
320 x 512
Qualitative Result
Open-Sora Plan
FIFO-Diffusion
x
VideoCrafter2
16
FreeNoise
100
320 x 512
VideoCrafter2
16
Gen-L--Video
100
320 x 512
LaVie + SEINE
chunked autoregressive
512
VideoCrafter2
16
FIFO-Diffusion
30
100
320 x 512
User Study
LaVie
16
FreeNoise
30
100
320 x 512
VideoCrafterl
FIFO-Diffusion
512
256 x 256
Motion Evaluation
VideoCrafterl
16
FreeNoise
0.5
512
100
256 x 256
Ablation study
zeroscope
24
FIFO-Diffusin
(1,4) 0.5
100
320 x 576
B.1 Details for user study
We randomly generated 30 prompts from ChatGPT-4 without cherry-picking, and generated a video for each prompt with 100 frames using each method. The evaluators were asked to choose their preference (A is better, draw, or B is better) between the two videos generated by FIFO-Diffusion and FreeNoise with the same prompts, on five criteria: overall preference, plausibility of motion, magnitude of motion, fidelity to text, and aesthetic quality. A total of 70 users submitted 111 sets of ratings, where each set consists of 20 videos from 10 prompts. We used LaVie as the bascline for FreeNoise, since it was the latest model officially implemented at that time.
14

---

C Algorithms of FIFO-Diffusion
This section illustrates pseudo-code for FIFO-Diffusion with and without latent partitioning and lookahead denoising.
Algorithm FIFO-Diffusion with diagonal denoising (Section 3.1
Require: N, f. co (ec(.), +(-) j....
Input: i.... 24,), [ri.... Tyl. c
Output: v
T T [i...Ty
Q - [2), j...;
for i = to N do
Q ((
# Equation 3
= + Q.dequeue()
# Fully denoised frame
s.append(Decl)
2, I)
# New random noise
Lenqueue (2++/) end for return v
Algorithm 2 Initial latent construction (Section 3.1
Require: f. c-).,, Dec(-), (F-) Input: 21:f ~ (0, I), (oo
Output: [=] i...; 1
Tfim: Ty]
Q " 2
for i 1 to f do Q + (,EE))))))
Qudequeue =,, ~ (0, Q.enqueue(2))
# New random noise
+ (y--i... Tf is Tf 1- end for return
# Varying timestep
[z] [2] :...; 2fy1
15

---

Algorithm 3 FIFO- with latent partitioning (Section 3.22)
Require: N. f. co(-), Dec(-), $()). Dec)))))))
# if latent partitioning
Input: i- / [ris.- Tnfl. c
Output:
Tiimi Tnfl
Q
P j.. 20 rnf-
for i = to N do
for à = 0 to - do
# Parallelizable
Tk +- +1f+1:(++11
2s - @kI+1i(k+2))
2x (lk:: Tki cg end for
# Equation
Q Qui...; Qn-1]
+ (.dequeee)
n. (Dec(eddDee(())
-, sta 1
Qeeqqeee(2*11) (2+tr1) end for return
Algorithm 4 FIFO-Diffusion with lookahead denoising (Section 3.3)
Require: eo)),, Dec(-), Φ))), Input: [= =// 2nyf [Tii... /
# if latent partitioning
Output:
T1i Tny]
Q
2 j. 2n
2 j.... 2
# dummy latents are required
for i 1 to N do
21, + Q 5
for k to 2n do
# Parallelizable
Tk +- + +1:(k+2))
2, - -¹ ff (1+1)
o + + Cr,
# Equation
end for =
26 +1
W.append(Decl) (ec(2z)))
e +1 z
Q - 10
1:5
2 +1.f :...: 2 r'+1:f
Q + [Qo: Q +1:f :...; Q
+1:f
2n
(eequeeu
itnf IO,
Tu.
Q (2(tnf)
end for return
16

---

D Qualitative results of FIFO-Diffusion
In Figures T0
to 15 we provide more qualitative results with baselines, VideoCrafter2 3 VidcoCrafter1 21 zeroscope and Open-Sora Plar 9
D.1
VideoCrafter2
20
40
60
80
(a) "A colony of penguins waddling an Antarctic ice sheet, 4K, ultra HD."
80
80
(b) "A colorful macaw flying the rainforest, ultra HD."
(c) "A dark knight riding on a black horse on the glassland, photorealistic, 4k, high definition." *
20
80
80
(d) "A high- aath-aatttude view of a hang glider in flight, high definition, 4K."
20
40
60
80
(c) "A nigh-speed motorcycle race on track, ultra HD, 4K resolution."
20
60
8D
(f) "A horse race in full gallop, capturing the speed and excitement, 2K, photorealistic.
Figure 10: Videos generated by FIFO-Diffusion with VideoCrafter2. The number on the top left of each frame indicates the frame index.
httpss/wwwgiiiiieeeeeeereeeepeer 576 http::/ggttttu..ooo///e/uuaaaaug
17

---

(a) "A pair of tango dancers performing Buenos Aires, 4K, high resolution."
20
80
(b) "A panoramic view of a peaceful Zen garden, high-quality, 4K resolution."
20
40
E0
80
(c) "A paraglider soaring over the Alps, photorealistic, 4K, high definition."
20
80
80
"A scenic hot air balloon flight at sunrise, high quality, 4K."
(c) scenic hot air balloon flight over Cappadocia, Turkey, 2K, ultra HD."
20
8D
(f) "A school of colorful fish swimming in a coral reef, ultra high quality. 2K.
20
60
80
(g) "A spectacular fireworks display over Sydney Harbour, 4K, high resolution.""
Figure 11: Videos generated by FIFO-Diffusion with VideoCrafter2. The number on the top left of each frame indicates the frame index.
18

---

SD
(a) "A spooky haunted house, foggy night, high definition."
(b) "A time-lapse of a busy construction site, high definition, 4K. "
20
40
60
80
(c) "A vibrant underwater scene of scuba diver exploring a shipwreck, 2K, photorealistic.
20
40
80
(d) "A vibrant, fast-paced salsa dance performance, ultra high quality, 2K."
3D
(c) 'An astronaut floating in space, high quality, 4K resolution."
"
8D
(f) "An astronaut walking on the moon's surface, high-quality, 4K resolution."
20
4D
60
80
"A majestic lion roaring in the African savanna, ultra HD, 4K."
Figure 12: Videos generated by FIFO-Diffusion with VideoCrafter2. The number on the top left of each frame indicates the frame index.
19

---

D.2
VideoCrafterl
(a) "A kayaker navigating through rapids, photorealistic, 4K, high quality."
20
40
80
BO
(b) "A pair of tango dancers performing Buenos Aires, 4K, high resolution." "
40
60
(c) panoramic view of the Himalayas from drone, high definition, 4K.
(d) "A paraglider soaring over the Alps, photorealistic, 4K, high definition."
20
40
80
80
professional surfer riding a large wave, high-quality, 4K."
"A school colorful fish swimming coral reef, ultra high quality, 2K
20
40
60
60
(g) An exciting mountain bike trail ride through a forest, 2K, ultra HD."
20
40
BO
80
(h) "A vibrant coral reef with diverse marine life, photorealistic, 2K resolution."
Figure 13: Videos generated by FIFO- Diff-Diffusion with VideoCrafterl. The number on the top left of each frame indicates the frame index.
20

---

D.3 zeroscope
20
40
(a) 'A beautiful cherry blossom festival, time-lapse, high quality."
(b) "A close-up of tarantula walking, high definition."
20
40
(c) "A thrilling white water rafting adventure, high definition." "
(d) "A detailed macro shot of blooming rose, 4K."
20
40
8C
(e) "A panoramic view of the Milky Way, ultra HD."
(I) "A mysterious foggy forest at dawn, high quality, 4K."
20
40
8C
g "A scenic cruise ship journey at sunset, ultra HD."
(h) "A lone tree in a vast desert, sunset, high definition."
Figure 14: Videos generated by FIFO-Diflusion with zeroscope. The number on the top left of each frame indicates the frame index.
21

---

D.4
Open-Sora Plan
80
163
240
320
(a) "A quiet beach at dawn, the waves gently lapping at the shore and the sky painted in pastel hues."
80
160
240
320
(b) "A snowy forest landscape with dirt road running through it. The road is flanked...
80
160
240
320
(c) "The majestic beauty of waterfall cascading down a cliff into serene lake."
80
160
240
320
(d) "Slow pan upward of blazing oak fire in an indoor fireplace."
80
160
240
320
(c) "The dynamic movement of tall, wispy grasses swaying in the wind. The sky above is..."
160
240
320
(1) "a serene winter scene in a forest. The forest is blanketed in a thick layer of snow, which..."
Figure 15: Videos generated by EIFO-Difusion with Open-Sora Plan. The number the top left of each frame indicates the frame index.
22

---

E
Multi-prompts generation for FIFO-Diffusion
E.1
Method
For multi-prompts generation, we simply change prompts sequentially during the inference. To be specific. let ci...., C* be k prompts, and 0 = no < T2 < < ni are increasing sequence of integers. Then, we use prompt condition c; for (ni-1 + 1)th th ni iterations.
E.2
Qualitative results
In Figures 16 and 17|
we provide more qualitative results based VideoCrafter2.
20
80
12c
140
160
180
200
226
240
260
280
300
(a) "Ironman running - standing + flying on the road, 4K, high resolution."
20
40
60
80
120
128
140
160
180
200
220
240
260
280
(b) "A tiger walking- + standing + resting on the grassland, photorealistic, 4k, high definition"
20
120
140
160
180
200
220
240
280
280
300
(c) "A teddy bear walking- standing dancing the street, 4K, high resolution."
Figure 16: Videos generated by FIFO-Diffusion with three prompts. The number on the top left of each frame indicates the frame index.
23

---

20
40
80
120
140
(a) "A tiger resting - walking on the grassland, photorealistic, 4k, high definition"
10c
120
140
160
180
(b) "A whale swimming on the surface of the ocean > jumps out of water, 4K, high resolution.
20
60
80
10C
120
140
160
180
(c) "Titanic sailing through the sunny calm ocean stormy with lightning, 4K, high resolution." *
80
(d) "A pair of tango dancers performing + kissing in Buenos Aires, 4K, high resolution."
Figure 17: Videos generated by FIFO-Diffusion with two prompts. The number the top left of each frame indicates the frame index.
24

---

Qualitative comparisons with other long video generation methods
In Figures 18 and 19 we provide more qualitative comparisons with other longer video generation
methods, FreeNoise 177. Gen-L-Video (30], and Va 33 + SEINE 4.
40
Ours
Friee
Va Gen-L-Videp
40
20
40
80
80
"A vibrant underwater scene of scuba diver exploring shipwreck, 2K, photorealistic.
Ours
20
40
FreeNoise
Vic+SEENNE Gen-L-Videe Video
20
40
80
80
40
(b) "A panoramic view peaceful Zen garden, high-quality, 4K resolution."
Figure 18: Qualitative comparisons with other long video generation techniques, Gen-L- Videe, FreeNoise, and LaVie + SEINE. The number in the top-left corner of each frame indicates the frame index.
25

---

20
40
20
40
60
80
LaVie+SEINE Gen-L-Video Video Froe
40
80
(a) "A pair of tango dancers performing Buenos Aires, 4K, high resolution." "
40
Onrs
FreeNoise
LaVie+SEINE Gen-L Gen-L-VVieo Video
O
20
40
60
80
20
40
(b) "A spooky haunted house, foggy night, high definition." "
Figure 19: Qualitative comparisons with other long video generation techniques, Gen-L- Lideoo FreeNoise, and LaVie + SEINE. The number in the top-left corner of each frame indicates the frame index.
26

---

G Motion evaluation
We measure optical flow magnitudes (i.e. average of optical flow magnitudes) to compare the amount of motion between FIFO-Diffusion and FreeNoise, for the videos generated with randomly sampled prompts from the MSR-VTI 33] test set. Figure 20 illustrates that over 65% of videos generated by FreeNoise are located in the first bin, indicating significantly less motion compared to FIFO-Diffusion. In contrast, our method generates videos with a broader range of motion.
350
Phasio (CuPS)
Prccho
250
50
Aucrage Optical Filne Magnitoks
Figure 20: Comparison of optical flow magnitudes between FIFO-Diffusion and FreeNoise.
27

---

H Ablation study
In Figures and we conduct an ablation study investigate the effectiveness of each component in FIFO-Diftussie We compare the results of FIFO-Diffusion only with diagonal denoising (DD), with the addition of latent partitioning with n=4 (DD LP), and lookahead denoising (DD LP LD).
20
89
DD
DD+LP
DD4LP+LD
20
40
80
(a) "A panoramic view of the Milky Way, ultra HD."
DD
DD+LP+LD DD+LP
20
40
80
80
(b) "A scenic cruise ship journey at sunset, ultra HD."
DD
DD+LP+LD DD+LP
20
10
80
80
(c) "A beautiful cherry blossom festival, time-lapse, high quality."
Figure 21: Ablation study. DD, LP, and LD signifies diagonal denoising, latent partitioning, and lookahead denoising, respectively. The number on the top-left corner of each frame indicates the frame index.
28

---

D
20
40
80
DD
DD+LP+LD DD+LP
40
80
(a) "A detailed macro shot of blooming rose, 4K."
20
40
80
DD
DD+LP4LD DD+LP
20
40
80
"A close-up of a tarantula walking, high definition."
Figure 22: Ablation study. DD, LP, and LD signifies diagonal denoising, latent partitioning, and lookahead denoising, respectively. The number the top-left corner of each frame indicates the frame index.
Potential Broader Impact
This paper leverages pretrained video diffusion models generate high quality videos. The proposed method can potentially be used to synthesize videos with unexpectedly inappropriate content since it based on pretrained models and involves no training. However, we believe that our method could mildly address ethical concerns associated with the training data of generative models.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions and scope are accurately represented in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

We provide the full set of assumptions and a complete proof for Lemma 3.2 and Theorem B.3
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: YYs)
Justification:
We provide detailed information reproduce the results in Sections and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: We provide the source codes in supplementary files to reproduce the experi- mental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https i
//nips. //nips.cc/pul wwcpppblccggguiiieceeeeeeoooeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the detailed information for implementations in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
Error bars are not reported because would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
We provide the computational cost and type of GPUs for each experiment in Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::///eaurpppp.cc..paaalallea
Answer: [Yes]
Justification:
Our research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss the potential broader impacts in Appendix I
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Since we do not release of data or models, our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification: We properly credit the original owners and the licenses of the used assets in Appendix B
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.paapesstt
package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide README.md to explain the usage of the released code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We provide detailed information for user study in Appendix B.1
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
Our survey does not incur any potential risks.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "2SrT4LF1BL", "reviewer": "Reviewer_bkyd", "summary": "Given a “base” diffusion based video generation model, this paper shows how one can use “diagonal denoising” to generate arbitrarily long videos which have similar visual quality as the base model and obey conditioning.  Notably, the proposed method does not require any training which makes it applicable in principle to any video diffusion model.  The idea is to input a sequence of frames with a sequence of different noise scales — the denoiser is then trained to “map” these frames at noise levels [tau_1… tau_f] -> [tau_0 … tau_{f-1}].  Frames are kept in a queue — they are enqueued at noise scale tau_f and and by applying the denoiser iteratively on this queue they eventually reach noise level tau_0=0 at which point they are dequeued (and “done”).  Some tricks are proposed for making this basic idea more accurate (such as partitioning the noise scale range and applying the above trick separately for the different partitions.", "questions": "There is a recent paper on “Rolling diffusion” by Ruhe et al that seems related to this work.  I wonder if authors could comment on the relevance.", "limitations": "yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This method is a proof-of-concept that given a base video diffusion model capable of generating a short clip, we can always use that same model to generate arbitrarily long clips without needing to retrain.\n\nIt is not the first such paper (two competitors include Freenoise and Gen-L) but this paper is particularly simple compared to the other methods and outperforms Freenoise in user-studies (which outperforms Gen-L).  I’m admittedly less familiar with Freenoise and Gen-L, but I suspect that compared to these other papers, this approach may also be more architecture-independent as it seems applicable to any video diffusion model; moreover, compared to Freenoise, this approach does not require attending to all frames, so it runs in bounded memory.", "weaknesses": "Not a weakness per-se but a recommendation:\nOne thing that would make this paper stronger would be to include long-video FVD evaluations on datasets like UCF and Kinetics (see, e.g. the recent TECO paper).  This would establish a baseline of performance on those datasets showing a “theoretical maximum FVD” that can be achieved by using a model that was not trained explicitly on long videos and would be a valuable contribution to the community.\n\nAnother suggestion for readibility is to improve the description of the method in the introduction.  It is hard to understand why there would be a train-test discrepancy until Section 4 and I kept on thinking that the authors were over-claiming that it wasn’t necessary to train until I understood the details of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Long-video FVD Evaluation**\n\nWe appreciate your invaluable recommendation. Including long-video evaluations on specific datasets would definitely strengthen our paper by evaluating general performance of our techniques for video generation.\n\nIn response, we have evaluated the class-conditional long video generation performance on the UCF-101 dataset by computing the Frechet Video Distance (FVD) and Inception Score (IS). Specifically, we computed $FVD_{128}$ and IS following the details and implementation outlined in [R2], utilizing the model from [R3] pretrained on the UCF-101 dataset. The results are shown in ***Table A***.\n\n***Table A*** $FVD_{128}$ and IS on the UCF-101 dataset. We sample 128-frame, $256\\times 256$ resolution 2048 videos utilizing the model from [R3] pretrained on the UCF-101 dataset, sampling by FIFO-Diffusion. We also compute the IS of [R3] with DDIM 64 steps. Note that we cannot compute $FVD_{128}$ of [R3], as [R3] generates 16-frame videos.\n| Method | $FVD_{128} (\\downarrow)$ | IS $(\\uparrow)$ |\n|--|--|--|\n| Latte [R3] | - | 69.42±3.90 |\n| Latte + FIFO-Diffusion | 1476.5 | 72.87±2.97 |\n\n\n**[W2] Readability**\n\nThank you for your valuable suggestion, and we apologize for any confusion. We will add a brief explanation in the introduction to clarify the training-inference gap with diagonal denoising. This should help readers understand the rationale behind our approach earlier and avoid any misunderstanding. We appreciate your feedback and will make these adjustments to enhance readability.\n\n**[Q1] Relevance to Rolling Diffusion Models**\n\nThank you for bringing this to our attention. We gladly plan to cite Rolling Diffusion Models [R1], acknowledging its relevance to our work, that [R1] and diagonal denoising process latents with different noise levels in a sliding window manner. Furthermore, we have demonstrated differences between the two approaches in terms of training burden, sliding window methodology, and the target task in the global rebuttal. We kindly ask you to refer to the official comments for further details.\n\n[R1] Ruhe et al., Rolling Diffusion Models, ICML, 2024.\n\n[R2] Ivan et al., StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2, CVPR, 2022.\n\n[R3] Xin et al., Latte: Latent Diffusion Transformer for Video Generation, arXiv, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a17mpos005", "reviewer": "Reviewer_Tbwe", "summary": "**Paper Overview:**\n\nThis paper introduces the FIFO-diffusion pipeline designed for training-free autoregressive video generation. \n- The core method, ***diagonal denoising***, processes latents at varying denoising levels sequentially. \n- To enhance the video generation capabilities with more extensive denoising steps, the authors have implemented ***latent partitioning***, thereby increasing the denoise window size. \n- This modification, however, presents a challenge in maintaining consistency across video partitions, which the authors address through the approach of ***lookahead denoising***, albeit at the cost of doubling the computation.\n\n\n**Personal Reflection:**\n\nInitially, I was very intrigued by this paper. However, upon a detailed review, I noticed some areas that could benefit from improvement. While the foundational ideas are quite promising, it's important to note that the paper does not adequately cite previous works related to its core techniques.", "questions": "I am curious about how the rolling method initiates in the absence of historical video data. Could you explain how the process starts when there is no prior video latent available?", "limitations": "The authors have discussed the limitations of this work.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The paper is articulate and well-structured, making it accessible and informative.\n2. Visual aids like charts and figures effectively illustrate the achieved results and methodologies.", "weaknesses": "**Methods:**\n1. The diagonal denoising technique bears a close resemblance to the previously work [Rolling diffusion model](https://arxiv.org/abs/2402.09470) (released at 12th Feb 2024), which unfortunately is not cited, raising concerns about the novelty.\n2. The method section heavily focuses on latent partitioning and lookahead denoising. But I think there should be some simpler solutions to decoupling the denoising steps from the window size.\n3. The concept of lookahead denoising seems similar to strategies employed in [MultiDiffusion](https://arxiv.org/abs/2302.08113) techniques, specifically in managing noise overlap to maintain partition consistency. However, there is no citation of this work, which might be seen as an oversight in acknowledging related prior work.\n\n**Result:**\n1. The core mechanism, \"diagonal denoising,\" lacks sufficient experimental explanation, especially regarding its efficacy in training-free contexts.\n2. Table 1 compares efficiency and computational costs across different FIFO settings but fails to include video quality metrics. This omission makes it unclear which setting offers the best balance of performance and quality, raising questions about the advantage of using more GPUs if fewer can achieve faster results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Relevance to Rolling Diffusion Models**\n\nThank you for bringing this to our attention. We gladly plan to cite Rolling Diffusion Models [R1], acknowledging its relevance to our work. Please understand that we became aware of Rolling Diffusion Models [R1] after the ICML conference, which was subsequent to the submission. We kindly ask you to refer to the global rebuttal.\n\n**[W2] Latent Partitioning and Lookahead Denoising**\n\nThank you for your reviews, but we found it challenging to fully understand your concerns regarding the second weakness. Could you please provide more details or examples to help us address this effectively?\n\nFrom our understanding, you may feel our focus on latent partitioning and lookahead denoising is disproportionate to their importance. We would like to clarify that these two methods are crucial for generating high-quality long videos, especially in a training-free manner. While diagonal denoising introduces the concept of long video generation, latent partitioning and lookahead denoising address its limitations and leverage its advantages. The seamless integration of these methods significantly enhances the quality of video generation.\n\n**[W3] Relevance to MultiDiffusion**\n\nFirst, please let us clarify that lookahead denoising is not relevant to “maintaining consistency across video partitions” as you summarized in the paper overview and the third weakness. Lookahead denoising aims to make all frames take advantage of earlier, clearer frames, leveraging characteristics of diagonal denoising. Our first-in, first-out strategy naturally mixes up the partitions, thereby inherently maintaining consistency between them.\n\nWhile lookahead denoising might seem similar to strategies employed in MultiDiffusion [R4] due to the overlap, the two methods are fundamentally different in purpose and implementation. The overlap in [R4] is a strategy to maintain consistency of the independent outputs, resulting in the optimization process described in equation (3) of [R4]. In contrast, our method does not require joint alignment of the outputs. The overlap is simply a resultant phenomenon and does not necessitate any optimization process.\n\nAlthough the suggested methods are not directly relevant, both papers address similar challenges by proposing methods to handle out-of-range input sizes within their respective domains. With this in mind, we plan to cite [R4] in the future version. Thank you for bringing this to our attention.\n\n\n**[W4] Experimental Explanation of Diagonal Denoising in Training-free Context**\n\nWe would like to clarify that diagonal denoising is not primarily designed for \"training-free\" algorithms but rather for generating long videos with fixed memory allocation. The key components that facilitate the training-free context in our techniques are latent partitioning and lookahead denoising, which seamlessly integrate with diagonal denoising.\n\nWe have demonstrated the experimental performance of diagonal denoising alone in our ablation study, both quantitatively (Table 2 in Section 5.2) and qualitatively (Figures 20 and 21 in Appendix H). The results indicate that, in training-free contexts, diagonal denoising by itself predicts less accurate noise (Table 2) and generates less favorable videos (Figures 20 and 21) compared to the complete version of FIFO-Diffusion.\n\nWe did not focus solely on diagonal denoising in other experiments because we believe that the success of FIFO-Diffusion in a training-free context is due to the integration of latent partitioning and lookahead denoising.\n\n**[W5] Table 1 with quality metric**\n\nWe have provided qualitative and quantitative comparisons between different FIFO settings in our ablation studies (Section 5.4 and Appendix H), particularly in L289-291. We also encourage you to visit our project page (Section F. Ablation Study) to view the generated video samples. Table 1 is intended to demonstrate the fixed memory allocation and parallelizability of our method.\nFor your reference, we report $\\text{FVD}_{128}$ and IS on UCF-101 dataset in ***Table A***, following the recommendation from Reviewer bkyd.\n\n\n***Table A*** $FVD_{128}$ and IS on the UCF-101 dataset. We sample $256\\times 256$ resolution 2048 videos utilizing the model from [R3] pretrained on the UCF-101 dataset, sampling by FIFO-Diffusion with and without lookahead denoising. LD denotes the inclusion of lookahead denoising.\n| Method | $FVD_{128} (\\downarrow)$ | IS $(\\uparrow)$ |\n|--|--|--|\n| FIFO-Diffusion (n=4) | 1497.1 | 70.49±1.86 |\n| FIFO-Diffusion (n=4, with LD) | **1476.5** | **72.87±2.97** |\n\n**[Q1] Initialization of FIFO-Diffusion**\n\nWe initiate the FIFO-Diffusion process with corrupted latents generated by the baseline model, as discussed in L129-130 and L166-169 of our paper. To summarize, since latent partitioning requires $nf$ latents and the baseline model generates $f$ initial latents, we repeat the first frame of the initial latents to construct $nf$ latents.\n\nFIFO-Diffusion can be initiated even without prior video latent. You can start directly with $nf$ random noises and use a varying timestep schedule. Specifically, for $n=1$, the timestep at the first iteration would be $\\tau = [\\tau_f, \\ldots, \\tau_f]$, as all latents are random noise. Subsequently, the timestep will be updated at each iteration, such as $[\\tau_{f-1}, \\ldots, \\tau_{f-1}, \\tau_f]$, then $[\\tau_{f-2}, \\ldots, \\tau_{f-2}, \\tau_{f-1}, \\tau_f]$, and so on. You can decode the dequeued latent into a frame after the $f^\\text{th}$ iteration. This strategy can be similarly expanded for $n>1$ by substituting $f$ with $nf$.\n\n[R1] Ruhe et al., Rolling Diffusion Models, ICML, 2024.\n\n[R3] Xin et al., Latte: Latent Diffusion Transformer for Video Generation, arXiv, 2024.\n\n[R4] Omer et al., MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, ICML, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zQf04ZGZR5", "reviewer": "Reviewer_Hj7n", "summary": "The authors introduce FIFO-diffusion, a method to enable pre-trained video diffusion models to generate infinitely-long videos via the proposed diagonal denoising. In diagonal denoising, in contrast to the conventional denoising, the noise levels of the frames in a window form a decreasing sequence. This way, at each denoising step, the clean frame at the head of the queue can de dequeued and a new noisy frame can be enqueued, allowing for generating sequences longer than the ones the model was trained on. The authors additionally propose two techniques to further improve the quality of the generated videos:\n1) Latent partitioning to increase the size of the queue and hence decrease the training-inference gap of diagonal denoising.\n2) Lookahead denoising to allow the denoising process to benefit from the difference in the noise levels of the denoised latents and let the more noisy latents to leverage the information in the less noisy latents to reduce the prediction error.", "questions": "1) **Latent partitioning**: The motivation to do latent partitioning is to decrease the training/inference gap by making the timestamp differences within the window smaller. However, by increasing the size of the window by n the number of frames a certain latent interacts with throughout the denoising process also increases by the factor of n. This also could positively influence the temporal consistency of the generated videos. Therefore, it is not completely clear, whether the improved quality comes from the decreased training/inference gap or from the increased window size. Could the authors provide more details regarding this?\n\n2) **Lookahead denoising**: It is not clear from the paper, how exactly the ground truth score is obtained in eq. 7. Could the authors explain this better?\n\nIn the current form I am leaning to rejecting the paper, but I would love to raise my score, if the authors address my concerns (in Weaknesses and in Questions) in the rebuttal.", "limitations": "More clarification would be preferred in the limitations section (see Weaknesses). Are there other limitations? E.g. violation of some basic assumptions made in the paper?", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper considers a very interesting and challenging problem of enabling video generation models to generate long videos. The proposed method is simple enough to be directly applied to a variety of video generation models. The paper is well-written, all the components are well-motivated and supported by theory and experiments. The quality of the generated videos is good.", "weaknesses": "1) **Prior work**: Diagonal denoising has been first proposed in [1] for motion generation and in [2] for video generation. In both works the models are trained with the different noise levels support. Discussion with respect to these works is missing in the paper. I understand that by the submission time those works were only published on arxiv, but a discussion would facilitate a proper contextualization of the proposed method within the field.\n\n2) **Limitations**: The main limitation of the method, as discussed by the authors, is the training/inference gap. And the authors state that training with the support of different noise levels could improve the performance. However, if the gap didn't exist (given that there are models that train with different noise levels) the solutions that form the main contribution of the paper would not be needed. Hence, such a statement weakens the contributions.\n\n3) **Evaluation**: Quantitative comparisons are limited. The only quantitative comparison (except for human evaluation) is done through FVD and shows that the prior work (FreeNoise) performs better than the proposed method. Moreover, the FVD of FreeNoise is not only lower than the FVD of FIFO-Diffusion (also mentioned by the authors), but is also more stable, while there is a growing trend in the FVD of FIFO-Diffusion with time. This shows that the quality of the generated sequences degrades over time. The authors claim that this is due to the lack of motion in the videos generated with FreeNoise (line 454). Some additional metrics measuring both the temporal consistency and the motion magnitude would better illustrate the tradeoff.\n\n[1] Zhang et al., Tedi: Temporally-entangled diffusion for long-term motion synthesis, arxiv 2023.\n\n[2] Ruhe et al., Rolling Diffusion Models, arxiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Prior work: Relevance to Tedi and Rolling Diffusion Model**\n\nThank you for bringing this to our attention. Since we became aware of Tedi [R5] and Rolling Diffusion Model [R1] after the ICML conference, which was subsequent to the submission, we gladly plan to cite [R5] and [R1], acknowledging its relevance to our work.\n\nBoth [R5] and [R1] aim to train diffusion models from scratch, while our focus is on achieving this without additional training. To this end, we exclusively introduce latent partitioning and lookahead denoising in our paper to realize a training-free approach. Particularly in the text-to-video (T2V) generation domain, we believe that training-free approaches are extremely valuable. T2V models require substantial computational resources (GPUs and storage) for preparing text-video pair datasets and large-scale training. By avoiding additional training, our method provides a more accessible and efficient solution for generating high-quality videos.\n\nAdditionally, we have demonstrated differences between [R1] and our work in terms of sliding window methodology and the target task in the third and fourth paragraphs of the global rebuttal. We kindly ask you to refer to the global rebuttal for further details.\n\n**[W2] Limitations: Beyond Gap Bridging**\n\nThe benefits of both latent partitioning and lookahead denoising extend beyond addressing the training-inference gap. Lookahead denoising leverages the advantage of diagonal denoising, where noisier frames benefit from clearer frames. This benefit persists regardless of the gap due to the inherent similarity of neighboring video frames. As shown in Table 2, lookahead denoising enhances the baseline model's performance, suggesting its usefulness even without the gap.\n\nWhile latent partitioning aims to reduce the training-inference gap, it also offers two additional significant advantages: facilitating parallelized inference on multiple GPUs and reducing discretization error, as mentioned in L170-176. Although the first advantage may diminish without the gap, the remaining two benefits, particularly parallelizable inference, are crucial in practice for reducing inference time, as evidenced in Table 1.\n\n**[W3] Evaluation: Quantitative Results**\n\nThank you for your valuable comments. We believe that measuring both quantitative motion magnitude and FVD scores supports our claim in L454: “FVD is more favorable to algorithms that generate nearly static videos due to the feature similarity to the reference frames.”\n\nIn response, we measure optical flow magnitudes (average of optical flow magnitudes) to compare the amount of motion between 512 videos generated by FIFO-Diffusion and FreeNoise. We provide the result plot in the PDF file of our global comment. The plot (Fig. A) illustrates that over 65% of videos generated by FreeNoise are located in the first bin, indicating significantly less motion compared to FIFO-Diffusion. In contrast, our method generates videos with a broader range of motion.\n\nAdditionally, following the recommendation from Reviewer bkyd, we evaluate long video FVD (i.e. $FVD_{128}$) and IS on the UCF-101 dataset. We kindly encourage you to refer to response [W1] to Reviewer bkyd for detailed results.\n\nRegarding the growing trend in the FVD, we argue that this does not imply instability in FIFO-Diffusion as we have discussed in L449-451. Since FIFO-Diffusion starts the process from a queue with corrupted latents, generated by the baseline model (the reference set for the FVD scores), it is natural for the FVD to increase rapidly in the first 16 windows.\n\n**[Q1] Latent partitioning: Window Size and Performance**\n\nWe appreciate your insightful comments. We argue that the improved quality primarily stems from addressing the training/inference gap, while it might also be positively influenced by the increased window size. Although latent partitioning increases both the number of partitions ($n$) and the queue length ($nf$), the denoising process for each partition is conducted independently. This means that each frame only attends to temporal consistency within its partition at each step. Instead, increasing model capacity ($f$) would more directly enhance temporal consistency within the queue.\n\nHowever, it is a valid point that the model might benefit from increasing $n$. Since the first-in, first-out mechanism mixes the partitions at every iteration, all frames in the queue implicitly interact with each other throughout the denoising process. This interaction could indeed contribute to maintaining overall consistency in the queue, as you suggested.\n\n**[Q2] Lookahead denoising: About Equation 7**\n\nWe apologize if Equation 7 was unclear.  We use these representations for coherence with the model output, $\\epsilon_\\theta$. Similar to training loss in general diffusion models, we directly add noise sampled from a normal distribution into the latents and set this added noise as the ground truth for the score. Thank you for pointing this out, we will revise the equation in the future version.\n\n[R1] Ruhe et al., Rolling Diffusion Models, ICML, 2024\n\n[R5] Zhang et al., Tedi: Temporally-entangled diffusion for long-term motion synthesis, arXiv, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iPCVjPSiMf", "reviewer": "Reviewer_uUJo", "summary": "This article proposes a training-free diagonal denoising method aimed at extending video diffusion models, which typically generate only short clips, to the production of long videos. The authors have validated their method across multiple baselines and demonstrated that it can be extended to different VDM architectures.", "questions": "The method presented in this article is straightforward and provides some insights. The biggest concern might be that the actual results are very poor, with so-called long videos consisting mostly of simple, repetitive actions. At this stage, I can only give a borderline reject score, while looking forward to the opinions of other reviewers.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The motivation for this work is commendable; generating long videos is a worthwhile topic to research, as past VDMs could only produce short videos of a few seconds.\n- Viewing denoising long videos as a FIFO-Queue is interesting. From the paper, it appears to work to some extent.\n- The discussion of related works is comprehensive. The overall description of the method is understandable and should be reproducible with some experience in video diffusion.", "weaknesses": "- What do the authors think about training models for longer video generation? SORA can generate videos up to 1 minute with remarkable temporal and even 3D consistency. An important rule in deep learning is scaling up. Methods that still work well or are meaningful when computation resources are upgraded are most valuable. The currently available video generation models are still not satisfactory. Applying them for training-free applications seems only to make the results worse and unsatisfactory.\n- Although the videos presented in the supplementary are long compared to current video generation models, most of them are not attractive enough. In most cases, there are just random motion alterations without any order or controllability. In this way, it seems not enough for practical usage. The most important reason is that you never really have a global perception of the generation process, and therefore the motions and contents tend to be repeated or meaningless alterations. I believe the authors should try to add additional controllability to their method. Otherwise, it is going to be not usable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[W1] Scaling-up and FIFO-Diffusion**\n\nThank you for your insightful comments. We agree that scaling up in deep learning is crucial, particularly for video generation. However, we argue that FIFO-Diffusion offers benefits even for scaled-up models as an universal inference technique, and may aid in scaling up dataset sizes.\n\nFirst, FIFO-Diffusion is an inference method applicable to a wide range of video diffusion models regardless of architecture (e.g., DiT or UNet), as shown in Figure 6. This implies that any future scaled-up model with longer and advanced video generation capabilities would also benefit from our technique by generating scalable lengths of videos than their original capacity.\n\nFurthermore, this expandability might reduce the burden of preparing datasets. With the scalability of FIFO-Diffusion during inference, those who want to generate extremely long videos can utilize much shorter video datasets. Given that the number of sufficiently long videos is much smaller than shorter ones, FIFO-Diffusion provides a valuable alternative for preparing highly long video datasets.\n\n**[W2, Q1] Controllability and Random Motion**\n\nWe believe the results shown in Figure 6 (c) and Appendix E address these concerns. The multi-prompt results demonstrate the controllability and capability of our technique for generating long-term ordered motion using varying text prompts. Our method effectively maintains visual consistency by keeping sufficient forward frames in the queue, enabling seamless motion transitions across prompt changes. This approach adds the desired level of control and order to the generated videos, making them more suitable for practical applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

**1. Relevance to Rolling Diffusion Models**

We became aware of Rolling Diffusion Models [R1] after the ICML conference, which was subsequent to the submission. We gladly plan to cite [R1], acknowledging its relevance to our work, as both techniques process latents with different noise levels in a sliding window manner. We will discuss the differences between the two works: burden of training, the sliding window mechanism, and the target task.

First, [R1] and our work aim to realize each technique from different perspectives: training-based and training-free. [R1] trains diffusion models from scratch to simultaneously process frames with different noise levels, while our work focuses on leveraging promising pretrained models. In this sense, we introduce latent partitioning and lookahead denoising that effectively mitigate the training-inference gap and exploit the benefits from diagonal denoising. Especially in text-to-video generation, training-free approaches are much more valuable in terms of saving computational resources. Our method offers a more accessible and efficient solution for high-quality video generation.

Second, there are clear discrepancies regarding the ‘sliding’ implementation. [R1] introduces two independent iteration axes (global and local diffusion time) during inference, corresponding to window shifting and the diffusion timesteps, respectively.  Consequently, [R1] adopts a double nested loop for sampling (Algorithm 2 in [R1]) and requires $NT$ iterations to generate $N$ frames. In contrast, our method employs an efficient entangled axis corresponding to both window shifting and diffusion steps, resulting in $N$ iterations to generate $N$ frames. While both works share the key concept ‘sliding window’, their implementations of sliding are substantially different.

Lastly, the two works address distinct areas within the video generation domain: video prediction and text-to-video. [R1] solely focuses on a video prediction task, extending the existing videos from the datasets. However, our goal is to facilitate text-to-infinite video generation. Our work does not require any existing videos or images from the dataset, relying solely on text for inference.

By clarifying these points, we hope to address your concerns and highlight the unique contributions of our approach.

**2. Motion Metric Comparison between FIFO-Diffusion and FreeNoise**

To address the third weakness of Reviewer Hj7n, we provide a figure of plot that compares the amount of motion between FIFO-Diffusion and FreeNoise in attached PDF file.

[R1] Ruhe et al., Rolling Diffusion Models, ICML, 2024.

Meta-review: This paper introduces a novel, training-free approach to generate arbitrarily long videos using a "diagonal denoising" technique applied to pretrained diffusion-based text-to-video generation models. The paper is well-written and clearly structured, making it both accessible and informative.

The reviewers generally appreciated the clarity and organization of the paper, but several concerns were raised during the review process. One common point of discussion was the missing citation of related work, specifically "rolling diffusion." The authors addressed this in their rebuttal, clarifying the differences between their approach and "rolling diffusion." The authors also noted that "rolling diffusion" can be considered concurrent work, as it was not formally published before the NeurIPS submission deadline. I agree with this assessment, and the explanation provided by the authors is satisfactory.

Two reviewers questioned the necessity of using latent partitioning and lookahead denoising to address the training-inference gap in diagonal denoising. After the rebuttal, one reviewer found the authors' response convincing, while the other did not provide further feedback.

The reviewer who recommended rejection primarily challenged the paper on two fronts. First, they questioned the motivation for pursuing training-free long video generation given the current limitations of video generation models. Second, they criticized the quality of the generated videos, noting that they often consist of simple, repetitive actions. However, it's important to recognize that the proposed method is orthogonal to the baseline model's quality. Furthermore, the multi-prompt results shown in Figure 6(c) and Appendix E demonstrate that the proposed method has the capability to control motion and generate long-term ordered sequences using varying text prompts.

While I acknowledge that the generated videos still exhibit flaws, such as issues with object motion and flickering effects, I believe the exploration of training-free methods for long video generation is a valuable contribution that could bring interesting insights to the community. The potential for this approach to inspire future research in this area outweighs the current limitations.

Given the novelty of the approach and its potential impact on the field, I have decided to accept this paper.
Decision: Accept (poster)