FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling
Authors: Haonan Qiu
Menghan Xia
Yong Zhang
Yingqing He
Xintao Wang
Ying Shan
Ziwei Liu
Venue: ICLR 2024 poster (2024)
Keywords: diffusion
video diffusion
video generation
tuning-free
Primary area: generative models
TL;DR: A tuning-free and time-efficient paradigm for longer video generation based on pretrained video diffusion models

Abstract: With the availability of large-scale video datasets and the advances of diffusion models, text-driven video generation has achieved substantial progress. However, existing video generation models are typically trained on a limited number of frames, resulting in the inability to generate high-fidelity long videos during inference. Furthermore, these models only support single-text conditions, whereas real-life scenarios often require multi-text conditions as the video content changes over time. To tackle these challenges, this study explores the potential of extending the text-driven capability to generate longer videos conditioned on multiple texts. 1) We first analyze the impact of initial noise in video diffusion models. Then building upon the observation of noise, we propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models while preserving content consistency. Specifically, instead of initializing noises for all frames, we reschedule a sequence of noises for long-range correlation and perform temporal attention over them by window-based fusion. 2) Additionally, we design a novel motion injection method to support the generation of videos conditioned on multiple text prompts. Extensive experiments validate the superiority of our paradigm in extending the generative capabilities of video diffusion models. It is noteworthy that compared with the previous best-performing method which brought about 255% extra time cost, our method incurs only negligible time cost of approximately 17%. Generated video samples are available at our website: http://haonanqiu.com/projects/FreeNoise.html.

Published as a conference paper at ICLR 2024
FREENOISE: TUNING-FREE LONGER VIDEO DIFFUSION VIA NOISE RESCHEDULING
Haonan Qiu Menghan X Yong Zhang", Yingqing He Xintao Wang', Ying Shan? Ziwei Liu
"Nayyang Technological University
"Tencent AI Lab 'Hong Kong University of Science and Technology
ABSTRACT
With the availability of large-scale video datasets and the advances of diffusion models, text-driven video generation has achieved substantial progress. How- ever, existing video generation models are typically trained on a limited number of frames, resulting in the inability generate high-fidelity long videos during in- ference. Furthermore, these models only support single-text conditions, whereas real-life scenarios often require multi-text conditions the video content changes over time. To tackle these challenges, this study explores the potential of extend- ing the text-driven capability to generate longer videos conditioned on multiple texts. 1) We first analyze the impact of initial noise in video diffusion models. Then building upon the observation of noise, we propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models while preserving content consistency. Specifically, instead of initializing noises for all frames, reschedule sequence of noises for long- range correlation and perform temporal attention over them by window-based fu- design a novel motion injection method to support the generation of videos conditioned on multiple text prompts. Extensive experiments validate the superiority of our paradigm in extending the generative capabilities of video diffusion models. It is noteworthy that compared with the previous best- performing method which brought about 255% extra time cost, our method incurs only negligible time cost of approximately 17% Generated video samples are available our website: http://hoonnguucompppseeseereeep
sion. 2) Additionally,
INTRODUCTION
Diffusion models bring breakthrough developments in image generation (Rombach et al. 2022) abling users without any art background to easily create unique and personalized designs, graphics, and illustrations based on specific textual descriptions. Building upon this success, there a grow- ing interest in extending this concept to video generation (He et al. 2022 Ge et al. 2023 Blattmann al. 20033, Wang et al., 2023c Luo et al., 2023). As targeting modeling higher dimensional data, video diffusion model demands a notably increased requirement in model capacity and data scale. As result, current video diffusion models are generally trained on small number of frames. Consequently, during the inference stage, the quality of the generated video tends to decrease as the length of the video increases due to longer videos are not supervised during the training stage. One straightforward approach is generate video fragments of the same length the training videos and then stitch them together, climinating the muining-inferene in disconnected and incoherent fragments. To address this issue, the fragments can be fused during the denoising process and smoothly connected in the final video (Wang et al. 2023a). However, the long-distance fragments often have large content gap fuse and thus struggles to maintain the content consistency in the long video. Although some auto-regressive-basse methods (Villegas) al. 2022) get rid of this problem by progressively generating the next frame, content consistency is still hard to guarantee due to the error accumulation.
gap. However, this method results
"Corresoonding Authors

---

Published as a conference paper at ICLR 2024
In McdoLDMM(Blattaa et al. 2023). the generated frame depends not only the initial noise for the current frame but also on the initial noises for all frames. This means that resampling the noise of any frame will significantly influence other frames due to the full interaction facilitated by the temporal attention layers. This makes it challenging to introduce new content while maintaining the main subjects and scenes of the original video. To address this challenge, we inspect the tempo- ral modeling mechanism of VideoLDM, where the temporal attention module is order-independent, whereas the temporal convolution module is order-dependent. Our experimental observation indi- cates that the per-frate per-frame noises serve as a foundation for determining the overall appearance, while their temporal order influences the content built upon that foundation. Motivated by this, we pro- pose FreeNoise, tuning-free and time-efficient paradigm to achieve longer video inference. The key idea to construct a sequence of noise frames with long-range correlation and perform tem- poral attention over them by the way of window-based fusion. It mainly contains two key designs: Local Noise Shuffling and Window Based Attention Fusion. By applying the local noise shuffling to sequence of fixed random noise frames for length extension, we achieve sequence of noise frames with both internal randomness and long-range correlation. Meanwhile, the window-based attention fusion enables the pre-trained temporal attention modules to process frames of any longer length. Particularly, the overlapped window slicing and merging operation only happens in temporal attention while introducing no computation overhead to other modules of the VideoLDM, which benefits the computational efficiency significantly.
In addition, most video generation models (Blattmamn et al. 2023 Luo et al., 2023 Ge al. 2023) only utilize a single-text condition to control the video even when multi-text conditions are given. For instance, the sentence "A man sleeps on the desk and then reads the book" which contains two stages but only one condition will be reflected in the generated video. This limitation arises from the fact that the training dataset usually contains only a single-text condition. However, in single-shot scene, the main subject usually involves multiple actions. To address the challenge of generating videos based on multiple prompts without tuning the pretrained models, we propose Motion Injection. This approach leverages the characteristics of diffusion models, where different time steps recover varying levels of information (image layout, shapes of the objects, and fine visual details) during the denoising process (Patashnik et al. 2023 Zhang al. 2023). It gradually injects new motion during the time steps associated with object shapes, following the completion of the previous motion. Importantly, this design does not introduce any additional inference time.
Our contributions are summarized as follows: 1) We investigate the temporal modeling mechanism of video diffusion models and identify the influence of initial noises. 2) We design tuning-free paradigm for longer video generation, which outperforms existing state-of-the-ar notably in both video quality and computational efficiency. 3) We propose an effective motion injection approach that achieves multi-prompl long video generation with decent visual coherence.
2
RELATED WORK
2.1 VIDEO DIFFUSION MODELS
Latent Diffusion Models (LDM). Diffusion models (Sohl-Dickstein et al. 2015 Ho et al., 2020)
are generative models that formulate a fixed forward diffusion process to gradually add noise to the data æg ~ p(xo) and learn denoising model to reverse this process. The forward process contains T timesteps, which gradually add noise to the data sample æn to yield æ, through parameterization trick:
(( Na,;;;; V1- 3ar,
(( = N(,;; Varao, (1 - (1-āz))) r) (1)
where 3e is a predefined variance schedule, / is the timestep, ãr = -i 0s and ot = 1 - 3r The reverse denoising process obtains less noisy data #:-1 from the noisy input a at each timestep:
Po (xt-1 xx) = N (x4-1: (x,, t), Eg (xx,, t))
(2)
Here Asg and Eg are determined through noise prediction network eo (æ,, l), which is supervised by the following objective function, where é is sampled ground truth noise and 0 is the learnable network parameters.
min Eq, ||c 1 co (zi, t1),
(3)
2

---

Published as a conference paper at ICLR 2024
Once the model is trained, we can synthesize a data Ty from random noise æT by sampling z iteratively. Recently, to ease the modeling complexity of high dimensional data like images, Latent Diffusion Model (LDM) (Rombach et al. 2022) is proposed to formulate the diffusion and denoising process in a learned low-dimensional latent space. is realized through perceptual compression with an autoencoder, where an encoder & maps to E Rg HxW to its latent code Zn E [Rx xw" and
a decoder 'D reconstructs the image w6 from the Zo. Then, the diffusion model operates on the image latent variables to predict the noise é.
Zu = E (Do)
By = D (zu) = To. ?= co (Z1,, y,t),
(4)
The network a sequence of the following layers, where h represents the hidden feature a certain layer and y denotes conditions like text prompts. Conv and ST are residual convolutional block and spatial transformer, respectively.
h' ST(CCov (h, t). y), ST = Projmn o (Attngetr o AttMeross MMLP) o Projout (5)
Video Latent Diffusson Model (VideoLDM) (Blattmann et al. 2023 extends LDM to video gen- eration and trains a video diffusion model in video latent space. The 2u E =='*Nx''>W> becomes 4 dimensions, and 0 consequently becomes temporal-aware architecture consisting of basic layers as the following equation, where Tconv denotes temporal convolutional block and TT denotes temporal transformers, serving as cross-frame operation modules.
h' = TTSSS((Coov(CCoT) t)). y)), TT = Projm. o (Attnemmp o Attnemmp o MLP) Projeut
Following the same architecture, some similar text-to-video models have been proposed (Blattmann jt al. 2023, Wang et al. 2023b), primarily differing training strategies or auxiliary designs (such as fps conditioning, image-video joint training, etc.). Align YourLLantt (Blattmann al. 2023 is
designed to train only the temporal blocks based on a pre-trained text-to-image model (i.e.. Stable Diffusion (SD) RRobacch et al. 2022)). In contrast, ModelScope (Wang et al. (023b) is proposed for fully training the entire model with a SD checkpoint pre-loaded.
2.2 LONG VIDEO GENERATION.
Generating long videos poses challenges due the increased complexity introduced by the temporal dimension, resource limitations, and the need to maintain content consistency. Many GANebased methods (Skorokhodov al. 2022 Brooks et al. 2022 Ge al. 2022) and diffusion-based meth- ods (Harvey et al. [2022] Voleti et al. 2022 Yu al. 2023 He et al. 2022 Yin et al. 2023 Ho et al.) 2022) are proposed generate long videos. Despite their advantages, those approaches necessitate extensive training on large long video datasets. Recently, tuning-free method, Gen-L-Vided (Wang al 2023a) is proposed and successfully extends the video smoothly by merging some overlapping sub-segments into smoothly changing long segment during the denoising process. However, their content consistency lacks preservation due to the large content gap among those sub-segments. Ben- efiting from the design of noise rescheduling, our paradigm FreeNoise preserves content consistency well in the generated long videos. Meanwhile, Gen-L-Video costs around 255% extra inference time while FreeNoise only costs 17% additional inference time approximately. Another demand in long video generation is multi multi-prompt control, as a single-text condition is often insufficient to describe content that evolves over time. While some recent works (Yin al. 2023 He al., 2023 Wang et al. 2023a) have explored this direction, they introduce a new lens when a new prompt is provided. Phenaki (Villegas et al. (022) utilizes an auto-regressive structure to generate one-shot long videos under multi-text conditions but suffers from noticeable content variation. In our paradigm, we can generate multiple motions while preserving the main subjects and scenarios.
METHODOLOGY
Given a VideoL.DM pre-trained on videos with a fixed number of Ntrain frames, our goal is to gen- crate longer videos (e.g., .M frames where M > Ntrain) without compromising quality by utilizing it for inference. We require the generated M video frames to be semantically accurate and tempo- rally coherent. In the following sections, we will first study the temporal modeling mechanism that challenges VideoLDM in generating longer videos. Subsequently, we will introduce our efficient, tuning-free free approach to overcome these challenges. To further accommodate multi molti-proompt settings, propose motion injection paradigm ensure visual consistency.
3

---

Published as a conference paper at ICLR 2024
(a) Inference with E
(b) Inference with [c,,
(c) Inference with E2
(d) Sliding window inference with [E,,62]
Figure I: Challenges of longer video inference. The random noises and have the same number of frames as the model was trained on. All the results are generated under the same text prompt: "a man is boating on a lake".
3.1 OBSERVATION AND ANALYSIS
Attentive-Scope Sensitivity. For longer video generation via VideoLDM, straightforward lution is to feed M frames random noises to the model for video generation through iterative denoising steps. Unfortunately, it fails generate desired result, as the example illustrated in Fig- ure (b). The reason is easy to understand: the temporal attention modules perform global cross- frame operations that make all frames attentive each other, however they are strictly trained to attend on Ntrain neighbor frames and struggle to handle more frames properly. In this case, the generated videos tend cause semantic incompleteness or temporal jittering.
Noise-Induced Temporal Drift. To bypass the issue above, one may argue to employ temporal sliding windows so that the tem- poral attention module can always process fixed number of frames. Indeed, this solution makes desired content with smooth tempo- ral transition. However, it struggles to main- tain the long-range visual consistency, ampled in Figure dd.. To identify the under- lying causes, explore the temporal model- ing mechanism that consists of two kinds of cross-frame operations: temporal attention and temporal convolution. Temporal attention is order-independent, whereas temporal convolu- tion is order-dependent. When temporal con- volutions are removed, the output video frames hold strict correspondence with the initial noise frames, irrespective of shuffling. In con-
w/o Tconv
w Tconv
ex-
(a) Inference with initial noise frames
Tconv w/o Tconv
trast, depending on the noise frame order, the temporal convolution introduces new content to ensure the output video's 's temporal continuity. Figure 2 demonstrates such a phenomena. It implies the conjecture that the per-frame noises serve as a foundation for determining the over- all appearance, while their temporal order influences the content built upon that foundation. So, is challenging for the temporal modules to achieve global coherence when independently sampled noises are combined for longer video generation.
(b) Inference with shuffle(E)
Figure 2: Case study on temporal modeling. For the 'w/o Teonv' results in (a) and (b), the frames corresponding to the same initial noise are marked with bottom lines of the same color.
3.2 NOISE RESCHEDULING FOR LONG-RANGE CORRELATION
To circumvent the challenges mentioned above, we propose a noise rescheduling paradigm for longer video inference. The key idea is to construct sequence of noise frames with long-range correlation and perform temporal attention over them by the way of window based fusion. To gain semantically meaningful and visually smooth videos, the model inference should satisfy two basic
4

---

Published as a conference paper at ICLR 2024
Noise Rescheduling
Iterative Denoising
Denoising U-Net
Decoder
Extending
Spatial CA
Temp
Noise (Meroin frames)
ZT
Z
7
Fusion
I L and (t 2Tarrt2 1> or (TaS<<
Multi-prompt based motion injection paradigm
Sliding window based attention fusion
Figure 3: Overview of our proposed method. Given Ntrain frame of random noise, first extend it to the target M frames as the initial noise 2T through noise rescheduling. Then, in the iterative denoising process, the multi-prompt injection paradigm is conducted in the spatial cross-attention layers (where t denotes timestep. denotes layer number, P denotes text prompt) and the sliding window based attention fusion is performed in temporal self-attention layers.
requirements: (i) the temporal attention only accepts fixed Nirain frames, to bypass the attentive- scope sensitivity issue; (ii) every Nirain frames of features fed to the temporal attention always correspond to Ntrain frames of independent and identically distributed noises, otherwise the gener- ation fails because of the out-o--distribution input. Specifically, we propose two effective designs to achieve this goal.
Local Noise Shuffle Unit. To acquire a vidco with M frames (M > Nirain), we initialize Ntroin frames of random noise [e1, E2..... Nerain independently and reschedule them for the remaining length:
[EI, 62, Nerain shuffle(es, E2),..,E)),..... shuffle(E) S(142)) (8(44)):
(7)
where S denotes the size of the local shuffle unit and is divisor of Ntrain- S; = i mod Nerains and is the frame index. The operator shuffle(.) denotes shuffling the order the frame sequence. Through such rescheduling strategy, we achieve sequence of noise frames with both internal randomness and long-range correlation. Note that, the randomness introduced by temporal shuffle has considerable capacity to bring about content variation, evidenced by Figure
Window based Attention Fusion. Given longer initial noise frames, the spatial modules of Vide- oLDM process them frame-wisely and the temporal convolution processes the frames in sliding window, which is the same case as they were trained. Differently, the temporal attention is per- formed in a global manner and frames longer than Ntrain triggers attentive-scope sensitivity. So, we need to deal with the computation of temporal attention so that it can process the longer sequence in the same way was trained. Specifically, instead of calculating the temporal attention over all frames, we only calculate temporal attention within each local sliding window of size U -Nraann
(Quitu KTSAU Vd
F IHU AttIlemp (AttiemmpQQutttt (Qu+tU KES+U, Va++o) Softmax
Ves+U,
where i is the frame index and j is the window index. Here, take the sliding stride the same value as 5 (the size of noise shuffle unit), so that each sliding window just covers Ntrain frames of independent and identically distributed noises, i.e. (2:.56 NN.....) with shuffled order. Figure 3 illustrates the diagram of our attention computation. As each frame is involved in the attention computation of multiple local windows, we need fuse these attentive outputs achieve a smooth temporal transition. According to our experiments, naively taking the average will cause dramatic variation in the boundaries of windows. Therefore, we propose fuse the window- based outputs in temporal smooth manner, namely computing the weighted sum by taking the frame index distance from each window center as weights:
e - F. FY T Σ -
P 1
where denotes absolute value, and d¹ is the central frame index of the j-th window that covers frame i. po is the output of the current temporal attention layer. Note that, the overlapped window
5

---

Published as a conference paper at ICLR 2024
slicing and merging operation only happen in temporal attention while introducing computation overhead to other modules of the U-Net, which benefits the computational efficiency significantly.
3.3 MOTION INJECTION FOR MULTI-PROMPT VIDEO GENERATION
Since the aforeementooee inference paradigm enables the generation of longer videos, it is natural to explore the potential for synthesizing videos with continuously changing events by utilizing multiple text prompts. This is more challenging because the generation process introduces additional varying factors (i.e. text prompts) that affect the video content mostly. In LDMs,, changing a text prompt with only one verb can lead to totally different video content, even with the same initial noises used (Cao ict al. 2023). Regarding this, we propose a motion injection strategy to modulate the influence of multiple text prompts video generation content. The key idea is to generate the whole video with the first prompt at most denoising steps (more correlated to scene layout and appearances) and use the target prompt only at some specific steps (more correlated to object shapes and poses).
In VideoLDM, text prompts are taken through the attsstntt mechanism:
F Attss (Ô, V,K,V) (F K (x)) V (p(),,
(10)
where Free is the intermediate features of the network, P is the text embedding by CLIP (Rad- ford et al. 2021a) encoder, and 15. g L7 are learned linear layers. According recent research works (Balaji et al. 2022 Cao et al., 2023). LDMs synthesize different levels of visual content- scene layout, shapes of the objects, and fine details, in the early, middle, and late steps of the de- noising process respectively. In our scenarios, we expect the overall layout and object appearance to be similar across prompts while the object poses shapes should follow the target text prompts. To this end, we gradually inject new motion through the cross attention layer during the time steps associated with object shapes, denoted [Ta Tg]. For the sake of simplicity, present our method in the case of two text prompts:
AllDemes (P,7))))1,(()).
if otherwise
L,
Motion Injection
(11)
Attme (Q,1g (
PI, if Ny: N. P N. N, (P2 Pt), if < N,
P
(12)
Pz. otherwise
where P. denotes the i-th prompt; P denotes the target prompt of motion injection, which depends on the frame index nı, and the frames between [N. N=] will be assigned with the linearly interpolated embedding to achieve smooth transition; 1 > L denotes the last L cross-attention layers of the U- Net (e.g. the decoder part). means that the decoder part will always be provided with the target prompt P across all the denoising steps, because the decoder features are more tightly aligned with the semantic structures as observed in MasaCtrl (Cao et 2023).
4 EXPERIMENTS
Setting up. We conduct experiments based on an open-source T2V diffusion model VideoCrafter (Chen et al. 2023) for both singe-prompt and multi-prompt longer video generations. The video diffusion model is trained on 16 frames and is required to sample 64 frames in the infer- ence stage. The window and stride size are set to = 16, S 4 as default.
Evaluation Metrics. To evaluate our paradigm, we report Frechet Video Distance (FVD) (Un- terthiner et al. 2018), Kernel Video Distance (KVD) (Unterthiner et al. 2019) and Clip Similarity (CLIP-SIM) (Radford et al.) 2021b). Since the longer inference methods are supposed to keep the quality of the original fixed-length inference, we calculate the FVD between original generated short videos and subset generated longer videos with corresponding lengths. CLIP-SIM is used to measure the content consistency of generated videos by calculating the semantic similarity among adjacent frames of generated videos.

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparison longer video generation.
Method
FVD ())
KVD
CLIP-SIM (1)
Inference Time
Direct
737.61
359.11
0.9104
21.97s
Sliding
224.55
44.09
0,9438
36.76s
GenL
177.63
21.06
0,9370
77.8%
Ours
85,83
7.06
0.9732
25.75s
Direct
Sliding
GenL
Ours
Qualitative comparisons of longer video generation. Left prompt: "A chiluahua in as- tronaut suit floating space, cinematic lighting, glow effect". Right prompt: "A very happy fuzzy panda dressed as chef eating pizza in the New York street food truck".
Figure 4:
4.1 LONGER VIDEO GENERATION
We mainly compare our proposed FreeNoise to other tuning-free longer video generation methods with diffusion models. We first directly sample 64 frames (Direct). Then we adopt temporal slid- ing windows so that the temporal attention module can always process a fixed number of frames (Sliding). The closest work our paradigm is Gen-L- Video (GenL), which extends the video smoothly by merging some overlapping sub-segments during the denoising process.
The synthesis results are shown Figure 4 In the first line, the dog has severe artifacts and the background of space is not clear. Obviously, directly sampling 64 frames through model trained on 16 frames will bring poor quality results due to the training-inferent gap. When we use tempo- ral sliding windows, the training-inferenee gap is eliminated thus more vivid videos are generated. However, this operation ignores the long-range visual consistency thus the resulting subject and background both look significantly different among different frames. Gen-L-Viden promotes the integration of frames by averaging the overlapping sub-segments and performs better in some cases. However, it fails maintain long-range visual consistency and suffers from content mutation. Ben- efiting from noise rescheduling, all sub-segments in our paradigm share similar main subjects and scenarios while still containing considerable content variation, keeping our main content even when the generated video becomes longer. Results shown in Figure 4 exhibit that our FreeNoise success- fully renders high-fidelity longer videos, outperforming all other methods.
In addition, we also compare the operation time of those methods on NVIDIA A100. As presented in Table [ observed that Gen-L-Vided exhibits the longest inference time, nearly four times longer than direct inference. This is primarily attributed to its default setting, which involves the nearly global sampling of the entire set of latents four times. However, our paradigm only brings less than 20% extra inference time by limiting most additional calculations within the temporal attention layers.
Table T shows quantitative results. The quality of videos generated by direct inference is extremely damaged by the training rrainin-inference gap, obtaining the worst FVD and KVD. The video quality from the sliding method and Gen-L-Video is obviously improved but still worse than the results generated by FreeNoise. Our FreeNoise also gains the best CLI, indicating the superiority of our method in content consistency.
7

---

Published as a conference paper at ICLR 2024
Table 2: User study. Users required pick the best among our proposed FreeNoise with the other bascline methods in terms of content consistency, video quality. and video-text alignment.
Method
Content Consistency
Video Quality
Video-Text Alignment
Direct
11.73%
10.80%
11.1%%
Sliding
6.17%
6.79%
8.02%
GenL
24.38%
26.85%
29.63%
Ours
57.72%
55.56%
51.23%
GenL
Shottersi
Ours w/o
Motion Injection
chuttessto
Ours
Figure 5: Qualitative comparisons of multi-promp video generation. Left multi-prompt: "A camel running the snow field" + "A camel standing on the snow field". Right multi-prompe "An astronaut resting on a horse" + "An astronaut riding a horse".
In addition, we conducted a user study to evaluate our results by human subjective perception. Users asked to watch the generated videos of all the methods, where each example is displayed in random order to avoid bias, and then pick up the best one in three evaluation aspects. As shown in Table [ our approach achieves the highest scores for all aspects: content consistency, video quality, and video-text alignment, outperforming baseline methods by a large margin. Especially for content consistency, our method has received almost twice many votes the second place.
Multi- pulti-prompt Video Generation. We extend our paradigm for multi-prompt video generation by introducing the Motion Injection method. As shown in Figure [ our method achieves coherent visual coherence and motion continuity: The camel gradually changes from running to standing while the distant mountains remain consistent appearances; The astronaut changes from resting on a horse to riding a horse naturally. However, when we purely use the strategy of noise rescheduling without motion injection, the scene will undergo unexpected changes because new prompt often introduces unexpected new contents other than the text description due to the inherent properties of the Stable Diffusion model. But it can still work in some cases when the main objects and scenarios are not obviously changed by the new prompt (like the bigfoot case in Figure 5). We also compare with the existing tuning-free state-of- saate-of-the-ate -heth-rrt method Gen-L-Video Figure that Gen-L. also achieves the conversion of two actions. However, due to the draw brawsacks of content mutation, its generated objects and scenarios are meaninglessly changed over time.
4.2 ABLATION STUDY
Ablation for Noise Rescheduling. As noise rescheduling plays an essential role in our method, typically conduct an ablation validate its importance, namely removing from our proposed inference paradigm. In addition, we also implement another variant of our method with the local accordingly).
noise shuffle unit size $ 8 (the sliding window stride is also changed
As shown in Figure 6 without noise rescheduling, our method fails to keep content consistent. Although each frame still matches the text description, they are not semantically connected. And when the sliding window stride is 8, the synthesized features across windows are interacted in a less tight manner. For example, the shape of the bowl is changed gradually Figure Since the stride value of 4 is able to achieve enough content consistency. we do not consider the smaller stride value of 2, which will bring the double extra inference time.
8

---

Published as a conference paper at ICLR 2024
Random
S=8
Ours
Figure 6: Ablation study on noise rescheduling. Left prompt: "A cat eating food out of a bowl". Right prompt: "A video of milk pouring over strawberries, blueberries, and blackberries".
Ablation for Motion Injection. To show the effec- tiveness of our design choices in Motion Injection, study two main hyper-parametees- -ayer lection and timestep selection, as expressed in Equa- tion [I In our design, the last L cross-attention layers of the U-Net (e.g. the decoder part) will al- ways be provided with the target prompt P across all the denoising steps and P1 only maintains the layout and visual details through the cross-attention layers before the in some denoising steps. For compari- son, construct another two variations that P: con- trol the only decoder part and all layers, respectively. Besides, the third variation allows Pi control the only decoder part across all the denoising steps. Fig- ure shows the results of those variations. When Pi is only allowed to control the decoder part, the run- ning motion of the horse is suppressed. Mcanwhile, the appearance of the horse is changed obviously (shown in Figure (a)).. When Pi is only allowed to control both the encoder and the decoder part, the appearance of the horse is kept but the running mo- tion is still suppressed (shown in Figure 7()). It is because the decoder features more tightly aligned with the semantic structures, as observed in MasaCtrl (Cao et al., 2023). Compared to layer factors, the choice of timesteps has relatively less influence. Even if we allow Pi to control the only decoder part across all the denoising steps, the horse can still switch smoothly from running to standing. However, this behavior influences the generation of layout and visual details. As result, a section of the horse's ' leg will suddenly disappear (shown in Figure ())). while this missing leg appears bent and curled up in the same frame of our final result (shown in Figure dd). Therefore, the selection of time steps can help to achieve more precise control over the content level.
(a) 1 L
(b)
1>0
(c)
t>1
<<
(d)
Ours
Figure 7: Ablation study motion injection. The multi-prompt is: "An astronaut riding horse" + "An astronaut resting on horse".
5 CONCLUSION
In this study, we addressed the limitations of current video generation models trained on limited number of frames and supporting only single-text conditions. We explored the potential of extending text-driven generative models to generate high-fidelity long videos conditioned on multiple texts. Through analyzing the impact of initial noise video diffusion models, we proposed a tuning- free and time-efficient paradigm to enhance the generative capabilities of pretrained models while maintaining content consistency. Additionally, we introduced a novel motion injection method to support multi-text conditioned video generation. Extensive experiments confirmed the superiority of our paradigm in extending the generative capabilities of video diffusion models. Notably, our method achieved this while incurring only approximately 1 17% additional time cost, compared to the previous best-performing method that required a 255% extra time cost.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
The primary objective of this project is to empower individuals without specialized expertise to create video art more effectively. Our paradigm, based the pretrained video diffusion model, assists the model in generating longer videos. It is important to note that the content generated by our tuning-free paradigm remains rooted in the original model. As result, regulators only need to oversee the original video generation model to ensure adherence to ethical standards, and our algorithm does introduce any additional ethical concerns.
REPRODUCIBILITY STATEMENT
We have introduced the algorithm and implementation details in detail in the paper. A researcher fa- miliar with the video diffusion model should be able to basically reproduce our method. In addition, we have implemented our FreeNoise on three advanced video generation models.
VideoCrafter (Chen et al.
2023; butp://gihttt.cmmAAlLaaaCCCCCCff
AnimateDiff Guo et al.
2023):
LaVie (Wang al.
2023d
ACKNOWLEDGEMENTS
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: ALSG2-PhDD-2222-01 --035T,,, the Ministry of Education, Singapore, under its MOE AcRF Tier (MOE-T2EP20221- 0012) and NTU NAP.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Tim Brooks, Janne Hellsten, Miik Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming- Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Advances in Neural Information Processing Systems, 1531779-33788 2022.
Mingdeng Cao, Xintao Wang. Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing. Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.19512. 2023.
songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305. 10474, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint S.Xi:23077.04755. 2023.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35: 27953-27965, 2022.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- els for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.3322) 2022.
Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, al. Animate-a-story: Storytelling with retrieval- augmented video generation. arXiv preprint arXiv:2307.06940, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-683 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint urXiv:
2022.
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprint arXiv:2310. 14400 2023.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10209-10218. 2023.
Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Lo- calizing object-level shape variations with text-to-image diffusion models. arXiv preprint arXiv:2303. 1306, 2023.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong 7hang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 8748-8763 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, Pp- 8748-8763. PMLR, 2021b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3626-3636, 2022.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 2256-2265, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.017777 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. ICLR, 2019.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399, 2022.
Vikram Voleti, Alexia Jolicoeur- Martineau, and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in Neural Information Processing Systems, 35:23371-23388 2022.
Fu- Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text long video generation via temporal co-denoising. arXiv preprint arXiv:2305. 18264, 2023a.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. 2023b.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability, arXiv preprint arXiv:2306. 02018, 2023c.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15103, 2023d.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang. Minheng Ni. Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint 2303 /2346, 2023.
12

---

Published as a conference paper at ICLR 2024
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18456-18466, 2023.
Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma. Tong- Yee Lee, Oliver Deussen, and Changsheng Xu. Prospect: Expanded conditioning for the personaliza- tion of attribute-ave aware image generation. arXiv preprint arXiv:2305. 16225, 2023.
13

---

Published as a conference paper at ICLR 2024
(a) Lens Moving with the Subject
(b) Subject Moving off the Screen
(c) Subject Moving within the Screen
Figure 8: FreeNoise can produce three types of videos exhibiting significant movement: (a) the lens moving with the subject, (b) the subject moving off the screen, and (c) the subject moving within the screen.
A APPENDIX: IMPLEMENTATION DETAILS
During sampling, we perform DDIM sampling (Song et 2020) with 50 denoising steps, setting DDIM eta to 0. The inference resolution is fixed at 256 x 256 pixels. The scale of the classifier-fre guidance is set 15.
For quantitative comparison, we generate a total of 2048 videos for each longer inference method, utilizing 512 prompts (from standard evaluation paper EvalCrafter (Liu al.) (223)) and initial- izing with 4 random initial noises. For calculating the FVD and KVD, we use videos generated by direct inference with training length (16 frames) as the reference set. To align the video length, we cut each long video (64 frames) into 4 segments with 16 frames. Correspondingly, we generate the same number (8192) of short videos with direct inference.
In the user study, we mixed our generated videos with those generated by the other three baselines. total of 27 users were asked to pick the best one according to the content consistency video quality and video-text alignment, respectively.
B APPENDIX: CASE ANALYSIS OF SIGNIFICANT MOVEMENT
Videos with significant movement be mainly divided into three types:
Lens Moving with the Subject. For videos of this type, the position of the subject does not change much and the movement is shown through the regression of the background (Figure a()).
Subject Moving off the Screen. For videos of this type, the subject will move off the screen (Figure (b)). However, the subject will suddenly appear again due to semantic constraints.
Subject Moving within the Screen. For videos this type, the subject will move within the sereen. Due to the size limitation of the screen, the subject will turn around (Figure c)). However, the current pretrained model behaves unnaturally when turning (even for original inference).
As shown in Figure FreeNoise is able to produce all three types. These three types auto- matically determined during the inference stage based on the sampled random noises and the given prompt.
14

---

Published as a conference paper at ICLR 2024
C APPENDIX: LIMITATION DISCUSSION
As repeated locally shuffled noises are used, FreeNoise has a weakening effect on introducing new content to the video the length increases. In some cases, the displacement of the subject limited due to this weakening effect of FreeNoise. However, FreeNoise does not obliterate motion variation or thoroughly fix the spatial structure of objects, such as an object moving from left right on the screen,
Currently the base model (inference without FreeNoise) only works well with videos of the lens moving with the subject and still struggles to deal with the videos of the subject moving off/within the screen effectively. Therefore, the performance of FreeNoise also constrained by the base model. We look forward to applying FreeNoise to more powerful video models in the future.
15


Reviews:
[{"review_id": "rW1zw5AFeg", "reviewer": "Reviewer_q7Nz", "summary": "This paper investigates the effect of initial noise on a diffusion model for video generation, and thus proposes a method to extend the ability of a pre-trained model to generate long videos without fine-tuning, by rescheduling the initial noise of the video frame and by using a window-based temporal attention to achieve long-range visual consistency. finally, a new method of injecting motion trajectories is proposed, which allows the model to generate videos in response to multiple text prompt.", "questions": "Try adding more diagrams to better explain the methods in the article, such as noise rescheduling and an overview of the pipeline for generating a video using the methods mentioned in the article.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed method is simple and effective to expand the model's ability to generate long videos without fine-tuning the model.\n\n2) The use of noise reschedule and window-based attention fusion allows for more consistent video generation.\n\n3) Motion inject allows the model to be fed with a variety of text prompts to generate longer videos with richer meanings.", "weaknesses": "1) The method described in this paper lacks suitable diagrams to help illustrate it.\n\n2) The proposed NOISE RESCHEDULING may limite the content variances of video generation since longer videos are produced by repeating the noises for the short ones. As is shown by the examples, the generated long videos looks like a short video that loops multiple times. I wonder whether this way can produce authentic long videos that contain continous various motions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "louOef1EG3", "reviewer": "Reviewer_pTfs", "summary": "This paper proposes a method to unlock the long video generation ability of a pretrained text-to-video generation model. The major technical components of this method include: 1. The analysis of artifacts and causes when generating long videos. 2. A noise schedule for long video generation. 3. Windowed attention fusion to keep attention perception field while avoiding content jump between windows, 4. Motion injection for varied textual prompts. The experiments show FreeNoise is a competitive long video generator.", "questions": "1.\tThe authors claim FreeNoise achieves the maximum long video of 512 frames due to GPU memory limit. Is it possible to unlock even long video generation ability by using the CPU offload technique? \n2.\tWith the help of ControlNet, is it possible to generate more diverse motion with FreeNoise?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This is a technically solid paper. Long video generation is a tricky long-standing problem. The authors propose a series of insights and techniques that have sufficient novelty to address the difficulties:\n1.\tThe analysis of long video artifacts and causes is valuable for developing better video generators. \n2.\tThe noise scheduling and window-based attention fusion address the long video difficulties mentioned in the analysis. They are simple yet effective. Window-based attention fusion addresses the notorious content jump problem, which will likely help develop future video generation foundation models.\n3.\tFreeNoise does not require additional UNet forward propagation. Therefore, the inference cost overshoot is low. \n4.\tThe qualitative results are marvelous. In human preference evaluation, FreeNoise still achieves the best. The authors provide an anonymous website to show more visual results. The image definition and motion consistency of FreeNoise are both good.\n5.\tThe motion injection technique successfully preserves video contents and drives the video to follow a new text prompt.\n6.\tThe qualitative ablations show each technical component of FreeNoise is effective and important.", "weaknesses": "Major concerns: \n1.\tI’m interested in detailed experiment settings. Please include diffusion sampler configurations, sample resolutions, frame stride, etc. in your future revision.\n2.\tPlease add a pipeline figure. It is not very easy to fully understand how and where FreeNoise is working on generating long videos.\n3.\tIs direct inference, sliding, GenL, and FreeNoise sharing the same pretrained text-to-video model? If it is, then the evaluation is very convincing since they can all generate the same short video using the prompts but only FreeNoise can achieve good long video results.\n\nMinor concerns:\n1.\tPage 7. In the second line. A full stop is missing before ‘Obviously’.\n2.\tIn which case FreeNoise may fail? A discussion over the limitations is welcomed.\n3.\t100 evaluation prompts have limited diversity. If it is feasible, please add more evaluation prompts to make the comparison more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TFV7NkJIg4", "reviewer": "Reviewer_C7UC", "summary": "This paper aims to extend the generative capabilities of pre-trained video diffusion models without incurring significant computational costs. It contains three different components, window-based temporal attention, noise rescheduling, motion injection.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is cleverly represented and very easy to follow.\n2. The proposed method is much more efficient than the baseline method Gen-L [1].\n3. The observation and Analysis part is well-designed and inspiring, and I appreciate this section.\n\n\n\n[1] Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising", "weaknesses": "1. Demo quality. From the demos, we can see that the motions of most generated results are restricted. For example in the ablation study, when the horse running, the background and the position of the horse do not change (even though the legs are moved), which is not a reasonable motion. Therefore, I would say that the proposed method corrupts the motions of the original diffusion models.\n\n2. Many ideas of the paper are used in previous works already. (1) For noise rescheduling, Reuse and Diffuse [1] proposed to reuse previous noise in the later frames. (2) For window-based temporal attention, Align Your Latents [2] applies a similar idea for long video generation, which does not change the temp conv part, but uses sliding local attention to resue the trained temporal attention. I think there's no intrinsic difference. (3) The motion injection part: interpolating the context is already proposed in Gen-L [3].\n\n3. The author says they picked only 100 prompts for quantitative experiments and then generated 2400 videos. This statement seems not explicit.\n\n\n[1] Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation\n[2] Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models\n[3] Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "85rEzsnT5s", "reviewer": "Reviewer_17jd", "summary": "This paper presents a training-free method for extending capabilities of short video diffusion models to generate temporally coherent, longer videos, as well as incorporate multi-text conditioning. The authors propose a novel noise schedule and fused window-based temporal attention to enable more in-distribution, coherent longer generations. In order to enable multi-text conditions, the authors introduce a motion injection method based on conditioning different text prompts at different stages of diffusion sampling.", "questions": "- How would the proposed window fusing (weighted by frame index distance) compared to a simpler scheme such as just merging the current and/or prior window (i.e. similar to a one or two hot version of the frame index weighting).\n- Have the authors explored other noise augmentation other than shuffling? In general, shuffling does not seem to be the most intuitive method for perturbing gaussian noise, as it also constrains the epsilons (per frame) to the original samples.\n- Section 3.1 mentions that \"temporal attention is order independent\". Does this imply that the VideoLDM does not have temporal positional embeddings? Or how would it be order independent if it did?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written, and easy to understand\n- The proposed noise schedule + temporal attention modification is interesting as a method to enforce better long-term consistency in video\n- Incorporating the proposed method is fairly simple, as it does not require any extra training on a pretrained short text-video diffusion model\n- Experiments are concrete, and show much quality generations compared baselines", "weaknesses": "- From looking at the generated videos, although the proposed method can more cleanly generate longer videos, it seems that the spatial structure of the video (e.g. location of a cat) is very similar throughout the entire video. I believe this may be due to the repetitive nature of shuffled noise reptitions which are generally highly correlated with the structure of the resulting video. So it seems that the method may have a hard time generating more dynamic changes in long videos, such as a cat walking across the screen or scene / camera changes. Could the authors comment on this, or if there are generated video examples with larger structural changes through the video?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)