ControlVideo: Training-free Controllable Text-to-video Generation
Authors: Yabo Zhang
Yuxiang Wei
Dongsheng Jiang
XIAOPENG ZHANG
Wangmeng Zuo
Qi Tian
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion models
video generation
Primary area: generative models

Abstract: Text-driven diffusion models have unlocked unprecedented abilities in image generation, whereas their video counterpart lags behind due to the excessive training cost.
To avert the training burden, we propose a training-free ControlVideo to produce high-quality videos based on the provided text prompts and motion sequences.
Specifically, ControlVideo adapts a pre-trained text-to-image model (i.e., ControlNet) for controllable text-to-video generation.
To generate continuous videos without flicker effect, we propose an interleaved-frame smoother to smooth the intermediate frames.
In particular, interleaved-frame smoother splits the whole videos with successive three-frame clips, and stabilizes each clip by updating the middle frame with the interpolation among other two frames in latent space.
Furthermore, a fully cross-frame interaction mechanism have been exploited to further enhance the frame consistency, while a hierarchical sampler is employed to produce long videos efficiently.
Extensive experiments demonstrate that our ControlVideo outperforms the state-of-the-arts both quantitatively and qualitatively. 
It is worthy noting that, thanks to the efficient designs, ControlVideo could generate both short and long videos within several minutes using one NVIDIA 2080Ti. 
Code and videos are available at [this link](https://github.com/YBYBZhang/ControlVideo).

Published as a conference paper at ICLR 2024
CONTROLVIDEO: TRAINING-FREE CONTROLLABLE TEXT-TO- VIDEO GENERATION
Yabo Zhang' Yuxiang Wei! Dongsheng Jiang? Xiaopeng Zhang?
Wangmeng Zuo' Qi Tian
Harbin Institute of Technology "Huawei Cloud
ABSTRACT
Text-driven diffusion models have unlocked unprecedented abilities in image gener- ation, whereas their video counterpart lags behind due to the excessive training cost. To avert the training burden, we propose a training-free Control to produce high-quality videos based on the provided text prompts and motion sequences. Specifically, ControlVideo adapts a pre-trained text-to eoxt-to-image model (i.e., Con- trolNet) for controllable sost-to-vided generation. To generate continuous videos without flicker effects, we propose an interleaved-fratt termediate frames. In particular, interleaved- frame smoother splits the whole video with successive three-frame clips, and stabilizes each clip by updating the middle frame with the interpolation among other two frames in latent space. Furthermore, a fully cross-frame interaction mechanism is exploited to further enhance the frame consistency, while a hierarchical sampler is employed to produce long videos efficiently. Extensive experiments demonstrate that our ControlVider outperforms the state-of-the-art, both quantitatively and qualitatively. It is worth noting that, thanks to the efficient designs, Control Video could generate both short and long videos within several minutes using one NVIDIA 2080Ti. Code and videos are available this link.
smoother to smooth the in-
INTRODUCTION
Large-scale diffusion models have made a tremendous breakthrough on text-to-image synthe- sis (Nichol et al., 2021; Rombach et al., 2022; Balaji et al., 2022; Ramesh et al., 2022; Saharia et al., 2022) and their creative applications (Gal et al., 2022; Wei et al., 2023; Ni et al., 2022; Hertz et al. 2022). Several studies (Ho et al., 2022b;a; Singer et al., 2022; Esser et al., 2023; Hong et al., 2022) attempt to replicate this success the video counterpart, i.e., modeling higher-dimensione complex video distributions in the wild world. However, training such text-to-viee videv model requires massive amounts of high-quality videos and computational resources, which limits further research and applications by relevant communities.
In this work, we study new and efficient form to avert the excessive training requirements: control- lable texi-to-video generation with text-to-image models. As shown in Fig. our method, termed ControlVideo, takes textual description and motion sequence (e.g., depth or edge maps) as conditions to generate videos. Instead of learning the video distribution from scratch, Control adapts the pre-trained text-to-iman image models (e.g., ControlNet (Zhang & Agrawala, 2023)) for high-quality video generation. With the structural information from motion sequence and the superior generation capability of image models, it is feasible to produce vivid video without additional training.
However, as shown in Fig. due to the lack of temporal interaction, individually producing each frame with ControlNet (Zhang & Agrawala, 2023) fails to ensure both (i) frame consistency and (ii) video continuity. Frame consistency requires all frames to be generated with a coherent appearance, while video continuity ensures smooth transitions between frames. Tune-A-Vides (Wu et al., 2022b) and Text2 Video-Zerr (Khachatryan et al., 2023) facilitate appearance consistency by extending self-attention to sparser frame attention. Nonetheless, such a cross- frommf interaction is not sufficient to guarantee video continuity. and visible flickers appear in their synthesized videos (as shown in Fig. and corresponding videos).

---

Published as a conference paper at ICLR 2024
Source video
ControlNet
man alpine wildness
Frame consistency
Video continuity
Text2Video-Zen
ControlVideo
man in alpine wildness
x-1 slice
x-1 slice
We visualize the frames and slice (pixels in red line of original frame) of TexxtVideo-Zero, and observe visible discontinuity in a-t slice. Right: Control adapted from ControlNet, achieves more continuous z-t slice across time, along with improved appearance consistency than Text2Video-Zer See videos for better view.
Figure I:
Training-free controllable text-to-video generation. Left:
Intuitively, a continuous video could be considered multiple continuous three-frame clips, so the problem of ensuring the video continuity is converted to ensuring all three-frame clips continuous. Driven by this analysis, we propose an interleaved-frame smoother enable continuous video generation. Specifically, interleaved-frant clips based on indices of middle frames, and separately smooths out their corresponding latents at different denoising steps. To stabilize the latent of each clip, we first convert it to predicted RGB frames with DDIM, followed by replacing the middle frame with the interpolated frame. Note that, the smoother is only applied at few timesteps, and the quality and individuality of interpolated frames can be well retained by the following denoising steps.
smoother divides all ttree-frame clips into even and odd
We further investigate the cross-frame mechanisms in terms of effectiveness and efficiency. Firstly, we explore fully cross-frame interaction that concatenates all frames to become a "larger image", and first empirically demonstrate its superior consistency and quality than sparser counterparts (see Sec. 4.4). Secondly, applying existing cross-frame mechanisms for long-video generation suffers from either heavy computational burden long-term inconsistency. Therefore, a hierarchical sampler is presented to produce long video in a top-down way. In specific, it pre-generates the key frames cross-frame frame attention for long-range coherence, followed by efficiently generating the short clips conditioned on pairs of key frames.
with fully cross-
We conduct the experiments on extensively collected motion-prompt pairs, and show that Con- trol Video outperforms alternative competitors qualitatively and quantitatively. Thanks to the efficient designs, ControlVideo produces short and long videos in several minutes using one NVIDIA 2080Ti.
In summary, our contributions are presented as follows:
We propose training-free free Control with interleaved-fiee -ramee smoother for consistent and contin- uous controllable text-to-video generation.
Interleaved-frant framee smoother alternately smooths out the latents of three-frame clips, effectively stabilizing the entire video during sampling.
We empirically demonstrate the superior consistency and quality of fully cross-frame interaction, while presenting hierarchical sampler for long-video generation in commodity GPUs.
2 BACKGROUND
Latent diffusion model (LDM) (Rombach al., 2022) is efficient variant of diffusion models (Ho et al., 2020) by applying the diffusion process in the latent space. LDM uses an encoder to compress an image æ into latent code (x). ) It learns the distribution of image latent codes 2u ~ Pdata (zu) in DDPM formulation (Ho et al., 2020), including a forward and backward process. The forward diffusion process gradually adds gaussian noise each timestep to obtain 2;:
q(2/2=11) = Niz;; V 11---1 1,
(1)
2

---

Published as a conference paper at ICLR 2024
or
ControlNet
moving lake
Discrete video
Smoother
Interleaved- Frame
Continuous video
X T
steps
ZT
2o
Cross- Frame Aztn
Text Cross-Attn
Conv Block
Attn Block
Figure Overview of ControlVideo. For consistency in appearance, Control adapts ControlNet to the video counterpart by adding cross- cross-frame interaction into self-attention modules. To further improve video continuity, interleaved-frett smoother is introduced stabilize video latents during denosing (see Alg. for details).
where (.3g ) are the scale of noises, and T denotes the number of diffusion timesteps. The backward denoising process reverses the above diffusion process to predict less noisy 2;-1:
pe 2 |x) (2 (z t), Eg (z t)).
The μ사 and Ng are implemented with denoising model with learnable parameters 0. When generating new samples, we start from 2T ~ _/ 1) and employ DDIM sampling to predict z;-1 of previous timestep:
2t-1 04-1
2
vI ax60 (2t,1)
+ 11 a1-1 EØ Ez( t),
(3)
"direction pointing to zz""
predicted 20"
where Or, = -I -1) We use z=-0 to represent "predicted 20" at timestep t for simplicity. Note that we use Stable Diffusion (SD) ea(2,, (2,,, T) as our base model, which an instantiation of text-guided LDMs pre-trained on billions of image-text pairs. denotes the text prompt.
ControlNet (Zhang& Agrawala, 2023) enables SD to support more controllable input conditions during text-to-image synthesis, e,g., depth maps, poses, edges, etc. The ControlNet uses the same U-Net (Ronneberger et al., 2015) architecture as SD and finetunes its weights to support task- specific conditions, converting (2,, r) co(z,, c, r), where denotes additional conditions. To distinguish the U-Net architectures of SD and ControlNet, we denote the former as the main U-Net while the latter as the auxiliary U-Net.
CONTROLVIDES
Controllable text-to-video generation aims to produce video of length N conditioned motion N-1 and text prompt As illustrated in Fig. 2, propose Control Video with interleaved smoother towards consistent and continuous video generation. ControlVideo adapted from ControlNet, adds foas-f interaction to self self-attention modules for frame consistency (in Sec. 3.1). To ensure video continuity, nnterleaved-framt smoother divides all three-frrae frree-frame clips into even and odd clips, and separately smooths their corresponding latents different denoising steps (in Sec. 3.2). Finally, we further investigate the cross-frame mechanisms in terms of effectiveness and efficiency, including fully cross-frame interaction and hierarchical sampler (in Sec. 3.3).
sequences [ci Ti=0
3.1 PRELIMINARY
The main challenge of adapting text-to-ine text-to-image models to the video counterpart to ensure temporal consistency. Leveraging the controllability of ControlNet, motion sequences could provide coarse- level consistency in structure. Nonetheless, due to the lack of temporal interaction, individually producing each frame with ControlNet leads drastic inconsistency in appearance (see row 2 in
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Interleaved-framt smoother
Require: [21) ,
{c' fi=0 timestep I.
1: 21-0 f 21 (=-1Taacc(s.t.e.r)
predict clean latents
Vai
2: 24-+0 ((z -0 2/-+0
convert latents to RGB space
3: if mod 2) = then
2k+11 b smooth all even three-frame clips (0 $28 2 11-0
4: for from 0 to N/2 do
5: 22-00 Interpolatte 22
2k-1
2t t--0
2k+11
6: else if mod 2) = then
smooth all odd three-frame clips (i)
2k+1 2k+22
a
7:
for from 0 to N/2 do
8:
2 + Interpolate 2 x0-00
9: -= f- (0)
D convert frames to latent space > predict less noisy latent
10:
21-1 f- Va-1 20 + VT a-- co(z,, t, c,
11: return 21-1
Fig. 5). Similar to previous works (Wu et al., 2022b; Khachatryan et al., 2023), we also extend original self-attention of SD U-Net to cross-frame attention, so that the video content could be temporally shared via inter-frame interaction.
In specific, VidtrolVideo inflates the main U-Net from Stable Diffusion along the temporal axis, while keeping the auxiliary U-Net from ControlNet. Analogous (Ho al., 2022b; Wu al., 2022b; Khachatryan et al., 2023), directly converts 2D convolution layers to 3D counterpart by replacing
3 x 3 kernels with x 3 x 3 kernels. Self-attention converted to cross-frame attention by querying from other frames as:
QK Vd
(4) where W W and W project into query, key, and value, respectively. zf and ū, denote ith latent frame and the latents of reference frames at timestep We will discuss the choices of cross-frame mechanisms (i.e., reference frames) in Sec. 3.3
Attention(Q) Q, K. V) = Softmax
V. where Q W 2/. K = W V W 2,
3.2 ENTERLEAVED-FRAND SMOOTHER
Albeit cross cross-frame interaction promisingly keeps frame consistency in appearance, they are still visibly flickering in structure. Discrete motion sequences only ensure coarse-level structural consistency, not sufficient to keep the continuous inter-frame transition. Intuitively, a continuous video could be considered multiple continuous three-frame clips, so we simplify the problem of ensuring the video continuity to ensuring all three-frame clips continuous.
Inspired by this, we propose an interleaved-frame smoother to enable continuous video generation. smoother divides all ttree-frame clips into even and odd clips based on indices of middle frames, and individually smooths their corresponding latents different timesteps. To stabilize the latent of each clip, we first convert it to predicted RGB frames with DDIM, following by replacing middle frame with the interpolated frame.
In Alg. 1. interleaved-
Specifically, at timestep i, we first predict the clean video latent 24-40 according to 2,:
vI area (z, t,
2
Vau
After projecting #c-0 into RGB video T:-+0 D(z,-)), convert it to more smoothed video Fc-+0 by replacing each middle frame with the interpolated one. Based on smoothed video latent E E():-o)), compute the less noisy latent - following DDIM denoising in Eq. 3:
2 Vai-1 + / 04-1 c0(21, 1, C,
We note that the above process is only performed a few intermediate timesteps, the individuality and quality of interpolated frames are also well retained by the following denoising steps. Additionally, the newly computational burden can be negligible (See Table 3).
4

---

Published as a conference paper at ICLR 2024
Source Videos
Structure Conditions
Tune-A-Video
Text2Video-Zero
ControlVideo
(Ours)
daring scaling treacherous and jagged peak in the alpine wilderness
A curious golden dog curiously wanders on the rocky mountain trail.
(a) Depth Maps
(b) Canny Edges
Figure Qualitative comparisons conditioned on depth maps and canny edges. Our Con- trol Video produces videos with better (a) appearance consistency and (b) video quality than others. In contrast, Tune-A-Vidhe fails to inherit structures from source videos, while Text2 Video-roo brings visible artifacts in large motion videos. See videos at qualitative comparisons.
3.3 CROSS-FRAME MECHANISMS FOR EFFECTIVENESS AND EFFICIENCY
Fully cross-frame interaction. Previous works (Wu et al., 2022b; Khachatryan et al., 2023) usually replace self-attention with sparser cross-frame mechanisms, egg., taking the reference frames as first or previous frames. Such mechanisms will increase the discrepancy between the query and key in self-attention modules, resulting in the degradation of video quality and consistency. In contrast, fully cross-frame interaction considers all frames as reference (i.e., becoming a "large image") "), so has a less generation gap with text-to-image models. We conduct comparison experiments above mechanisms in Fig. and Table 3. Despite slightly more computational burden, fully cross-frame interaction empirically shows better consistency and quality than the sparser counterparts.
Hierarchical sampler. Applying existing cross-frame mechanisms for long-video generation suffers from either heavy computational burden long-term inconsistency, limiting the practicability of Control Video. For more efficient long-video synthesis, we introduce hierarchical sampler to produce long video clic-by-clip, which is implemented with two types of cross-frame mechanisms. At each timestep. long video 2, {2)) is separated into multiple short video clips with the key
selected key frames 2t
Jk=n- where each clip is of length N. and the kth clip is Ij=kNe+1 Then, we pre-generate the key frames with fully cross-frame
{2)
KN.. 1 No
denoted as 2 {
1)N.11
attention for long-range coherence, where reference frames are 2 &N Jk=0 Conditioned on each pair of key frames, i.e, reference frames as (=A.N. 2, (k+1)N.), we sequentially synthesize their corresponding clip 2t holding the holistic consistency.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Implementation details. VontrolVideo is adapted from ControlNet (Zhang & Agrawala, 2023). and our interleaved-frame smoother employs a lightweight RIFE (Huang et al., 2022) to interpolate
5

---

Published as a conference paper at ICLR 2024
Table Quantitative comparisons of ControlVides with other methods. We evaluate them on 125 motion-prompt pairs in terms of consistency, and the best results are bolded.
МETHOО
Structure Condition
FC (x10-)
PC (x10-))
WE (x
Mine-A-Video Wu et al. (2022b)
DDIM Inversion
94.53
31.57
18.16
Text2Video-Zero Khachalryan al. (2023)
Canny Edge
95.17
30.74
8.76
Control Video (ours)
Canny Edge
96.83
30.75
2.75
Zer2Viido-Zero Khachatryan et al. (2023)
Depth Map
95.99
31.69
10.36
ControlVided (ours)
Depth Map
97.22
31.81
5.81
the middle frame of each three-frame clip. The synthesized short videos are of length 15, while the long videos usually contain about 100 frames. Unless otherwise noted, their resolution is both 512 512. During sampling, we adopt DDIM sampling (Song et al., 2020a) with 50 timesteps, and interleaved-frante
smoother is performed on predicted RGB frames at timesteps {30,31 by default. With the efficient implementation of xFormers (Lefaudeux et al., 2022), Contro Video could produce both short and long videos with one NVIDIA RTX 2080Ti in about 2 and 10 minutes, respectively.
Datasets. To evaluate our Vodtrol we collect 25 object-centric videos from DAVIS dataset (Pont-Tuset et al., 2017) and manually annotate their source descriptions. Then, for each source description, ChatGPT (OpenAI, 2022) is utilized to generate five editing prompts automati- cally, resulting in 125 video- video-prompt pairs total. Finally, we employ Canny and MiDaS DPT-Hybrid model (Ranftl et al., 2020) to estimate the edges and depth maps of source videos, and form 125 motion-prompt pairs as our evaluation dataset. More details are provided in Appendix A.
Metrics. We evaluate the video quality from three perspectives. (i) Frame consistency (FC): the average cosine similarity between all pairs of consecutive frames, and (ii) Prompt consistency (PC); the average cosine similarity between input prompt and all video frames. (iii) Warping error (WE) (Lai al., 2018): the average error between all frames and their warped frames using optical flow.
Baselines. We compare our Control with three publicly available methods: (i) Tune-A- Video (Wu et al., 2022b) extends Stable Diffusion to the video counterpart by finetuning it on a source video. During inference, it uses the DDIM inversion codes of source videos to provide structure guidance. (ii) TTexr2Video-74 first-only cross-frame, attention on Stable Diffusion without finetuning. (iii) Follow-Your Your-Pose (Ma al., 2023) is initialized with Stable Diffusion, and is finetuned on LAION-PoSe (Ma et al., 2023) to support human pose conditions. After that, is trained on millions of videos (Xue al., 2022) to enable cmporally-consisste video generation.
(Khachatryan et al., 2023) is based on ControlNet, and employs the
4.2 QUALITATIVE AND QUANTITATIVE COMPARISONS
Qualitative results. Fig. 3 first illustrates the visual comparisons of synthesized videos conditioned on both (a) depth maps and (b) canny edges. As shown in Fig. (a), our Control demonstrates better consistency in both appearance and structure than alternative competitors. Tune-A- A-Ve-A-Video fails to keep the temporal consistency of both appearance and fine-grained structure, e.g., the col color of coat and the structure of road. With the motion information from depth maps, Text2Video-Zero achieves promising consistency in structure, but still struggles with incoherent appearance videos e.g., the color of coat. Besides, Control also performs more robustly when dealing with large motion inputs. As illustrated in Fig. 3 (b), Tune-A-Video ignores the structure information from source videos. Video- adopts the first-only cross-frame mechanism trade off frame quality and appearance consistency, and generates later frames with visible artifacts. In contrast, with the proposed fully cross-frame mechanism and interleaved-frant smoother, our Control
handle large motion generate high-quality and consistent videos.
Fig. 4 further shows the comparison conditioned on human poses. From Fig. 4, Tune-A-Video only maintains the coarse structures of the source video, i.e., human position. Tex12Video-Zen and Bollow-Your-Pee produce video frames with inconsistent appearance, e,g., changing faces of iron man (in row 4) or di sappearing objects in the background (in row 5). In comparison, our Control Video performs more consistent video generation, demonstrating its superiority, More qualitative comparisons are provided in Appendix

---

Published as a conference paper at ICLR 2024
Table User preference study. The numbers denote the percentage of raters who favor the videos synthesized by our VontrolVideo over other methods.
Method Comparison
Video Quality
Temporal Consistency
Text Aligament
Ours Tune-A-Vided Wu ct al. (2022b)
73.6% 76.0%
83.2% 81.6%
68.0% 65.6%
Ours Text2Viden-Zero Khachatryan (2023)
Videos
Source Video
Structure Conditions
Individual
Tune-A-Video
First-only
Text2Video-Zero
Sparse-causal
Fully Cross frame
ControlVidea
[Qurs)
Fully Cross frame Smoother
Iron moonwalk on the road.
mighty elophart marches steadily through the rugged terrain.
Figure 5: Qualitative ablation studies cross- framm mechanisms and interleaved -rrmme smoother. Fully cross-frame interaction pro- duces video frames with higher quality and con- sistency than other mechanisms, and adding the smoother further enhances the video smoothness. See corresponding videos for better comparison.
Figure 4: Qualitative comparisons on poses. Tune-A-Video only preserves original human positions, while TexttVideo-Zee and Follow- Your-Pose produce frames with appearance inco- herence. Our Control Video achieves better con- sistency in both structure and appearance. See videos at qualitative comparisons.
Quantitative results. We have also compared our ControlVideo with existing methods quantita- tively on 125 video-promp prompt pairs. From Table our ControlVideo conditioned on depth outperforms the state-of-the-arr methods in terms of metrics, which consistent with the qualitative re- sults. In contrast, despite finetuning on a source video, Tune-A-Vides still struggles to produce temporally coherent videos. Although conditioned on the same structure information, Text2 Text2Video- Zero obtains worse frame consistency and warping eтror than Control Video. For each method, the depth--oodiiioneed models generate videos with higher frame and prompt consistency than the canny-condition counterpart, since depth maps provide smoother motion information.
4.3
USER STUDY
We then perform the user study to compare our Control conditioned depth maps with other competing methods. In specific, we provide each rater structure sequence, text prompt, and synthesized videos from two different methods (in random order). Then we ask them to select the better synthesized videos for each of three measurements: (i) video quality, (ii) temporal consistency throughout all frames, and (iii) text alignment between prompts and synthesized videos. The evaluation set consists 125 representative druuttur-promp pairs. Each pair is evaluated by raters, and we take majority vote for the final result. From Table 2, the raters strongly favor synthesized videos from all three perspectives, especially in temporal consistency. On the other hand, Tune-A-Vider fails to generate consistent and high-quality videos with only DDIM inversion for structural guidance, and Text2Video-Zero also produces videos with lower quality and coherency.
7

---

Published as a conference paper at ICLR 2024
Table Quantitative ablation studies on cross-frame mechanisms and interleaved- smoother. The results indicate that our fully cross-framm mechanism achieves better frame consistency than other mechanisms, and the interleaved frame smoother significantly improves the frame consistency.
Cross-Frami Mechanism
FC (x10 3)
PC (x10 ?)
WE (x10 3)
Time Cost (min)
Individual
89.94
30.79
20.13
1.2
First-only
94.92
30.54
8.91
1.2
Sparse-Causal
95.06
30.59
7.05
1.5
Fully
95.36
30.76
5.93
3.0
Fully + Smoother
96.83
30.79
2.75
3.5
steamship on the ocean, at sunset, sketck style
Frame
Frame 11
Frame 22
Frame 33
Frame 44
Frame 55
Frame 66
Frame 77
Frame 88
Frame 99
Frame 110
Frame 121
Figure 6: A long video produced with our hierarchical sampling. Motion sequences are shown on the top left. Using the efficient sampler, our Control generates a high-quality long video with the holistic consistency. See videos at long video generation.
4.4 ABLATION STUDY
Effect of fully cross-frame interaction. To demonstrate the effectiveness of the fully cross-frame frame interaction, we conduct a comparison with the following variants: i) individual: interaction between all frames, ii) first-only: all frames attend to the first one, iii) sparse-causal: each frame attends to the first and former frames, iv) fully: our fully cross-frame, refer Sec. Note that, all the above models extended from ControlNet without any finetuning. The qualitative and quantitative results are shown in Fig. and Table 3, respectively. From Fig. 5, the individual cross frame mechanism suffers from severe temporal inconsistency, e.g., colorful and black whit- frames. The first-only and sparse-causal mechanisms reduce some appearance inconsistency by adding cross- frame interaction. However, they still produce videos with structural inconsistency and visible artifacts, e.g., the orientation of the elephant and Fig. 5). In contrast, due to less generation gap with ControlNet, our fully cross-frame interaction performs better appearance coherency and video quality. Though the introduced interaction brings an extra 2x time cost, acceptable for a high-quality video generation.
nose (row 3 in
Effect of interleaved-frame smoother. We further analyze the effect of the proposed interleaved- frame smoother. From Table and last two rows of Fig. 5, our seerleaved-frame smoother greatly improves video smoothness, e.g., mitigating structural flickers in red boxes. We provide more ablation studies on the timestep choices of the smoother in Appendix C and ablation studies.
4.5 EXTENSION TO LONG-VIDEO GENERATION
Producing long video usually requires an advanced GPU with high memory. With the proposed hierarchical sampler, our ControlVided achieves long video generation (more than 100 frames) in a -fficieen manner. As shown in Fig. 6, our ControlVideo can produce a long video with consistently high quality. Notably, benefiting from efficient sampling, it only takes approximately ten minutes to generate 100 frames with resolution 512 x 512 in one NVIDIA RTX 2080Ti. More visualizations of long videos can be found in Appendix D.
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
Text-to-image synthesis. Through pre-training on billions of image-text pairs, large-scale gen- erative models (Nichol et al., 2021; Balaji et al., 2022; Saharia et al., 2022; Ramesh et al., 2022; Rombach et al., 2022; Ramesh et al., 2021; Chang et al., 2023; Ding et al., 2021; 2022; Yu et al., 2022; Sauer et al., 2023; Kang al., 2023; Huang et al., 2023) have made remarkable progress in creative and photo-realistic visual generation. Various frameworks have been explored to enhance image quality, including GANs (Goodfellow et al., 2020; Sauer et al., 2023; Kang et al., 2023), autoregressive models (Nichol et al., 2021; Chang et al., 2023; Ding et al., 2021; 2022; Yu et al., 2022), and diffusion models (Ho al., 2020; Balaji et al., 2022; Saharia al., 2022; Ramesh et al.. 2022; Rombach et al., 2022). Among these generative models, diffusion- based models are well open-sourced and popularly applied to several downstream tasks, such as image editing (Hertz et al., 2022; Meng et al., 2021) and customized generation (Gal et al., 2022; Wei et al., 2023; Kumari et al., 2022; Ruiz et al., 2022). Besides text prompls, several works (Zhang & Agrawala, 2023; Mou et al., 2023) also introduce additional structure conditions to pre- pre-trained text-to-image diffusion models for controllable text-to rext-to-image generation. Our Control Video is implemented based on the controllable text-to-image models to inherit their ability of high-quality and consistent generation.
Text-fo-video synthesis. Large text-to-vided generative models usually extend text-to-image mod- els by adding temporal consistency. Earlier works (Wu et al., 2022a; Hong et al., 2022; Wu et al., 2021; Villegas et al., 2022) adopt an autoregressive framework to synthesize videos according to given descriptions. Capitalizing on the success of diffusion models in image generation, recent works (Ho et al., 2022a;b; Singer et al., 2022) propose to leverage their potential produce high-quality videos. Nevertheless, training such large-scall video generative models requires extensive video-text pairs and computational resources. To reduce the training burden, Gen-1 (Esser et al., 2023) and Follow- Your-Pose (Ma et al., 2023) provide coarse temporal information (e.g.. motion sequences) for video generation, yet are still costly for most researchers and users. By replacing self-attention with the sparser cross-frame mechanisms, Tune-A- Aiieeo (Wu et al., 2022b) and Text2Video-Zer (Khacha- tryan et al., 2023) keep considerable consistency in appearance with little finetuning. Control also adapts text-to-image diffusion models without any training, but generates vidcos with better temporal consistency and continuity.
DISCUSSION
In this paper, we present a training-free framework, namely VontrolVideo, towards consistent and continuous controllable text-to-video generation. ControlVideo, inflated from ControlNet, introduces an interleaved-frant frame smoother to ensure video continuity. Particularly, interleaved-fratt frame smoother alternately smooths out the latents of three-frame clips, and stabilizes each clip by updating the middle frame with the interpolation among other two frames in latent space. Moreover, we empirically demonstrate the superior performance of fully cross-framd interaction, while presenting hierarchical sampler for long-video generation in commodity GPUs. Quantitative and qualitative experiments on extensive motion-promp pairs demonstrate that Control Video achieves state-of-the- tfttt---sssss in terms of frame consistency and video continuity.
Broader impact. Large-scale diffusion models have made tremendous progress in text-to-video synthesis, yet these models are costly and unavailable to the public. Control focuses on training- free controllable text-to-video generation. and takes an essential step in efficient video creation. Concretely, Control Video could synthesize high-quality videos with commodity hardware, hence, being accessible to most researchers and users. For example, artists may leverage our approach to create fascinating videos with less time. Moreover, Control Video provides insights into the tasks involved in videoss, e.g., video rendering, video editing, and video-to-video translation. On the flip side, albeit we do not intend to use our model for harmful purposes, it might be misused and bring some potential negative impacts, such as producing deceptive, harmful, or explicit videos. Despite believe that they could be well minimized with some steps. For example,
the above concerns, an NSFW filter can be employed to filter out unhealthy and violent content. Also, we hope that the government could establish and improve relevant regulations restrict the abuse of video creation.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by National Key RD Program of China under Grant No. 20212D0112100 and the National Natural Science Foundation of China (NSFC) under Grant No. U19A2073.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-iman -iitge diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01224, 2022.
Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Hing-Hsuan Yang. Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 3xXiv23301.00044 2023.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. NeurlPS, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang, Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint 200::2204.1217., 2022.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprini arXiv:2208.011 01618,, 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 2020.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with attention control. arXiv preprint (rXii:2208.01666 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neur!PS, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300 2022a.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
Tianyu Huang, Yihan Zeng, Bowen Dong. Hang Xu, Songcen Xu, Rynson WH Lau, and Wangmeng Zuo. Textfield3d: Towards enhancing open-vocabulary 3d generation with noisy text fields. arXiv preprint arXiv:2309. 17175, 2023.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV, 2022.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In CVPR, 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv: 13439, 2023.
10

---

Published as a conference paper at ICLR 2024
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun Yan Zhu. Multi-conceppt customization of text-to-image diffusion. arXiv preprint arXiv:2212.04488, 2022.
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming- Hsuan Yang. Learning blind video temporal consistency. In Proceedings of the European conference on computer vision (ECCV), pp. 170-185, 2018.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. /tteps:/ggiiie com/ Facebookresearcchkee wwwpcc///.ooorrr..ooooo 2022.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Fol- low your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint arXiv:2304.01 186, 2023.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yaa Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01075 2021.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint 2.t.:2302..8553, 2023.
Minheng Ni, Zitong Huang, Kailai Feng, and Wangmeng Zuo. Imaginarynet: Learning object detectors without real images and annotations. arXiv preprint arXiv:2210.06886, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
TB OpenAl. Chatgpt: Optimizing language models for dialogue. OpenAl, 2022.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung and Luc Van Gool. The 2017 davis challenge on vidco object segmentation. arXiv preprint
arXiv:1704.00675, 2017.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text- based video editing. arXiv preprint arXiv:2303.0953. 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 20Xx:2204.00125,, 2022.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint a008 12242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic exxteto-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. | 2022.
11

---

Published as a conference paper at ICLR 2024
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to- text-to-image synthesis. arXiv preprint arXiv:2301.09 00951,, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-vidoo generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020b.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hemann Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399, 2022.
Wen Wang, kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero- shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303. /7599, 2023.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302. / 2023.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2 14806, 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In ECCV, 2022a.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212. 2022b.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. CVPR, 2022.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint 2rxiv:2206. 10789, 2022.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05544 2023.
12

---

Published as a conference paper at ICLR 2024
A. DATASET DETAILS
In Table 4, we select 25 representative videos from DAVIS dataset (Pont-Tuset et al., 2017) and manually annotate their source captions. After that, ask ChatGPT to generate five edited prompts for each source caption, following the instruction like: Please generate five new sentences that similar to "A man dances the road", while being more diverse and highly detailed. Finally, obtain 125 video- video-prompt pairs in total, and use them to evaluate both canny and depth conditioned generation.
B. USER STUDY DETAILS
We conduct user study to compare VontrolVideo against two other methods 125 samples, and ask five raters to answer questions in each sample. In Fig. 7, there are three questions involving in (i) video quality, (ii) temporal consistency, and (ii) text alignment. The raters are given unlimited time to make the selection. After collecting their answers, we take a majority vote as the final result for each sample, and present statisties Table
C. MORE ABLATION STUDIES
During inference, adopt DDIM sampling with T 50 timesteps, which iteratively denoises Gaussian noise from T to O.
Which timesteps does interleaved-framte smoother perform at? In Fig. 8, we explore three timestep choices at different noise levels, including (48, 49) at large noise level, {30, 31} at middle noise level, and {0, 1} little noise level. When using the smoother timesteps {48, 49). the processed video is still unstable, since structure sequences bring additional flickers at the following timesteps. At timesteps {0, }} nearby image distribution, applying the interleaved-framt smoother leads to visible distortion in some frames. In contrast, performing smoothing operation at middle timesteps (30, 31} promisingly deflickers the video, while preserving the quality and individuality of interpolated frames.
How many timesteps are used in interleaved-frame smoother? Fig. shows the smoothed videos using interleaved-frant frame smoother at different numbers of timesteps. Applying the smoother two consecutive timesteps (i.e., 2 steps) could smooth the entire video with little video quality degradation. As the number of smoothing steps increases, the processed video is much smoother, but some frames become slightly blurred. Thus, for higher quality and efficiency, we set the number of smoothing timesteps as 2 by default.
Non-deterministic DDPM-style sampler. ControlVideo can also employ a non-deterministic DDPM-style sampler during inference. Following Eq. in DDIM (Song et al., 2020b), predict _ from 2, via (i.e., line of Alg. in paper):
21-1 v 12 VI 04-- (0 C, + ozht,
where e; and on controls the level of random noise. DDPM results presents the generated videos of VontrolVideo at different noise levels. Notably, the noise level increases, ControlVideo generates more photo-realistic videos with dynamic details, e.g., ripples in the water.
D. MORE VISUALIZATIONS AND COMPARISONS
Fig. 10, Fig. 11, and Fig. 12 show more video visualizations conditioned on canny edges, depth maps, and human poses. Fig. 14, Fig. 15, and Fig. 16 present qualitative comparisons conditioned on canny edges, depth maps, and human poses. Fig. 13 provides an additional long video. More comparisons with video editing methods (Qi al., 2023; Wang et al., 2023) are shown in this link.
Firstly, Vid2Vid-Zero and FateZero are designed for video editing by hybrid of fully and sparse- casual cross-frame attention, and does not investigate different attention mechanisms in depth. In contrast, our ControlVides focuses on continuous controllable text-to-video generation, and first empirically investigate the superiority of fully cross-frame attention. Secondly, Fig, 18 shows their qualitative comparisons on video editing. As one can see, the edited videos of Control Video not only have consistent structure with source videos, but also aligns better with text prompts.
13

---

Published as a conference paper at ICLR 2024
Table 4: Names and captions of selected videos from DAVIS dataset.
Video Name
Source Caption
blackswan
black swan moving on the lake
boat
boat moves in the river
breakdance-flare
man dances on road bus moves on the street a camel walks on the desert
bus
camel
norrruundabut
jeep turns road moves building a jeep forest road cow walks on the grass a dog walks on the ground an elephant walks the ground flamingo wanders in the water golden fishers swim the water man hikes on mountain player is playing hockey on the ground
car-shadow
car-turn
cows
dog
elephant
flamingo
gold-fish
hike
hockey
kite-surf
man is surfing on the sea three women stands on the lawn
lab-coat
longboard
is playing skateboard the alley a mallard swims
mallard-water
water
mbbic-ttrick
man riding motorbike a rhino walks on the rocks sailing boat moves on the sea girl is playing the swings man playing tennis selfie of walking man
rhino
surf
swing
tennis
walking
determined man is trudging up a snowy and ley mountain slope, braving the biting cold and fierce winds
Text
Structure Sequence
Video
Video
Prompt
Between Method & :
Which video higher quality
Which video has better temporal consistency across all frames?
3.
Which video aligns better with text prompt?
The instruction of user study. A user study sample consists of a text prompt, structure sequence. and synthesized videos from two different methods (in random order). The raters asked to answer the above three questions for each sample.
Figure 7:
E. LIMITATIONS.
While our ControlVideo enables consistent and high-quality video generation, it still struggles with producing videos beyond input motion sequences. For example, in Fig. 17, given sequential poses of Mhael Jackson' s moonwalk k, it is difficult to generate a vivid video according to text prompts like Iron man runs on the street. In this link, when input text prompts (e.g., rabbit) seriously conflict with input motion (e.g..) ), the synthesized videos usually tend to align with input motion, ignoring the implicit structure in text prompts. To increase the ratio of text prompts over structure, we decrease the scale of ControlNet À to 0.3 (A = 1 by default). Therefore, it can be seen À = 0.3 that achieves a better trade-off between two input conditions than 1. the future, we will explore how to adaptively modify input motions according to text prompts, so that users can create more vivid videos.
14

---

Published as a conference paper at ICLR 2024
A dusty old jeep was making way down the winding forest road, creaking and groaning with each bump and turn.
Timesteps (48, 49)
Increase the time of steps
Timesteps (30, 31)
Timesteps (0. 1)
w/o smoother
Structure Sequence
Figure 8: Ablation on timestep choices in interleaved-framt smoother. We apply interleaved- frame smoother at different timesteps, including {48, 49} large noise level, (30. 31} at middle noise level, and (0, 1} at little noise level. Among them, using the smoother at timesteps {30, 31) promisingly mitigates the flicker effect while ensuring high quality. Results best seen at 500 %
zoom.
sleek black jeep speeding along the forest road, dodging and rocks with
steps
Increase the number of steps
6 steps
steps
2 steps
step
Structure Sequence
Figure 9: Ablation on the number of timesteps used in interleaved-frame smoother. Applying the smoother consecutive timesteps (i.e., 2 steps) effectively reduces the flickers in structure. As we increase the number of smoothing steps, the processed video becomes smoother, but some frames slightly blurred. Therefore, we set the number of smoothing steps two by default. Results best seen at 500 % zoom.
15

---

Published as a conference paper at ICLR 2024
A white swan moving on the lake, eartoon style.
majestic camel gracefully strides across the scorching desert sands.
shiny red jeep smoothly turns narrow, winding road in the mountains.
A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump.
A man leisurely hiking through a lush and verdant forest.
Figure 10: More video visualizations conditioned on canny edges. Results best seen at 500 % zoom.
16

---

Published as a conference paper at ICLR 2024
striking mallard floats effortlessly on the sparkling pond.
A gigantic yellow jeep slowly turns on wide, smooth road in the city.
contented cow ambles across the dewy, verdant pasture.
A sleek boat glides effortlessly through the shimmering river, gogh style.
A majestic sailing boat cruises along the vast, azure sea.
Figure 11: More video visualizations conditioned on depth maps. Results best seen at 500 % zoom.
17

---

Published as a conference paper at ICLR 2024
Top: Hulk is jumping on the strest, cartoon style
Bottom: The Simpsons in the city, Hockney style.
Top: Goku a mountain range, surreal style.
Bottom: Wonder Woman in desert, Pop Art style.
Top: man, wearing pink clothes, moonwalk sunset.
Bottom: James bond moonwalk on the beach, animation style.
Figure 12: More video visualizations conditioned on human poses. Results best seen 500% zoom.
Hulk is dancing on the beach, cartoon style.
Figure 13: Additional long video visualization. Results best seen 500% zoom.
18

---

Published as a conference paper at ICLR 2024
Source Videos
Structure Conditions
Tune-A-Video
Text2Video-Zero
Controlvideo (Ours)
A mighty elephant marches steadily through the rugged terrain.
A skiny silver vehicle gracefully mansuvers towards modern glass building.
Figure 14: More qualitative comparisons conditioned on canny edges. Results best seen at 500 % zoom.
Source Videos
Structure Conditions
Tune-A-Video
Text2Video-Zero
ControlVideo (Ours)
daring man performing gravity-defying stunts on high-speed, blue motorbike in an empty parking
A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump and turn.
Figure 15: More qualitative comparisons conditioned depth maps. Results best at 500% zoom.
19

---

Published as a conference paper at ICLR 2024
Source Videos
Structure Conditions
Text2Video-Zero
Follow-Your-Pose
ControlVideo (Ours)
dances futuristic city, cyberpunk style.
robot dances on road, animation style.
Figure 16: More qualitative comparisons conditioned on human poses. Results best seen at 500 % zoom.
Source Video
Human Pose
Synthesized Video
Iron man rurs in the road.
Figure 17: Limitation visualizations. VontrolVideo struggles with producing videos beyond input motion sequences. The motion of text prompt Iron man runs on the street does not align with the given sequential poses of Michael Jackson' moonwalk, which degrades the video quality and consistency. See videos at limitations.
20

---

Published as a conference paper at ICLR 2024
Videos
Structure Conditions
Vid2Vido-Zero
ControlVidee (Ours)
An. Audi Q7 goes on snow trail, aerial view with now-capped mountains in the background.
Drone flyover of the Canadian National Tower, surrounded by martian desert.
Figure 18: Qualitative comparisons with Vid2Vid-Zero. Inconsistent objects and prompts are colored in red.
21


Reviews:
[{"review_id": "aOtemRuSfO", "reviewer": "Reviewer_XMqd", "summary": "This paper introduces \"ControlVideo,\" a training-free framework that significantly improves text-driven video generation. It addresses issues like appearance inconsistency and flickers in long videos through innovative modules for frame interaction and smoothing. ControlVideo outperforms existing methods, efficiently generating high-quality videos within minutes.", "questions": "•\tCould you provide additional results for long video generation to further validate the method's efficacy?\n\n•\tIs there a potential solution to the flickering background issue mentioned in the second weakness?\n\n•\tWould it be possible to employ a non-deterministic DDPM-style sampler as an alternative to DDIM?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "•\tThe proposed method is straightforward, easily implementable, and reproducible, making it accessible for further research and application.\n\n•\tThe paper introduces novel techniques for long video generation, and the \"interleaved-frame smoother\" effectively improves frame consistency.\n\n•\tThe results demonstrate improvements over existing methods, substantiating the paper's claims.", "weaknesses": "•\tWhile the full-attention mechanism and \"interleaved-frame smoother\" enhance frame consistency, they also significantly increase the computational time.\n\n•\tThe background appears to flicker in relation to the foreground in some examples. For instance, in the \"James Bond moonwalk on the beach, animation style\" video on the provided website, the moon inconsistently appears and disappears.\n\n•\tThe paper lacks quantitative comparisons with Text2Video-Zero in the context of pose conditions, which could be a significant oversight given the importance of pose in video generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bM87LlRDtA", "reviewer": "Reviewer_HJ3a", "summary": "This paper proposes ControlVideo, a training-free framework that can produce high-quality videos based on the provided text prompts and motion sequences (e.g., different modalities). ControlVideo adapts a pre-trained text-to-image model (i.e., ControlNet) for controllable text-to-video generation. The paper introduces an interleaved-frame smoother that alternately smooths out the latents of successive three-frame clips by updating the middle frame with the interpolation among the other two frames in the latent space, aiming to stabilize the temporal continuity of the generated videos. Besides, a fully cross-frame interaction mechanism is exploited to further enhance the frame\nconsistency, and a hierarchical sampler is employed to produce long videos more efficiently. Experimental results demonstrate that the proposed ControlVideo outperforms the state-of-the-art baselines both quantitatively and qualitatively.", "questions": "- Why does the hierarchal sampler improve model efficiency? It seems all the frames still need to be generated, although it is a top-down generation from key frames.\n\n- It is suggested to remove some content about the background and preliminary since such information is well-known.\n\n- The reviewer is interested if the proposed ControlVideo can be extended to generate more challenging new information, such as a novel view/unseen part of an object.\n\n-  Will the authors release all the code, models, and data to ensure the reproducibility of this work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written, well organized, and easy to follow. The symbols, terms, and concepts are adequately defined and explained. The language usage is good.\n\n- The proposed method is simple and easy to understand. Sufficient details are provided for the readers.\n\n- The experiments are generally well-executed. The empirical results show the effectiveness of the proposed method, showing certain advantages over state-of-the-art baselines.", "weaknesses": "- The qualitative results showcase certain advantages of the proposed method over state-of-the-art baselines in controllable text-to-video generation. However, by checking the provided video results, the temporal consistency can still be improved. Also, in some cases, the background looks unchanged. Some visual details can still be improved. Providing more discussions on these could strengthen this paper further.\n\n- The fully cross-frame interaction mechanism considers all frames as the reference, which thus increases the computational burden. What is the intuition to consider all the frames as a large image? Why not select some key frames to reduce redundant information? It is interesting to provide more discussions and analysis on this.\n\n- The paper mentioned that the proposed interleaved-frame smoother is performed on predicted RGB frames at timesteps {30, 31} by default. It can be more interesting if more studies and analyses on different steps to apply such a mechanism are provided.\n\n- It seems the interleaved-frame smoother still brings more computational cost and affects the model efficiency due to the additional conversion and interpolation steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T1ClNll9KI", "reviewer": "Reviewer_aLwG", "summary": "This work focuses on training-free controllable text-to-video generation tasks. It introduces an interleaved-frame smoother method to generate smoother frames. Additionally, it modifies cross-frame interaction to better utilize Stablediffusion's weights, enhancing frame continuity.", "questions": "As shown above, despite the method's average innovation and some shortcomings, I believe the exploration in this direction is worthwhile. \n- I hope the authors can complete more experiments and cases, preferably providing an analysis of failure cases. \n- Relying solely on the demo examples provided in the paper makes it challenging to be fully convinced.\n- If the authors can address my concerns, I will consider giving a higher score.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The writing is clear and easy to follow.\n- It is a training-free method, not relying on large-scale training, and has low computational resource requirements.\n- The ablation experiments are well-designed and easy to understand.", "weaknesses": "- Overall, the innovation is average; applying ControlNet to video editing or generation is straightforward and easily thought of.\n- The experiments are not comprehensive; there are too few baseline comparisons, and the experimental validation is limited to just over 20 examples, making the results less convincing.\n- Limited by the absence of structure condition, this method can mainly edit videos with similar motion. Its effectiveness diminishes for videos with different motions or poses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WLE6qF9WVd", "reviewer": "Reviewer_EvXG", "summary": "This paper proposes a trainig-free framework to produce videos based on provided text prompts and motion sequences. An interleaved-frames smoother and a fully cross-frame interaction mechanism with a hierarchical sampler are proposed to enhance the quality of synthesized videos. The authors demonstrate that they achieve sota performance by entensive experiments.", "questions": "Just as stated in the weakenss section, I am skeptical about the potential of training-free framework in video generation area. I'd be keen to hear the authors discuss potential future research directions in this direction.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy-to-follow. \n2. The proposed method is highly efficient and does not need training at all. Nevertheless, the quality of synthesized videos are not bad.", "weaknesses": "1. The proposed components (inter-frame interpolation, and cross-frame attention) are more-or-less explored in recent works. As such, I'm uncertain if this research will provide substantial insights to the community.\n\n2. The proposed metrics (frame consistency, and prompt consistency) leverage CLIP model. Given that the CLIP model primarily operates in a deeply semantic and abstract domain, it often misses finer image details. Consequently, I'm inclined to think that the suggested metric might not adequately assess temporal consistency (for example, critizing jittering and discontinuity). Thus, the assertion that the proposed methods attain improved temporal consistency seems to lack robust quantitative backing.\n\n3. This training-free framework cannot capture fine-grained motion pattern in video. Therefore, I believe it may not be the optimal approach for producing high-quality video content. Instead, I think finetuning the model on large-corpus video data might help improve the quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)