VideoComposer: Compositional Video Synthesis with Motion Controllability
Authors: Xiang Wang
Hangjie Yuan
Shiwei Zhang
Dayou Chen
Jiuniu Wang
Yingya Zhang
Yujun Shen
Deli Zhao
Jingren Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Synthesis
Video Diffusion Model
Compositional Synthesis

Abstract: The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models are publicly available at
https://videocomposer.github.io.

Compocr: Compositional Video Synthesis with Motion Controllability
Xiang Wang
Hangjie Yuan
Shiwei Zhang
Dayou Chen' Jiuniu Wang'
Yingya Zhang'
Yujun Shen
Deli Zhao?
Jingren Zhou!
Alibaba Group Ant Group
(xiaolao. wx, mannanggjie.c yhj, zhangjin zsw.aaalbabb--in com
[dayou. cdy, wangjiuniu. wjn, yingya. zyy, jingren (ahhuuaalllbaaaa incaaaiiinicen com (shenyujun0302, zhaodel @gmail.
Abstract
The pursuit of controllability as higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement cross-frame temporal consistency. Based the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos explicit control signal provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporte Condition encoder (STC- encoder) that serves unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models are publicly available https::/videoccoposse .git.ub..ttt.....
1 Introduction
Driven by the advances in computation, data scaling and architectural design, current visual generative models, especially diffusion- based models, have made remarkable strides automating content creation, empowering designers to generate realistic images or videos from textual prompt as input [24, 48, 53, 62]. These approaches typically train a powerful diffusion model [48] conditioned by text [23] on large-scale video-text and image-text datasets [2, 51], reaching unprecedented levels of fidelity and diversity. However, despite this impressive progress, a significant challenge remains in the limited controllability of the synthesis system, which impedes its practical applications.
Most existing methods typically achieve controllable generation mainly by introducing new conditions, such as segmentation maps [48, 65], inpainting masks [72] or sketches [38, 79], in addition to texts. Expanding upon this idea, Composer [28] proposes new generative paradigm centered on the concept of compositionality which is capable of composing image with various input conditions, leading to remarkable flexibility. However, Composer primarily focuses on considering
"Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

"Rotation view of beautiful long haired slanding the forest"
(a) Text and sketch
Image and depth
(c) Style, depth, skeich and motions
moving golden moon"
(d) Tex, style, hand-crafted motions and hand-crafted sketch
Figure 1: Compositional video synthesis. (a-c) VideoComposer is capable of generating videos that adhere textual, spatial and temporal conditions their subsets; (d) VideoComposer can synthesize videos conforming to expected motion patterns (red stroke) and shape patterns (white stroke) derived from two simple strokes.
multi-level conditions within the spatial dimension, hence may encounter difficultics when comes video generation due the inherent properties of video data. This challenge arises from the complex temporal structure of videos, which exhibits a large variation of temporal dynamics while simultaneously maintaining temporal continuity among different frames. Therefore, incorporating suitable temporal conditions with spatial clues to facilitate controllable video synthesis becomes significantly essential.
Above observations motivate the proposed VideoComposer, which equips video synthesis with improved controllability in both spatial and temporal perception. For this purpose, we decompose video into three kinds of representative factors, i.e., textual condition, spatial conditions and the crucial temporal conditions, and then train latent diffusion model to recompose the input video conditioned by them. In particular, we introduce the video-specific motion vector kind of temporal guidance during video synthesis to explicitly capture the inter-frame dynamics, thereby providing direct control over the internal motions. To ensure temporal consistency, we additionally present unified STC-encoder that captures spatio-temporal relations within sequential input utilizing cross-frame attention mechanisms, leading to an enhanced cross-frame consistency of the output videos. Moreover, STC-encoder serves as an interface that allows for efficient and unified utilization of the control signals from various condition sequences. As result, VideoComposer is

---

capable of flexibly composing video with diverse conditions while simultaneously maintaining the synthesis quality, as shown in Fig. 1. Notably, we can even control the motion patterns with simple hand-crafted motions, such as an arrow indicating the moon's 's trajectory in Fig. 1 feat that is nearly impossible with current methods. Finally, we demonstrate the efficacy of VideoComposer through extensive qualitative and quantitative results, and achieve exceptional creativity in the various downstream generative tasks.
Related work
Image synthesis with diffusion models. Recently, research efforts on image synthesis have shifted from utilizing GANs [18], VAEs [31], and flow models [14] to diffusion models [9, 19, 23, 32, 34, 54, 59, 74, 80, 81] due more stable training, enhanced sample quality, and increased flexibility in a conditional generation. Regarding image generation, notable works such as DALL-E 2 [46] and GLIDE [40] employ diffusion models for text-to-image generation by conducting the diffusion process in pixel space, guided by CLIP [44] or classifier-free approaches. Imagen [50] introduces generic large language models, i.e., T5 [45]. improving sample fidelity. The pioneering work LDMs [48] uses an autoencoder [15] to reduce pixel-level redundancy, making LDMs computationally efficient. Regarding image editing. pix2pix-zero [42] and compt-to-promyy editing [21] follow instructional texts by manipulating cross-atiention maps. Imagic [29] interpolates between an optimized embedding
and the target embedding derived from text instructions to manipulate images. DiffEdit [1 1] introduces
automatically generated masks to assist text-driven image editing. To enable conditional synthesis with flexible input, ControlNet [79] and T2I-Adapter [38] incorporate a specific spatial condition into the model, providing more fine-grained control. One milestone, Composer [28], trains multi-condition diffusion model that broadly expands the control space and displays remarkable results. Nonetheless, this compositionality has not yet been proven effective in video synthesis, and VideoComposer aims to fill this gap.
Video synthesis with diffusion models. Previous methods [13, 58, 67] usually adopt GANs for video synthesis. Recent research has demonstrated the potential of employing diffusion models for high-quality video synthesis [5, 20, 25, 30, 36, 62, 75]. Notably, Imagen Video [24] and Make-A- Video [53] both model the video distribution in pixel space, which limits their applicability due to high computational demands. In contrast, Magic [83] models the video distribution in the latent space, following the paradigm of LDMMs [48], significantly reducing computational overhead. With the goal of editing videos guided by texts, VideoP2P [33] and vid2vid-zero [66] manipulate the cross- attention map, while Dreamix [37] proposes an image-video mixed fine-tuning strategy. However, their generation editing processes solely rely text-based instructions [44, 45]. A subsequent work, Gen-1 [16], integrates depth maps alongside texts using cross- mechanisms to provide structural guidance. Both MCDiff [8] and LaMD [27] target motion-guided video generation; the former focuses on generating human action videos and encodes the dynamics by tracking the keypoints and reference points, while the latter employs learnable motion latent to improve quality. Nevertheless, incorporating the guidance from efficient motion vectors or incorporating multiple guiding conditions within a single model is seldom explored in the general video synthesis field.
Motion modeling. Motion cues play a crucial role in video understanding fields, such as action recognition [1, 4, 6, 43, 60, 63, 64], action detection [10, 68, 77, 82], human video generation [39, 41, 67], etc. Pioneering works [1, 6, 39, 43, 63, 67] usually leverage hand-crafted dense optical flow [76] to embed motion information or design various temporal structures to encode long-range temporal representations. Due to the high computational demands of optical flow extraction, several attempts in compressed video recognition [7, 52, 69, 78] have begun to utilize more efficient motion vectors as an alternative to represent motions and have shown promising performance. In contrast to these works, we delve into the role of motions in video synthesis and demonstrate that motion vectors can enhance temporal controllability through a well-designed architecture.
3 VideoComposer
In this section, we will comprehensively present VideoComposer to showcase how can enhance the controllability of video synthesis and enable the creation of highly customized videos. Firstly, we in brief introduce Video Latent Diffusion Models (VIDMMs) upon which VideoComposer is designed, given their impressive success in various generative tasks. Subsequently, we delve into the details of
3

---

Video Latent Diffusion Model
STC-encoder
Text
xn (RF)xCoha
Style
Decoder
SiLU
AvgPool2D
Noise
Conditon Fusion
SILUU
STC- encoder
STC-
STC-
STC-
STC- escoder
STC-
Conv213
Temporal Trans.
Single Image
Single Sketch
Motion
Depth Seg.
Mask Sey.
Sketch Sey.
Concatenate
Reshame
Textual Condition
Spatial Conditions
Temporal Conditions
Figure 2: Overall architecture of VideoComposer, First, video is decomposed into three types of conditions, including textual condition, spatial conditions and temporal conditions. Then, we feed these conditions into the unified STC-encoder or the CLIP model to embed control signals. Finally, the resulting conditions are leveraged to jointly guide VLDMs for denoising.
VideoComposer's architecture, including the composable conditions and unified Spatio- Temporal Condition encoder (STC-encoder) as illustrated in Fig. 2. Finally, the concrete implementations, including the training and inference processes, will analyzed.
3.1 Preliminaries
Compared images, processing video requires substantial computational resources. Intuitively, adapting image diffusion models that process in the pixel space [40, 46] to the video domain impedes the scaling of VideoComposer to web-scale data. Consequently, we adopt a variant of L.DMs that operate in the latent space, where local fidelity could be maintained to preserve the visual manifold.
Perceptual video compression. To efficiently process video data, follow LDMs by introducing a pre-trained encoder [15] to project given video E fx xW x3 into a latent representation R. xxhxxx Subsequently, decoder is adopted map the latent representations back to the pixel space æ = D(z). We set H/h = W/w = 8 for rapid processing.
E(x), where
Diffusion models in the latent space. To learn the actual video distribution P(x), diffusion models [23, 54] learn to denoise a normally-distne distributerdddd noise, aiming to recover realistic visual content. This process simulates the reverse process of a Markov Chain of length T. T is set to 1000 by default. To perform the reverse process on the latent, it injects noise to z to obtain a noise-corrupted latent z, following [48]. Subsequently, apply denoising function co(; t) on 2, and selected conditions c, where E (1....,T) T). The optimized objective can be formulated as:
LYLDM = (E [lle (z c, 1
To exploit the inductive bias of locality and temporal inductive bias of sequentiality during denoising, we instantiate ( as 3D UNet augmented with temporal convolution and cross-attention mechanism following [25, 49, 62].
3.2 VideoComposer
Videos composable conditions. We decompose videos into three distinct types of conditions, i.e. textual conditions, spatial conditions and crucially temporal conditions, which can jointly determine the spatial and temporal patterns in videos. Notably, VideoComposer is generic compositional framework. Therefore, more customized conditions can incorporated into VideoComposer depending the downstream application and are not limited to the decompositions listed above.

---

Textual condition. Textual descriptions provide an intuitive indication of videas in terms of coarse- grained visual content and motions. In our implementation, we employ the widely used pre-trained text encoder from OpenCIIP ViT-H/14 to obtain semantic embeddings of text descriptions.
Spatial conditions. To achieve fine-grained spatial control and diverse stylization, we apply three spatial conditions to provide structural and stylistic guidance: i) Single image. Video is made up of consecutive images, and single image usually reveals the content and structure of this video. We select the first frame of a given video as spatial condition to perform image-to-video generation. ii) Single sketch. We extract sketch of the first video frame using PiDiNet [55] as the second spatial condition and encourage VideoComposer to synthesize temporal conssssett video according to the structure and texture within the single sketch. iii) Style. To further transfer the style from one image to the synthesized video, we choose the image embedding as the stylistic guidance, following [3, 28]. We apply pre-trained image encoder from OpenCI.IP ViT-H/14 to extract the stylistic representation.
Temporal conditions. To accomplish finer control along the temporal dimension, we introduce four temporal conditions: i) Motion vector. Motion vector video- specific element is represented as two-dimension vectors, i.e., horizontal and vertical orientations. It explicitly encodes the pixel-wise movements between two adjacent frames, visualized by red arrows in Fig. 3. Due to the natural properties of motion vector, we treat this condition synthesis. Following [52, 69], we extract motion vectors in standard MPEG-4 format from compressed videos. ii) Depth sequence. To introduce depth information, we utilize the pre-trained model from [47] to extract depth maps of video frames. iii) Mask sequence. To facilitate video regional editing and inpainting, we manually add masks. We introduce tube masks [17, 57] to mask out videos and enforce the model to predict the masked regions based on observable information. iv) Sketch sequence. Compared with the single sketch, sketch sequence can provide more control details and thus achieve precisely customized synthesis.
as motion control signal for temporal-smode
Figure 3: Examples of motion vectors.
STC-encoder. Sequential conditions contain rich and complex space-time dependencies, posing challenges for controllable guidance. In order to enhance the temporal awareness of input conditions, we design a Spatio-Temporal Condition encoder (STC-encoder) to incorporate the space-time relations, as shown in Fig. 2. Specifically, a light-weight spatial architecture consisting of two 2D convolutions and an average pooling layer is first applied to the input sequences, aiming to extract local spatial information. Subsequently, the resulting condition sequence is fed into a temporal Transformer layer [61] for temporal modeling. In this way. STC-encoder facilitates the explicit embedding of temporal cues, allowing for a unified condition interface for diverse inputs, thereby enhancing inter- frame- consistency. It worth noting that we repeat the spatial conditions of single image and single sketch along the temporal dimension to ensure their consistency with temporal conditions, hence facilitating the condition fusion process.
After processing the conditions by STC-encoder, the final condition sequences are all in an identical spatial shape z, and then fused by element-wise addition. Finally, we concatenate the merged condition sequence with z, along the channel dimension as control signals. For textual and stylistic conditions organized as sequence of embeddings, utilize the cross-attention mechanism inject textual and stylistic guidance.
3.3 Training and inference
Two-stage training strategy. Although VideoComposer can initialize with the pre-training of LDMs [48], which mitigates the training difficulty to some extent, the model still struggles in learning to simultancously handle temporal dynamics and synthesize video content from multiple compositions. To address this issue, we leverage a two-stage training strategy to optimize VideoComposer. Specifically, the first stage targets pre-training the model to specialize in temporal modeling through text-to-video generation. In the second stage, we optimize Videooomposer to excel in video synthesis controlled by the diverse conditions through compositional training.
Inference. During inference, DDIM [80] employed to enhance the sample quality and improve inference efficiency. We incorporate Hassifier-frre guidance [22] to ensure that the generative results
5

---

"Smiling woman in comboyy hat with whest
"Tiger"
the Autumn gidl walking city pack"
yellow
"Honcy collecting bloosting sunflower"
"The Matterhenn foreground"
"Aetial 11A track over Rosphumu in Istanbal
Figure 4: Compositional image-to-vider generation. We showcase six examples, each displaying spatial condition and textual
generated videos. upper video is generated using given single condition describing the scene. The lower video is generated by incorporating an additional sequence temporal conditions to facilitate finer control over the temporally evolving structure.
adhere to specified conditions. The generative process can be formalized
( c, cæ(ze, c) w (eo c2, Ez(2,, C1, t))
where w is the guidance scale: C₁ and C2 are two sets of conditions. This guidance mechanism extrapolates between two condition sets, placing emphasis the elements in (c2 \ ei) and empowering flexible application. For instance, in text-driven video inpainting, C2 represents the expected caption and masked video, while e₁ is empty caption and the same masked video.
Experiments
4.1
Experimental setup
Datasets. To optimize VideoComposer, leverage two widely recognized and publicly accessible datasets: VebVid10M [2] and LAION-400M [51]. WebVidIOM [2] is a large-scale benchmark scrapped from the web that contains 10.3M video-caption pairs. LAION-400M [51] is image- caption paired dataset, filtered using CLIP [44].
Evaluation metrics. We utilize two metrics to evaluate VideoComposer: To evaluate video continuity, we follow Gen-l [16] to compute the average CLIP cosine similarity two consecutive frames, serving frame consistency metric; ii) To evaluate motion controllability, we adopt end-point-error [56, 73] as motion control metric, which measures the Euclidean distance between the predieted and the ground truth optical flow for each pixel.
4.2 Composable video generation with versatile conditions
In this section, we demonstrate the ability of VideoComposer to tackle various tasks in a controllable and versatile manner, leveraging its inherent compositionality. It's important note that the conditions employed in these examples are customizable to specific requirements. We also provide additional results in the supplementary material for further reference.

---

Mask
TwanS bazily dritting trimming
T2-Black swan lazily drifting and trimming toself on calm river"
T3: "Coastal view of Cleveland Road and Robin Hood's Bay, blue sky and sea with white clouds"
Figure 5: Compositional video inpainting, By manually adding masks to videos, VideoComposer can perform video inpainting. facilitating the restoration of the corrupted parts according textual instructions. Furthermore, by incorporating temporal conditions specifying the visual structure, VideoComposer perform customized inpainting that conforms to the prescribed structure.
Compasitional Image-to-videu generation. Compositional training with single image endows VideoComposer with the ability of animating static images. Fig. 4, we present six examples to demonstrate this ability. VideoComposer is capable of synthesizing videos conformed to texts and the initial frame. To further obtain enhanced control over the structure, we can incorporate additional temporal conditions. We observe resultant videos consistently adhere to the given conditions.
Compositional video inpainting. Jointly training with masked video endows the model with the ability of filling the masked regions with prescribed content, as shown in Fig. 5. VideoComposer can replenish the mask-corrupted regions based textual descriptions. By further incorporating temporal conditions, i.e, depth maps and sketches, we obtain more advanced control over the structure.
Compositional sketch-to-video generation. Compositional training with single sketch empowers VideoComposer with the ability animating static sketches, as illustrated in Fig. 6. We observe that VideoComposer synthesizes videos conforming to texts and the initial sketch. Furthermore, observe that the inclusion mask and style guidance can facilitate structure and style control.
4.3 Comparative experimental results
Depth-to-video controllability comparison. In Fig. we compare our VideoComposer with TextoVidoo-eero [30] and existing fate-of-the-art Gen-l [16]. We observed that Text2Video-Zen suffers from appearance inconsistency and structural flickering due to the lack of temporal awareness. Meanwhile, Gen-l produces video with color inconsistency and structure misalignment (revealed by the orientation of the bird head). The video generated by VideoComposer is faithful to the structure of the input depth sequence and maintains continuous appearance. This shows the superiority of our VideoComposer terms of controllability.
Text-fo-video generation performance. Although VideoComposer is not specifically tailored for text-to-video generation, versatility allows VideoComposer perform the traditional text-to-video generation task effectively. In Tab. 1, follow the evaluation settings in Video LDM [5] to adopt

---

"Pigeon not behind
"A beautiful looking in office"
Shrike collurie"
Figure 6: Compositional sketch-fo-video generation. In the first example, the upper video is generated using text a single sketch as conditions, while the lower generated by using an additional mask sequence for finer control over the temporal patterns. For the last two examples, the upper video is generated using a single sketch and a textual condition, while the lower is generated with an additional style from specified image.
Table I: Text-to-video generation performance on MSR-VTT.
Method
Zero-shot FVD J
CLIPSIM
GODIVA [70]
No
0,2402
Nüwa [71]
No
0.2439
CogVideo (Chinese) [26]
Yes
0.2614
Cog Video (English) [26]
Yes
1294
0.2631
Magic Video [83]
Yes
1290
Make-A A-Video [53]
Yes
0.3049
Video LDM [5]
Yes
0.2929
Text-to-video pre-training (First stage)
Yes
803
0,2876
VideoComposer
Yes
580
0.2932
Fréchet Video Distance (FVD) and CLIP Similarity (CLIPSIM) as evaluation metrics and present the quantitative results of text-to-video generation on MSR-VTT dataset compared to other existing methods. The results the table demonstrate that VideoComposer achieves competitive performance compared to state-of-the-ee ohtthe-rrt text-to-video approaches. In addition, VideoComposer outperforms our first-stage text-to-vides viveo pre-training, demonstrating that VideoComposer can achieve compositional generation without sacrificing its capability of text-to-video generation. In the future, we aim to advance VideoComposer by leveraging stronger text-to-video models for more powerful synthesis.
4.4 Experimental results motion control
Quantitative evaluation. To validate superior motion controllability, we utilize the motion control metric. We randomly select 1000 caption-video pairs and synthesize corresponding videos. The results are presented in Tab. 2. We observe that the inclusion of motion vectors as condition reduce the motion control error, indicating an enhancement of motion controllability. The incorporation of STC-encoder further advances the motion controllability.

---

"Bom Swallow nesting bird box"
Input depth
Text2Video Zero
Gen-1
VideoComposer
Figure 7: Comparison with methods for controllable video generation. Results are generated utilizing the textual condition and the depth sequence in the first row. Tex(2Vieeo-Zero suffers from appearance inconsistency and structural flickering due the lack of temporal awareness. Gen-l produces video with color inconsistency structural misalignment. The video generated by VideoComposer is faithful the structure of the input depth sequence and maintains continuous appearance.
"American Staffordshire
lying
abandomed building"
(a) Source video
(b) Depth-gided generation
(c) Sketch Sccich-guded generation
(d) Motion guided generation
Figure 8: Video-to-video translation. We extract sequence of depth maps, sketches or motion vectors from the source video. along with textual descriptions, perform the translation. By utilizing motion vectors, achieve static- backe-back removal.
Evaluating the motion controllability. "Text" and "MV" represent the utilization and motion vectors as conditions for generation.
Table 2:
Motion vectors prioritizing moving visual Thanks to the nature of motion vectors, which encode inter-frame variation, static regions within an image are inherently omitted. This prioritization of moving regions facilitates motion control during synthesis. In Fig. 8, we present results of video- to-video translation substantiate such superiority. We observe that motion vectors exclude the static background, i.e., human legs, a feat that other temporal conditions such as depth maps and sketches cannot accomplish. This advantage lays the foundation for broader range of applications.
Method
Text MV Motion control 4
w/o STC-encoder
4.03
w/o STC-encoder
2.67
VideoCompoaer
2.18
Versatile motion control with motion vectors. Motion vectors, casily derived from hand-crafted strokes, enable more versatile motion control. In Fig. 9, we present visualization comparing Video [26] and VideoComposer. While Video limited insufficient text-guided motion control, VideoComposer expands this functionality by additionally leveraging motion vectors derived from hand-crafted strokes facilitate more flexible and precise motion control.
4.5 Ablation study
In this subsection, we conduct qualitative and quantitative analysis on VideoComposer, aiming to demonstrate the effectiveness of incorporating STC-encoder.
Quantitative analysis. In Tab. 3, we present the frame consistency metric computed on 1000 test videos. We observe that incorporating STC-encoder augments the frame consistency, which

---

box Rom left to right"
(e) Vilev
AA moving box"
"A figer walking the grussland"
(b) VideCoompsser: hand-erafied motions
Figure 9: Versatile motion control using hand-crafted motions. (a) Limited motion control using
CogVideo [26]. (b) Fine- tine-grained and ficxible motion control, empowered by VideoComposer.
enprace Srcceee Crediton
Sansage
"Young Iaby playing beal"
Mexion
Figure 10: Qualitative ablation study. Wc present three representative examples. The last two rows of vidcos display generated videos conditioned on textual condition and one additional temporal condition (i.e., sketches, depth maps motion vectors). Regions exhibiting deficiencies fidelity are emphasized within red boxes.
attribute to its temporal modeling capacity. This observation holds for various temporal conditions such sketches, depth maps and motion vectors.
Qualitative analysis. In Fig. 10, we exem- plify the usefulness of STC-encoder. We observe that in the first example, videos generated by VideoComposer without STC- encoder generally adhere to the sketches but omit certain detailed information, such as several round-shaped ingredients. For the left two examples, VideoComposer without STC- encoder generates videos that are structurally inconsistent with conditions. We can also spot the noticeable defects in terms of human faces and poses. Thus, all the above examples can validate the efffcctiveness of STC-encoder.
Table 3: Quantitative ablation study of STC-encoder.
"Conditions" denotes the conditions utilized for generation.
Method
Conditions
Frame consistency t
w/o STC-encoder
Text and sketch sequence
0.910
Vi VideoComposer
0.923
w/o STC-encoder
Text and depth sequence
0.922
VideoComposer w/o STC-eneoder
0.928
Text and motion vectors
0.915
VideoComposer
0.927
Conclusion
In this paper, we present VideoComposer, which aims explore the compositionality within the realm of video synthesis, striving to obtain flexible and controllable synthesis system. In particular, we explore the use of temporal conditions for videos, specifically motion vectors, as powerful control signals to provide guidance in terms of temporal dynamics. An STC-encoder is further designed as a unified interface to aggregate the spatial and temporal dependencies of the sequential inputs for fiter-frame consistency. Our experiments, which involve the combination of various conditions to augment controllability, underscore the pivotal role of our design choices and reveal the impressive creativity of the proposed VideoComposer.
10

---

References
[1] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: video vision transformer. In ICCV, pages (836-6846, 2021.
[2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman, Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1728 -1788, 2021.
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine. Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers, arXiv preprint 22 2022.
[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space- timee-time attention all you need for video understanding? In ICML, volume 2. page 4, 2021.
[5] Andreas Blattmann. Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pages 22563-22575, 2023.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pages 6299-6308, 2017.
[7] Jiawei Chen and Chiu Man Ho. Mm-vit: Multi-modal video transformer for compressed video action recognition. In WACV, pages 1910-1921, 2022.
[8] Tsai-Shien Chen, Chich Hubert Lin, Hung-Yo Tseng, Tsung- Yi Lin, and Ming-Hsuan Hsuan Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304. 14404, 2023.
[9] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arXiv preprint arXiv:2307.0944 09481, 2023.
[10] Feng Cheng and Gedas Bertasius. Tallformer: Temporal action localization with a long-oomooo transformer. In ECCV, pages 503-521, 2022.
[11] Guillaume Couairon, Jakoh Verbeck, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint 27X14:2210.11427, 2022.
[12] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. NeurIPS, pages 16344-16359, 2022.
[13] Emily Denton Rob Fergus. Stochastic video generation with learned prior. ICMI., pages 1174-1183. PMI.R 2018.
[14] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint
2014.
[15] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-1222 12883, 2021.
[16] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint 1rXiv:2302.03011 2023.
[17] Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal leamers. In NewrlPS, pages 35946-35955 2022.
[18] Ian Goodfellow. Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, pages 139 199-144,22 2020.
[19] Shaozhe Hao, Kai Han, Shihao Zhao, Kwan- Yee Wong. Vico: Detail- preie-ireseriing visual condition for personalized text-to-ine text-to-image generation. arXiv preprint arXiv:2306.0097 2023.
[20] William Harvey, Sacid Nadcriparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. arXiv preprint arXiv:2205. 11495, 2022.
[21] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman. Yael Pritch, and Daniel Cohen-Or, Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint Xrxiv:2008.000 01626, 2022.
11

---

[22] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXi::2207112598 2022.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prohabilistic models. NewrlPS, pages 6840-6851, 2020.
[24] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 0rXi:22100023003 2022.
[25] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.0344 2022.
[26] Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:220S. 15868, 2022.
[27] Yaosi Hu, Zhenzhong Chen, and Chong Luo. Lamd: Latent motion diffusion for video generation. arXiv preprint arXiv:2304.1100) 2023.
[28] Lianghua Huang. Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou, Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint 2302:22302.0977,, 2023,
[29] Bahjat Kawar. Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276. 2022.
[30] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roherto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint 2rXXi:2303.13430 2023.
[31] Diederik Kingma and Max Welling, auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.
[32] Pandeng Li, Then-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. In NewrIPS, 2023.
[33] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross- attention control. arXiv preprint arXiv:2303.04770 2023.
[34] Zhiheng Liu. Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05125.1 2023.
[35] Ilya Loshchilov and Frank Hutter. Decoupled xxXiv:/711.05100,,200 2017.
weight decay regularization. arXiv preprint
[36] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Vidcofusion: Decomposed diffusion models for high-quality video generation. In CVPR, 2023.
[37] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch. Yaniv Leviathan. and Yedid Hoshen. Dreamix: Video diffusion models general video editors. arXiv preprint arXiv:2302.01329 2023.
[38] Chong Mou, Xintao Wang. Lianghin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image di diffusion models, arXiv preprint arXiv:2302. 08445.. 2023.
[39] Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Rengiang Min. Conditional image-to- video generation with latent flow diffusion models. In CVPR, pages 18444-18455, 2023.
[40] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 2xXiv:2112.10744 2021.
[41] Katsumori Ohnishi, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Hierarchical vidco generation from orthogonal information: Optical flow and texture. In AAAI, volume 32, 2018
[42] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun Yan Zhu. Zero-shot image-to-ing tomiiage translation. arXiv preprint 20 2023.
12

---

[43] Zhaofan Qiu. Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV. pages 5533-5541, 2017.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML. pages 8748-878 87788877.2222 2021.
[45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, pages 5485-5551, 2020.
[46] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122. 2022.
[47] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Puttern Analysis and Machine Intelligence, pages 1623 1637, 2020.
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[49] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234 2444-241,22 2015.
[50] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NeurIPS, pages
2022.
[SI] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-texi pairs. arXiv preprint arXiv:2111.02114 2021.
[52] Zheng Shou, Xudong Lin, Yannis Kalantidis, Laura Sevill--Lara, Marcus Rohrbach, Shih-Fu Chang, and Zhicheng Yan. Dmc-net: Generating discriminative motion cues for fast compressed video action recognition. In CVPR, pages 1268-1277, 2019.
[53] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video- text-to-viden generation without text-video data. arXiv preprint 2rXiv:2009. 14792, 2022.
[54] Jascha Dickssiit,,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, In ICCV, pages 2256-2266 2265, 2015.
[55] Zhuo Su. Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen, and Li Liu. Pixel difference networks for efficient edge detection. In ICCV, pages 5117-5127, 2021.
[56] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, pages 402-419, 2020.
[57] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autocncoders are data-efficient learners for self-supervised video pre-training In NeurlPS, 2022.
[58] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, pages 1526-1535, 2018.
[59] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. NeurIPS, pages 11287-11300 1302, 2021.
[60] Gül Varol, Ivan Laptev, and Cordelia Schmid. Long-term temporal convolutions for action recognition. IEEE transactions pattern analysis and machine intelligence, pages 1510-1517, 2017.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
[62] Jiuniu Wang Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint ar8/v:2308.06571, 2023.
[63] Limin Wang. Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, pages 20-36, 2016.
13

---

[64] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. In CVPR. pages 1895-1904, 2021.
[65] Tengfei Wang. Ting Zhang, Bo Zhang. Hao Ouyang. Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-ing translation. arXiv preprint arXiv:2205.12952. 2022.
[66] Wen Wang. Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang. and Chunhua Shen. Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303. 17599, 2023.
[67] Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. G3an: Disentangling appearance and motion for video generation. In CVPR, pages 5264-5273, 2020.
[68] Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid. Learning to track for spatio-temporal action localization. In ICCV, pages 3164-3172, 2015.
[69] Chao- Wu, Manzil Zaheer, Hexiang Hu, R Manmatha, Alexander J Smola, and Philipp Krähenbühhl Compressed video action recognition. In CVPR, pages 6026-6035, 2018.
[70] Chenfei Wu, Lun Huang. Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain vidcos from natural deseriptions. arXiv preprint orXiv:2104. 14806, 2021.
[71] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang. Yuejian Fang. Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In ECCV, pages 720-736. Springer, 2022.
[72] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.0503: 2022.
[73] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via glohal matching. In CVPR, pages 8121-8130, 2022.
[74] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Texl-to-image generation via large mixture diffusion paths. arXiv preprint arXiv:2305. 18295, 2023.
[75] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203.09481, 2022.
[76] Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-1 1 optical flow. In Pattern Recognition, pages 214-223, 2007.
[77] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan. Graph convolutional networks for temporal action localization. In ICCV, pages 7094-7103, 2019.
[78] Bowen Zhang. Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang. Real-time action recognition with enhanced motion vector cnns. In CVPR, pages 2718-2726, 2016.
[79] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffussoo models. arXiv preprint Xiv: 05543, 2023.
[80] Qinsheng Zhang. Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffuuson implicit models. arXiv preprint 22xiv22066005644 2022.
[81] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Yuan, Kwan Yee Wong. Uni-controlnet: All-in-one control to text-to -omagee--mmee diffusion models. In NeurlPS, 2023.
[82] Yue Zhao, Yuanjun Xiong, Limin Wang. Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In ICCV, pages 2914-2923 2017.
[83] Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
14

---

"Water is poured into the beer mug"
"Girl holding christmas lights"
"Luxury expensive table serving romantic dinner with candles"
"Woman designer making pattern for new dress"
"Woman designer making pattern new dress"
(Style)
Compositional sketch sequence-to-video generation. We showease five examples, each displaying video generated from a sequence of sketches and textual description. The final example additionally incorporates a style condition.
Figure All:
Appendix
In this Appendix, we first elaborate on more implementation details (Appendix A) and present more experimental results (Appendix B). Next, we provide section of discussion (Appendix C) on the limitations and potential societal impact of VideoComposer.
A More implementation details
Pre-training details. We adopt AdamW [35] the default optimizer with a learning rate set to 5 x 10-5, In total, VideoComposer is pre-trained for 400k steps, with the first and second stage being pre-trained for 132k steps and 268k steps, respectively. In terms of two-stage pre-training, we allocate one fourth of GPUs to perform image pre-training, while the rest of the GPUs are dedicated to video pre-training. We use center crop and randomly sample video frames to compose video input whose F 16, H 256 and W 256. During the second stage pre-training, we adhere [28], using a probability 0. to keep all conditions, a probability of 0.1 to discard conditions, and independent probability of 0.5 to keep or discard specific condition. Regarding the use of
15

---

Web [2], we sample frames from videos using various strides to ensure frame rate equal to 4, aiming to maintain consistent frame rate. We train VideoComposer jointly on video-text and image-text pairs by treating images 'one-frame" videos. When using image-text pairs for training, the shape the input noise 1 x h x w x C, where the temporal length is (i.e. F 1). In experiments, we use separate STC-encoders for different conditions without weight sharing. In our ablation study of STC-encoder, the baseline method (w/o STC-encoder) entails removing the temporal Transformer in STC-encoder while retaining the spatial convolution, designed to verify the effectiveness of incorporating temporal modeling. The spatial convolution remains in place to ensure the dimensions of all input conditions are consistent.
The structure of 3D UNet as cg(; ", t). To leverage the benefits of LDMs pre-trained on web-scale image data, i.e., Stable Diffusion", we extend the 2D UNet to 3D UNet by introducing temporal modeling layers. Specifically, within a single UNet block, we employ four essential building blocks: spatial convolution, temporal convolution, spatial transformer and temporal transformer. The spatial blocks are inherited from LDMs, while temporal processing blocks are newly introduced. Regarding temporal convolution,
stack four convolutions with 1 x 3 kernel, ensuring the temporal receptive field is ample for capturing temporal dependencies; regarding temporal transformer, we stack one Transformer layer and accelerate its inference using flash attention [12].
More details about two-stage training strategy. The two-stage training strategy utilized in VideoComposer is designed to methodologically address the learning challenge. In the first stage, VideoComposer focuses on learning the temporal dynamics by only leveraging the textual condition. This foundation allows for a focused understanding of temporal relationships within the video content. In the second stage, VideoComposer builds on the temporal modeling ability acquired from the first stage to perform compositional training. In this stage, it extends its learning by utilizing all three kinds of conditions: textual, spatial, and temporal conditions. The major difference between the two stages lies in this incorporation of additional conditions, leading to comprehensive learning of synthesizing video content from multiple compositions.
B More experimental results
In this subsection, we aim to provide additional experiments that complement the findings presented in the main paper and showcase more versatile controlling cases.
Compositional sketch sequence-to-vider generation. Compositional training with sketch sequences enables Vi VideoComposer to possess the ability of generation videos adhering sketch sequences. This generation paradigm lays more emphasis on the structure control, which differs from compositional sketch-to-video generation and can viewed as video-to-video translation. In Fig. A11, we exemplify this capacity. We observe videos' fidelity to the provided conditions, including texts, sketches and style.
Compositional depth sequence-to-video generation. Conducting compositional training with depth sequences allows Vi VideoComposer to effectively generate videos accordance with depth sequences. In Fig. A12, we illustrate this capability. Videos generated with VideoComposer faithfully adhere to the given conditions, including text prompts, depth maps, and style.
Motion transfer. Incorporating motion vectors as a composition of videos enables motion transferability. In Fig. A13, we conduct experiments demonstrate such capability. Through utilizing hand-crafted motion vectors or motion vectors extracted from off-the-shelf source videos, we can transfer the motion patterns to synthesized videos.
C Discussion
Limitations. Due to the absence of a publicly available large-scale and high-quality dataset, we have developed VideoComposer using the watermarked WebVidl0M dataset. As a result, the synthesized videos contain watermarks, which affect the generation quality and lead to less visually appealing results. Furthermore, in order reduce the training cost, the resolution of the generated videos limited to 256 Consequently, some delicate details might not be sufficiently clear. In the future,
"hapss//github..om/Sabbbllll Sita
16

---

"Fun hurse - 3D Animation"
ehutter oakenn enu
sek
oeken ook Bhunt oeck
"Ripe cherry tomatoes fall into bowl of water slow motion"
"Ginger and slieed lemon rotating round white plate"
Sliee Poutorsttee sstcck Mutfersteck corteeers
h
"Yellow daffodils open up their blossoms"
"Yellow daffodils open up their blossoms"
(Style)
Figure A12:
Compositional depth sequence-to-videe generation. We showcase examples. each displaying a video generated from a sequence of depth maps and textual description. The final example additionally incorporates style condition.
we plan to utilize super-resolution models to expand the resolution of the generated videos to improve the visual quality.
Potential societal impact. VideoComposer, as generic video synthesis technology, possesses the potential to revolutionize the content creation industry, offering unprecedented flexibility and creativity, and hence, promising significant commercial advantages. Traditional content creation processes are labor- and cost-intensive. VideoComposer could alleviate these burdens by enabling designers to manipulate subjects, styles, and scenes through instructions spanning human- wwrtttn text, and styles and subjects sourced from other images. Moreover, VideoComposer could potentially revolutionize education industry by creating unique and customized video scenarios for teaching complex concepts.
However, it's necessary to note that VideoComposer also represents a dual-use technology with inherent risks to society. As with prior generative foundation models, such as Imagen Video [24] and Make-A-Vides [53], VideoComposer inherits the implicit knowledge embedded within the pre-trained model (i.e., StableDiffussoo)) and the pre-trained dataset (i.e., WebVid and LAION). Potential issues include but not limited to the propagation of social biases (such as gender and racial bias) and the creation of offensive content.
17

---

Source video (Camera moves left)
Motion transfer
(Single image)
Source video
(Camera moves down and right)
Motion transfer
(Single image)
Source video (Camera moves left)
Motion transfer
(Single image)
(Hamd-crafied
mostions)
(Single imape)
Figure A13: Motion transfer. showcase four examples, each displaying video generated from a single image and motions. In the first three examples, we transfer the motion patterss in a source video to the generated video by extracting and utilizing motion vectors. The final example incorporates hand-crafted motions instcad.
Given that VideoComposer is research-oriented project aimed at investigating compositionality in diffusion-based video synthesis, our primary focus lies in scientific exploration and proof of concept. If VideoComposer is deployed beyond the scope of research, we strongly recommend several precautionary measures ensure its responsible and ethical use: (i) Rigorous evaluation and oversight of the deployment context should be conducted; (ii) Necessary filtering of prompts and generated content should be implemented to prevent misuse.
18


Reviews:
[{"review_id": "ImWl3Y6pz7", "reviewer": "Reviewer_akz1", "summary": "The paper presents a method for compositional video synthesis. It introduces motion vectors from compressed videos as a control signal for temporal dynamics. The motion vector can be combined by other conditions such as sketch, and depth map. Both qualitative and quantitative results show that the proposed method can control the spatial-temporal patterns. ", "questions": "1. In the video translation demo (2:11-2:16), the right example's output does not have a consistent color (white before jumping and brown afterwards). Is there any particular reason why the color consistency fails to hold in such a case?\n\n2. I'd suggest moving some quantitative results (Table A1) to the main text.\n", "limitations": "The authors have addressed the limitations in the supplementary materials.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The motion controlled generation result (fig 8) using hand-crafted strokes is interesting. \n\n+ Table A1 shows effectiveness of the proposed method quantitatively compared to previous methods.\n\n+ The paper is well written and easy to follow.", "weaknesses": "- There are a few GAN-based video synthesis approaches that are worth discussing in the related work. For example, MoCoGAN [1] approaches the problem by decomposing motion and content. \n \n- The two-stage training strategy needs more clarification. What is \"compositional training\" particularly in the second stage?  How does it differentiate from the \"text-to-video\" generation in the first stage?\n\n- In line 164-165, the authors \"repeat the spatial conditions of a single image and single sketch along the temporal dimension\". If the input condition is simply repeated, what's the point of applying a temporal Transformer? It will be equivalent to applying the spatial operation only and repeat at the latent space but with higher computation cost, no? (for motion vector, I totally agree that a spatial-temporal modeling would be necessary.)\n\n- Motion vectors can be less meaningful in the background due to lack of high-level semantics. It can also be clearly seen from the top row in Fig 4. I wonder if the authors treat the motion vector field equally for all locations. It seems that the generated results with motion conditions has more blurry background.\n\n- From Figure 2 and Figure 1(d), my impression is that the conditions (say motion and depth) can be combined together. However, in ablation studies (table 2), only one condition is added at a time. Another ablation that studies all combinations of these conditions will be favored. \n\n[1] Tulyakov, Sergey, et al. \"Mocogan: Decomposing motion and content for video generation.\" CVPR 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Discuss the GAN-based methods like MoCoGAN in the Related Work.**\n\nWe greatly appreciate your intention of improving the Related Work by comparing with GANs, such as MoCoGAN.\n- Different motivations. MoCoGAN is an unconditional method that aims to improve the quality of video generation, while VideoCompoer is a conditional approach to enhance the controllability during synthesis.\n- Different methodology. MoCoGAN decomposes videos into content and motion by sampling the latent gaussian noise from different space, but VideoComposer composes a video with textual, spatial, and temporal conditions in the input phase, resulting in a vast design space for customizable video creation.\n\nWe will include MoCoGAN and other GAN-based methods in our next version.\n\n\n\n**Q2: Clarifying the two-stage training strategy.**\n\n\nWe apologize for unclear presentation. The two-stage training is designed to methodologically address the learning challenge.\n\n- In the first stage, VideoComposer focuses on learning the temporal dynamics by only leveraging the textual condition.  This foundation allows for a focused understanding of temporal relationships within the video content.\n- In the second stage, VideoComposer performs **compositional training** by utilizing textual, spatial, and temporal conditions, building on the temporal modeling ability. The major difference between two stages lies in this incorporation of additional conditions, leading to a comprehensive learning of synthesizing video content from multiple compositions.\n\n\nWe have further detailed the explanation of this concept in the revised version.\n\n**Q3: Justifications for \"repeat the spatial conditions of a single image and sketch\".**\n\nThanks for pointing out this observation. We greatly value your suggestion to improve the efficiency, and acknowledge that simply repeating the latent feature can be an alternative. Additionally, we also think it to be plausible to utilize the temporal transformer:\n- **Unified interface**: We design the STC-encoder as a unified interface to incorporate different conditions (*i.e.*, single and sequential inputs) without re-designing the architecture, which can equip VideoComposer with better expansibility. Thus, repeating the input can adhere to this spirit well.\n- **Minimal computational cost**: The computational cost of the temporal transformer is negligible compared to the 3D UNet. Specifically, the computational overhead for inferring one STC-encoder is just 0.042\\% of that required by the 3D UNet. \n\n\n\n\n**Q4: Question about the utilization of motion vectors and the resultant blurry background in Figure 4.**\n\nThank you for the careful questions. This question touches on two aspects:\n\n\n- Treatment of Motion Vectors. In Figure 4, we treat the motion vector field equally for all locations. However, we offer flexibility during inference by using partial motion vectors.\n- Background Blurriness in Figure 4. We attribute it to the motion magnitude difference between two examples. The first video in the tiger example of Figure 4 contains minimal magnitude of motion, thus generating clear background. In comparison, the second video in the tiger example of Figure 4 (*i.e.*, the example in Figure R1(b)) utilizes motion vectors displaying large magnitude of motion, thereby easily resulting in blurriness. If we constrain the motion to a small magnitude as shown in Figure R1(a), we obtain clearer background.\n\n\n\n\n**Q5: More ablation studies of combining more conditions should be provided in Table 2.**\n\nThanks for raising this valuable concern.\n\n- **The purpose of Table 2**: As the reviewer mentioned, multiple conditions can be utilized together to guide the generated videos. However, we want to clarify that the aim of Table 2 is to demonstrate the usefulness of the STC-encoder in enhancing the temporal awareness of input conditions. By comparing results with and without STC-encoder with identical input conditions, we can highlight such effectiveness, using the metric of frame consistency. We do not expect that adding more conditions will consistently improve this metric.\n- **More comprehensive study**: Recognizing the validity of your suggestion, we have expanded Table 2 to encompass more conditions. Additional results presented in Table R1 demonstrate how the STC-encoder functions effectively with various combinations of conditions, confirming its versatility and applicability.\n\n**Table R1**: Quantitative ablation study of STC-encoder. \"Conditions\" denotes the conditions utilized for generation. \n\n\n\n|Methods|Conditions|Frame consistency|\n|:-|:-:|:-:|\n|*w/o* STC-encoder / VideoComposer|Text, sketch sequence, and depth sequence|0.911 / **0.918**|\n|*w/o* STC-encoder / VideoComposer|Text, sketch sequence, and motion vectors|0.912 / **0.919**|\n|*w/o* STC-encoder / VideoComposer|Text, depth sequence, and motion vectors|0.916 / **0.923**|\n|*w/o* STC-encoder / VideoComposer|Text, sketch sequence, depth sequence, and motion vectors|0.914 / **0.920**|\n\n**Q6: Color inconsistency of the tiger in video translation demo (2:11-2:16).**\n\n\nThank you for the careful observations. The problem of color consistency is a long-standing challenge in video generation. In this particular example, we conjecture that this stems from **the lack of an explicit textual condition**: The web-scale pre-training data contains both white and brown tigers. The coarse description applied doesn't specifically differentiate between them. We also provide **one possible solution** to address this inconsistency: providing more specified textual instructions, particularly regarding the color of the tiger. By regenerating the video using an identical random seed, as shown in Figure R2 (below), we achieve more satisfactory color consistency in the resultant video.\n\n\n\n**Q7: Moving results in Tab. A1 to the main paper.**\n\nThanks for the valuable suggestion. We will move it to the main paper to ensure a comprehensive comparison.\n"}, {"review_id": "EE8YOrMUaT", "reviewer": "Reviewer_BaoX", "summary": "VideoComposer is a tool designed to enhance video synthesis by incorporating textual, spatial, and temporal conditions. It uses motion vectors from compressed videos to guide temporal dynamics and employs a Spatio-Temporal Condition encoder to effectively integrate spatial and temporal relations of inputs. This improves inter-frame consistency and allows for greater control over the synthesized video's spatial and temporal patterns.\n", "questions": "How are image-text pairs utilized in the training process of the model?", "limitations": "What are the limitations?", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The VideoComposer offers better control over video synthesis, temporal guidance using motion vectors, improved inter-frame consistency with its Spatio-Temporal Condition encoder, versatility in accepting various forms of inputs, and high customizability, resulting in more precise and desired synthesized videos.", "weaknesses": "An ablation study could be conducted on VideoComposer, where each component is removed in turn to evaluate its impact on overall performance. This would help evaluate the value of training under multiple conditions versus a single condition. \n\nAdditionally, comparing VideoComposer to a simpler method like Text2Video-Zero [a] with ControlNet [b] would demonstrate whether the increased complexity of VideoComposer yields significantly better results, hence justifying its sophistication. \n\n[a] Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators, L Khachatryan et al.\n[b] Adding Conditional Control to Text-to-Image Diffusion Models, L. Zhang et al. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Q1: An ablation study could be conducted on VideoComposer, where each component is removed in turn to evaluate its impact on overall performance. This would help evaluate the value of training under multiple conditions versus a single condition.**\n\nThanks for your suggestion. We want to clarify that adding conditions will not necessarily improve the performance, and we do not claim this as a motivation for VideoComposer. Rather, our objective is to enhance the controllability by decomposing videos into various conditions. This approach allows VideoComposer to execute multiple tasks with a single model after just one-time training. Recognizing the validity of the reviewer's suggestion, we aim to validate whether adding more conditions will augment the controllability. Specifically, we seek to better reconstruct the original video by incrementally introducing the textual condition, depth map condition, and single image condition. As illustrated in Figure R8 of the attached one-page PDF file, we observe an improved alignment between the reference video (top row) and the generated videos. If the reviewer raises further concerns, we are more willing to address them and clarify any ambiguities.\n\n**Q2: Additionally, comparing VideoComposer to a simpler method like Text2Video-Zero [a] with ControlNet [b] would demonstrate whether the increased complexity of VideoComposer yields significantly better results, hence justifying its sophistication.**\n\nThanks for the valuable suggestion. To address this concern, we provide examples to demonstrate the superiority of the depth map-conditioned generation ability of VideoComposer. In Figure R4, we compare our VideoComposer with Text2Video-Zero and existing state-of-the-art Gen-1. We observed that Text2Video-Zero suffers from appearance inconsistency and structural flickering due to the lack of temporal awareness. Meanwhile, Gen-1 produces a video with color inconsistency and structure misalignment (revealed by the orientation of the bird head). The video generated by VideoComposer is faithful to the structure of the input depth sequence and maintains a continuous appearance. This shows the superiority of our VideoComposer in terms of controllability.\n\n**Q3: How are image-text pairs utilized in the training process of the model?**\n\nThanks. We train VideoComposer jointly on video-text and image-text pairs by treating images as 'one-frame' videos. When using image-text pairs for training, the shape of the input noise is $1 \\times h \\times w \\times c$, where the temporal length is 1 (*i.e.*, $F=1$).\n\n**Q4: What are the limitations?**\n\nDue to the page constraints, we include the discussion of limitations in the Sec. C of the Supplementary Material. In brief, the limitations include **(i)** the use of a watermarked pre-training dataset that leads to visually unappealing videos and **(ii)** the resultant low-resolution videos due to computational constraints.\n"}, {"review_id": "PN1L9oGNfC", "reviewer": "Reviewer_EXcw", "summary": "This work proposes a new method called VideoComposer for conditional video generation, especially for video-to-video translation. VideoComposer is constructed upon the Video Latent Diffusion Model and introduces an STC-encoder to integrate multiple spatial and temporal conditions such as RGB images, sketches, motion vector sequences, etc. The architecture design involves simple 2D convolutions and temporal transformer layers. The conditional features are fed into the U-Net input together with noise. The demonstrated results have good temporal consistency.", "questions": "The comparisons with other methods are not sufficient and the ablation study is not well presented.", "limitations": "The societal impact has been discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is one of the pioneering works in controllable video synthesis. The temporal consistency of the video results is impressive, considering that its conditioning modeling enables several editing abilities such as image-to-video translation and motion/depth/sketch-driven local/global video editing.\n\n- The jointly training strategy is good for flexible inference within one model, e.g., video inpainting, without second training.\n\n- The paper organization and illustrations are easy to follow.", "weaknesses": "- The authors could have tried other design choices for integrating Condition Fusion as input into the U-Net, such as integration through cross-attention.\n\n- In line 215, it is claimed that “we observe that the inclusion of mask and style guidance can facilitate structure and style control.” However, the corresponding evidence should be presented for the style representation extracted by clip image encoder and concatenated with text embedding.\n\n- It seems that a single STC-encoder is used for all different conditions via random dropout. It would be interesting to see if different STC-encoder weights for different conditions are better.\n\n- The examples in Figure 6 with reference image look like failure cases. Besides, the tiger texture and box shape are changed in Figure 8. It would be helpful to see more discussion and analysis on this part.\n\n- The ablation study of STC-encoder is not presented in a fair way. The main benefit of using STC-encoder comes from the video information condition instead of the network design.\n\n- The important comparisons and discussions with other methods are not sufficient, such as VideoP2P and vid2vid-zero mentioned in the related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Q1: Integrating the Condition Fusion into U-Net through cross-attention.**\n\nThe suggestion to use cross-attention for Condition Fusion in the U-Net is appreciated. While our current choice might not potentially be the optimal one, improving the micro-design of conditioning is beyond the scope of this work and requires considerable computational overhead associated with pre-training on LAION and WebVid. Therefore, we finalize this design choice more from the perspective of empirical analysis.\n\nIn VideoComposer, regarding the injection of global conditions, such as textual condition and style condition, we opt for cross-attention mechanisms following Stable Diffusion as it contains the high-level and abstractive information. Due to the quadratic complexity of cross-attention mechanisms, implementing cross-attention for Condition Fusion (which is not pooled in the spatial dimension) introduces considerable computational overhead, especially when compared against the relatively efficient concatenation approach.\n\n**Q2: Evidence should be provided to prove the statement in Line 215 that \"mask and style guidance can facilitate structure and style control\".**\n\nWe appreciate the reviewer's observation. Our description in line 215 might have been ambiguous. To align with this statement, we add the corresponding result in Figure R3 by adding the style condition. The generated video adheres to the given text, style and mask sequence. We will revise the confused descriptions in the next version. Feel free to raise questions if we misunderstand the question.\n\n**Q3: Is the STC-encoder shared across all conditions? Would the STC-encoder perform better with distinct weights?**\n\n Thanks. We use separate STC-encoders for different conditions without weight sharing by default. In Figure R5, we compare the results using STC-encoders *w/* and *w/o* weight sharing on video inpainting, and find that the former causes performance degradation for conditions like mask sequence. We attribute this to the uniqueness of different conditions, and sharing weights may lead to modeling difficulties. We will further improve the description of the STC-encoder in the revision. \n\n**Q4: More analysis should be provided to clarify: (i) the examples in Figure 6 with reference image look like failure cases; (2) the tiger texture and box shape are changed in Figure 8.**\n\n We appreciate your observation about Figure 6 and Figure 8.\n\nFirstly, We want to clarify that the role of the reference image is primarily as a global condition, offering stylistic features such as color and some aspects of content. As such, the generated videos tend to resemble the reference in terms of color and certain content attributes. According to our observation, we think the examples in Figure 6 exhibit such properties, thereby serving as successful cases.\n\nWe acknowledge that in Figure 8, the tiger texture and box shapes are compromised as the video progresses. This stems from a lack of structural control since we only provide the motion condition and the textual condition. To enable temporally consistent generation, we augment the motion-controlled video generation with a simple structure control by adding an auxiliary single-sketch condition, as shown in Figure R6. This addition has substantially improved the shape consistency in the generated video, rendering results that outperform those presented in the main paper.\n\n\n**Q5: The ablation study of STC-encoder is not presented in a fair way. The main benefit of using STC-encoder comes from the video information condition instead of the network design.**\n\nThank you for bringing our concern.\n\nWe might fail to precisely communicate the setting of this ablation study. To clarify, in our ablation study of STC-encoder, the baseline method (*w/o* STC-encoder) entails removing the temporal Transformer in STC-encoder while retaining the spatial convolution, designed to verify the effectiveness of incorporating temporal modeling. The spatial convolution remains in place to ensure the dimensions of all input conditions are consistent.\n\nUnder such a setting, VideoComposer and the baseline method both take advantage of the informative video conditions as input and, therefore, are compared in a fair way. As indicated in Line 242, such an ablation study underscores the significance of the STC-encoder's temporal modeling capacity during the input phase.\n\nIn light of your feedback, we will supplement the details of the baseline method in the revised version to avoid confusion.\n\n**Q6: Comparisons and discussions with methods, such as VideoP2P and vid2vid-zero, are not sufficient.**\n\n Thank you for raising this point. VideoP2P and vid2vid-zero are customized for video editing, which require the access to the original reference video (*i.e.*, the video to be edited) in order to finely optimize the video editing model for every video to be manipulated. However, our VideoComposer can circumvent this time-consuming re-training, and video generation can be performed given video conditions without accessing the original video.\n\nFor a more illustrative comparison with VideoP2P and vid2vid-zero, we present results in Figure R7. In this example, our objective is to transform the dog in the video to a tiger using an updated textual condition. Even though VideoP2P and vid2vid-zero need to specially optimize the model with the reference video, they still have difficulty maintaining structural consistency due to the lack of sequential structure guidance. In contrast, the video edited by VideoComposer can retain the structural alignment with the reference video while ensuring temporal continuity.\n"}, {"review_id": "fLLAD8aO69", "reviewer": "Reviewer_Pp5z", "summary": "This work aims to allows users to flexibly compose a video with textual conditions, spatial conditions, and temporal conditions.\nIt introduces a novel framework namely VideoComposer based on the paradigm of compositional generation.\nTo be specific, it introduces the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics.\nMoreover, it develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. \nExtensive experiments demonstrate that VideoComposer control the spatial and temporal patterns simultaneously within a synthesized video in various forms.", "questions": "See weakness.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It introduces motion vector as a more flexible user-guided signal.\n2. It proposes Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs.\n3. Extensive experiments show the effectiveness and superiority of VideoComposer.", "weaknesses": "1. What is the difference between the roles of the ``Style`` of CLIP and ``Single Image`` of STC-encoder? They both seem to provide content to videos.\n2. VideoComposer only obtain comparable performance with prior video generative models. Is it more efficient than previous methods? The authors could give their comparisons in training cost and inference time.\n3. Lack of extensive visualization comparisons with existing video generative models. The authors are encouraged to provide extensive qualitative comparisons in video generation task.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Q1: What is the difference between the roles of the Style of CLIP and Single Image of STC-encoder? They both seem to provide content to videos.**\n\n Thank you for highlighting this point.\n- **Style condition**.  The style condition mainly encapsulates the holistic characteristics of the input image, capturing elements like its color, style, and content.  Analogous to the textual condition, it functions as a **global** descriptor, providing a generalized perspective of the image. When utilizing an image as this global condition, spatial pooling is employed to derive a singular embedding.  This spatial reduction, however, causes the loss of detailed structural information, which can lead to generalized content interpretation. Thus, it is possible that the generated video can have varying poses (if containing humans or animals) and slight color shifts compared to the reference image.\n- **Single image condition**. The primary role of the single image condition is to serve as the initial frame for the video being generated.  As such, it conveys **local** details and intricacies of the image, ensuring the video's initiation is aligned pixel-for-pixel with the provided image.\n\nIn conclusion, both conditions influence the video's content and color, but the depth and granularity of influence differ.\n\n\n\n**Q2: VideoComposer only obtain comparable performance with prior video generative models. Is it more efficient than previous methods? The authors could give their comparisons in training cost and inference time.**\n\nThank you for raising this point. We want to clarify that the primary objective of VideoComposer is to enhance controllability and applicability, rather than to reduce training cost and inference time. Recognizing the value of the reviewer's concern, we also provide a brief discussion in terms of **efficiency**.\n- **Controllability and versatility**. One of the primary advantages of VideoComposer over previous methods is its augmented controllability. Both Make-A-Video and Video LDM primarily support video generation from text descriptions, limiting their capacity for customized, controllable video generation. In contrast, VideoComposer benefits from diverse conditional guidances, facilitating video generation utilizing user-defined textual, spatial, and temporal conditions. This versatility paves a path to truly compositional video generation tailored to specific requirements.\n- **Broad applicability**. VideoComposer's design isn't just about controllability, but also about applicability.  Our model's capability to address multiple video generation tasks without the need for repeated re-training underscores its unique value. We have equipped one model with a multitude of application scenarios, marking a significant superiority to existing works.\n- **Efficiency**. While we can't provide direct numerical comparisons due to the unavailability of source codes from competitive methods like Make-A-Video and Video LDM, we'd like to highlight certain design features that contribute to VideoComposer's efficiency. Specifically, unlike Make-A-Video, which leverages multiple cascaded models, VideoComposer adopts a unified approach, possibly reducing both the training cost and inference time. We anticipate that this design could allow VideoComposer to achieve comparable efficiency to existing methods such as Video LDM since both base on Stable Diffusion and extend it by adding additional temporal layers.\n\nIn summary, while we acknowledge the request for direct comparisons in efficiency, we'd like to emphasize VideoComposer's controllability and versatility.  These features make our model a crucial contribution to the current landscape of video synthesis.\n\n**Q3: Lack of extensive visualization comparisons with existing video generative models. The authors are encouraged to provide extensive qualitative comparisons in video generation task.**\n\nThanks for the suggestion. We show more qualitative comparisons with existing Text2Video-Zero and Gen-1 in Figure R4. We observed that Text2Video-Zero suffers from appearance inconsistency and structural flickering due to the lack of temporal awareness. Meanwhile, Gen-1 produces a video with color inconsistency and structure misalignment (revealed by the orientation of the bird head). The video generated by VideoComposer is faithful to the structure of the input depth sequence and maintains a continuous appearance. The above experiments demonstrate the superiority of our method in terms of controllability.\n"}]

Author rebuttal:
We appreciate the reviewers for their positive comments and constructive feedback on our paper. We are encouraged that VideoComposer is recognized for its merits, including

- clarity in presentation [Reviewer akz1, EXcw]
- superior performance both quantitatively and qualitatively over prior methods [Reviewer akz1, Pp5z]
- introduction of innovative motion vectors [Reviewer Pp5z, BaoX]
- a novel interface for handling inputs [Reviewer Pp5z, BaoX]
- its heightened controllability and customizability [Reviewer akz1, BaoX, EXcw].

We also acknowledge the concerns raised and will address them comprehensively. These insightful reviews lead to multiple improvements of our original manuscript.

In this rebuttal, we have included some new figures in **the newly uploaded one-page PDF**. Here, we briefly summarize the content of these figures to facilitate quick reference.

- Figure R1 shows the effect of varying motion magnitude by sampling at different frame rates of the motion vector condition. [Reviewer akz1]
- Figure R2 gives the video translation results, suggesting that we can leverage a detailed prompt to generate a more desired and color-consistent video. [Reviewer akz1]
- Figure R3 gives the compositional video generation results, where the video is generated using text, image and mask sequence as conditions. [Reviewer EXcw]
- Figure R4 compares VideoComposer with other existing methods such as Text2Video-Zero and Gen-1. [Reviewer BaoX, Pp5z] 
- Figure R5 shows the comparison of VideoComposer using STC-encoder with and without weight sharing on video inpainting task. [Reviewer EXcw]
- Figure R6 includes the experiment of motion control with single sketch condition. [Reviewer EXcw]
- Figure R7 compares VideoComposer with VideoP2P and vid2vid-zero.  [Reviewer EXcw]
- Figure R8 shows the comparison of video generation with single and multiple conditions. [Reviewer BaoX]


We are open to discussions and are committed to addressing any concerns from the reviewers, ensuring the continual refinement of VideoComposer.

Meta-review: This paper receives consistent review scores of 5/6/5/6.

The paper presents VideoComposer, a model which allows users to flexibly compose a video with textual, spatial, and temporal conditions. Both qualitative and quantitative results show that the proposed method can synthesize controllable high-quality videos.

All reviewers appreciate the contributions of the paper in controllable video synthesis. Almost all concerns they raised in the first round of review have been addressed in the rebuttal.

The authors are suggested to include the additional ablation studies and clarifications they provided during rebuttal in the final version.
Decision: Accept (poster)