SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Frechet Video Distance (FVD) and CLIP Similarity (CLIPSIM) as evaluation metrics and present the quantitative results of text-to-video generation on MSR-VTT dataset compared to other existing methods. The results in the table demonstrate that VideoComposer achieves competitive performance compared to state-of-the-art text-to-video approaches. In addition, VideoComposer outperforms our first-stage text-to-video pre-training, demonstrating that VideoComposer can achieve compositional generation without sacrificing its capability of text-to-video generation. In the future, we aim to advance VideoComposer by leveraging stronger text-to-video models for more powerful synthesis.
4.4 Experimental results of motion control
Quantitative evaluation. To validate superior motion controllability, we utilize the motion control metric. We randomly select 1000 caption-video pairs and synthesize corresponding videos. The results are presented in Tab. 2. We observe that the inclusion of motion vectors as a condition reduces the motion control error, indicating an enhancement of motion controllability. The incorporation of STC-encoder further advances the motion controllability.
Figure 7: Comparison with methods for controllable video generation. Results are generated utilizing the textual condition and the depth sequence in the first row. Text2Video-Zero suffers from appearance inconsistency and structural flickering due to the lack of temporal awareness. Gen-1 produces a video with color inconsistency and structural misalignment. The video generated by VideoComposer is faithful to the structure of the input depth sequence and maintains a continuous appearance.
"American Staffordshire Terrier is lying on the floor in an abandoned building"
(a) Source video
(b) Depth-guided generation
(c) Sketch-guided generation
(d) Motion-guided generation
Figure 8: Video-to-video translation. We extract a sequence of depth maps, sketches or motion vectors from the source video, along with textual descriptions, to perform the translation. By utilizing motion vectors, we achieve static-background removal.
Motion vectors prioritizing moving visual cues. Thanks to the nature of motion vectors, which encode inter-frame variation, static regions within an image are inherently omitted. This prioritization of moving regions facilitates motion control during synthesis. In Fig. 8, we present results of video-to-video translation to substantiate such superiority.
We observe that motion vectors exclude the static
background, i.e., human legs, a feat that other temporal conditions such as depth maps and sketches cannot accomplish. This advantage lays the foundation for a broader range of applications.
Table 2: Evaluating the motion controllability. "Text" and "MV" represent the utilization of text and motion vectors as conditions for generation.
Method	Text	MV	Motion control ↓
w/o STC-encoder	✓	4.03
w/o STC-encoder	✓	✓	2.67
VideoComposer	✓	✓	2.18
Versatile motion control with motion vectors. Motion vectors, easily derived from hand-crafted strokes, enable more versatile motion control. In Fig. 9, we present visualization comparing CogVideo [26] and VideoComposer. While CogVideo is limited to insufficient text-guided motion control, VideoComposer expands this functionality by additionally leveraging motion vectors derived from hand-crafted strokes to facilitate more flexible and precise motion control.
In this subsection, we conduct qualitative and quantitative analysis on VideoComposer, aiming to demonstrate the effectiveness of incorporating STC-encoder.
Quantitative analysis. In Tab. 3, we present the frame consistency metric computed on 1000 test videos. We observe that incorporating STC-encoder augments the frame consistency, which we
"A box moves from left to right"
"A moving box"
(a)CogVideo
(b) VideoComposer: text and hand-crafted motions
Figure 9: Versatile motion control using hand-crafted motions. (a) Limited motion control using CogVideo [26]. (b) Fine-grained and flexible motion control, empowered by VideoComposer.
"A tiger walking on the grassland"
Figure 10: Qualitative ablation study. We present three representative examples. The last two rows of videos display generated videos conditioned on a textual condition and one additional temporal condition (i.e., sketches, depth maps or motion vectors). Regions exhibiting deficiencies or fidelity are emphasized within red boxes.
attribute to its temporal modeling capacity. This observation holds for various temporal conditions such as sketches, depth maps and motion vectors.
Qualitative analysis. In Fig. 10, we exemplify the usefulness of STC-encoder. We observe that in the first example, videos generated by VideoComposer without STC-encoder generally adhere to the sketches but omit certain detailed information, such as several round-shaped ingredients. For the left two examples, VideoComposer without STC-encoder generates videos that are structurally inconsistent with conditions. We can also spot the noticeable defects in terms of human faces
Table 3: Quantitative ablation study of STC-encoder. "Conditions" denotes the conditions utilized for generation.
Method	Conditions	Frame consistency ↑
w/o STC-encoder	Text and sketch sequence	0.910
VideoComposer	0.923
w/o STC-encoder	Text and depth sequence	0.922
VideoComposer	0.928
w/o STC-encoder	Text and motion vectors	0.915
VideoComposer	0.927
and poses. Thus, all the above examples can validate the effectiveness of STC-encoder.
In this paper, we present VideoComposer, which aims to explore the compositionality within the realm of video synthesis, striving to obtain a flexible and controllable synthesis system. In particular, we explore the use of temporal conditions for videos, specifically motion vectors, as powerful control signals to provide guidance in terms of temporal dynamics. An STC-encoder is further designed as a unified interface to aggregate the spatial and temporal dependencies of the sequential inputs for inter-frame consistency. Our experiments, which involve the combination of various conditions to augment controllability, underscore the pivotal role of our design choices and reveal the impressive creativity of the proposed VideoComposer.
Figure A11: Compositional sketch sequence-to-video generation. We showcase five examples, each displaying a video generated from a sequence of sketches and a textual description. The final example additionally incorporates a style condition.
In this Appendix, we first elaborate on more implementation details (Appendix A) and present more experimental results (Appendix B). Next, we provide a section of discussion (Appendix C) on the limitations and potential societal impact of VideoComposer.
A More implementation details
Pre-training details. We adopt AdamW [35] as the default optimizer with a learning rate set to  $5 \times 10^{-5}$ . In total, VideoComposer is pre-trained for 400k steps, with the first and second stage being pre-trained for 132k steps and 268k steps, respectively. In terms of two-stage pre-training, we allocate one fourth of GPUs to perform image pre-training, while the rest of the GPUs are dedicated to video pre-training. We use center crop and randomly sample video frames to compose the video input whose  $F = 16$ ,  $H = 256$  and  $W = 256$ . During the second stage pre-training, we adhere to [28], using a probability of 0.1 to keep all conditions, a probability of 0.1 to discard all conditions, and an independent probability of 0.5 to keep or discard a specific condition. Regarding the use of
WebVid10M [2], we sample frames from videos using various strides to ensure frame rate equal to 4, aiming to maintain a consistent frame rate. We train VideoComposer jointly on video-text and image-text pairs by treating images as 'one-frame' videos. When using image-text pairs for training, the shape of the input noise is  $1 \times h \times w \times c$ , where the temporal length is 1 (i.e.,  $F = 1$ ). In experiments, we use separate STC-encoders for different conditions without weight sharing. In our ablation study of STC-encoder, the baseline method ( $w / o$  STC-encoder) entails removing the temporal Transformer in STC-encoder while retaining the spatial convolution, designed to verify the effectiveness of incorporating temporal modeling. The spatial convolution remains in place to ensure the dimensions of all input conditions are consistent.
The structure of 3D UNet as  $\epsilon_{\theta}(\cdot ,\cdot ,t)$ . To leverage the benefits of LDMs pre-trained on web-scale image data, i.e., Stable Diffusion<sup>2</sup>, we extend the 2D UNet to a 3D UNet by introducing temporal modeling layers. Specifically, within a single UNet block, we employ four essential building blocks: spatial convolution, temporal convolution, spatial transformer and temporal transformer. The spatial blocks are inherited from LDMs, while temporal processing blocks are newly introduced. Regarding temporal convolution, we stack four convolutions with  $1\times 1\times 3$  kernel, ensuring the temporal receptive field is ample for capturing temporal dependencies; regarding temporal transformer, we stack one Transformer layer and accelerate its inference using flash attention [12].
More details about two-stage training strategy. The two-stage training strategy utilized in VideoComposer is designed to methodologically address the learning challenge. In the first stage, VideoComposer focuses on learning the temporal dynamics by only leveraging the textual condition. This foundation allows for a focused understanding of temporal relationships within the video content. In the second stage, VideoComposer builds on the temporal modeling ability acquired from the first stage to perform compositional training. In this stage, it extends its learning by utilizing all three kinds of conditions: textual, spatial, and temporal conditions. The major difference between the two stages lies in this incorporation of additional conditions, leading to a comprehensive learning of synthesizing video content from multiple compositions.
B More experimental results
In this subsection, we aim to provide additional experiments that complement the findings presented in the main paper and showcase more versatile controlling cases.
Compositional sketch sequence-to-video generation. Compositional training with sketch sequences enables VideoComposer to possess the ability of generation videos adhering to sketch sequences. This generation paradigm lays more emphasis on the structure control, which differs from compositional sketch-to-video generation and can be viewed as video-to-video translation. In Fig. A11, we exemplify this capacity. We observe videos' fidelity to the provided conditions, including texts, sketches and style.
Compositional depth sequence-to-video generation. Conducting compositional training with depth sequences allows VideoComposer to effectively generate videos in accordance with depth sequences. In Fig. A12, we illustrate this capability. Videos generated with VideoComposer faithfully adhere to the given conditions, including text prompts, depth maps, and style.
Motion transfer. Incorporating motion vectors as a composition of videos enables motion transferability. In Fig. A13, we conduct experiments to demonstrate such capability. Through utilizing hand-crafted motion vectors or motion vectors extracted from off-the-shelf source videos, we can transfer the motion patterns to synthesized videos.
Limitations. Due to the absence of a publicly available large-scale and high-quality dataset, we have developed VideoComposer using the watermarked WebVid10M dataset. As a result, the synthesized videos contain watermarks, which affect the generation quality and lead to less visually appealing results. Furthermore, in order to reduce the training cost, the resolution of the generated videos is limited to  $256 \times 256$ . Consequently, some delicate details might not be sufficiently clear. In the future,
$^{2}$ https://github.com/Stability-AI/stablediffusion
Figure A12: Compositional depth sequence-to-video generation. We showcase five examples, each displaying a video generated from a sequence of depth maps and a textual description. The final example additionally incorporates a style condition.
we plan to utilize super-resolution models to expand the resolution of the generated videos to improve the visual quality.
Potential societal impact. VideoComposer, as a generic video synthesis technology, possesses the potential to revolutionize the content creation industry, offering unprecedented flexibility and creativity, and hence, promising significant commercial advantages. Traditional content creation processes are labor- and cost-intensive. VideoComposer could alleviate these burdens by enabling designers to manipulate subjects, styles, and scenes through instructions spanning human-written text, and styles and subjects sourced from other images. Moreover, VideoComposer could potentially revolutionize education industry by creating unique and customized video scenarios for teaching complex concepts.
However, it's necessary to note that VideoComposer also represents a dual-use technology with inherent risks to society. As with prior generative foundation models, such asImagen Video [24] and Make-A-Video [53], VideoComposer inherits the implicit knowledge embedded within the pre-trained model (i.e., StableDiffusion) and the pre-trained dataset (i.e., WebVid and LAION). Potential issues include but not limited to the propagation of social biases (such as gender and racial bias) and the creation of offensive content.
Figure A13: Motion transfer. We showcase four examples, each displaying a video generated from a single image and motions. In the first three examples, we transfer the motion patterns in a source video to the generated video by extracting and utilizing motion vectors. The final example incorporates hand-crafted motions instead.
Given that VideoComposer is a research-oriented project aimed at investigating compositionality in diffusion-based video synthesis, our primary focus lies in scientific exploration and proof of concept. If VideoComposer is deployed beyond the scope of research, we strongly recommend several precautionary measures to ensure its responsible and ethical use: (i) Rigorous evaluation and oversight of the deployment context should be conducted; (ii) Necessary filtering of prompts and generated content should be implemented to prevent misuse.