SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiang Wang $^{1*}$  Hangjie Yuan $^{1*}$  Shiwei Zhang $^{1*}$  Dayou Chen $^{1*}$  Jiuniu Wang $^{1}$  Yingya Zhang $^{1}$  Yujun Shen $^{2}$  Deli Zhao $^{1}$  Jingren Zhou $^{1}$
$^{1}$ Alibaba Group  $^{2}$ Ant Group {xiaolao.wx, yuanhangjie.yhj, zhangjin.zsw}@alibaba-inc.com {dayou.cdy, wangjiuniu.wjn, yingya.zyy, jingren.zhou}@alibaba-inc.com {shenyujun0302, zhaodeli}@gmail.com
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models are publicly available at https://videocomposer.github.io.
Driven by the advances in computation, data scaling and architectural design, current visual generative models, especially diffusion-based models, have made remarkable strides in automating content creation, empowering designers to generate realistic images or videos from a textual prompt as input [24, 48, 53, 62]. These approaches typically train a powerful diffusion model [48] conditioned by text [23] on large-scale video-text and image-text datasets [2, 51], reaching unprecedented levels of fidelity and diversity. However, despite this impressive progress, a significant challenge remains in the limited controllability of the synthesis system, which impedes its practical applications.
Most existing methods typically achieve controllable generation mainly by introducing new conditions, such as segmentation maps [48, 65], inpainting masks [72] or sketches [38, 79], in addition to texts. Expanding upon this idea,Composer [28] proposes a new generative paradigm centered on the concept of compositionality, which is capable of composing an image with various input conditions, leading to remarkable flexibility. However,Composer primarily focuses on considering
"Rotation view of a beautiful long-haired woman standing in the forest"
(a) Text and sketch
(b) Image and depth
(c) Style, depth, sketch and motions
(d) Text, style, hand-crafted motions and hand-crafted sketch
Figure 1: Compositional video synthesis. (a-c) VideoComposer is capable of generating videos that adhere to textual, spatial and temporal conditions or their subsets; (d) VideoComposer can synthesize videos conforming to expected motion patterns (red stroke) and shape patterns (white stroke) derived from two simple strokes.
multi-level conditions within the spatial dimension, hence it may encounter difficulties when comes to video generation due to the inherent properties of video data. This challenge arises from the complex temporal structure of videos, which exhibits a large variation of temporal dynamics while simultaneously maintaining temporal continuity among different frames. Therefore, incorporating suitable temporal conditions with spatial clues to facilitate controllable video synthesis becomes significantly essential.
Above observations motivate the proposed VideoComposer, which equips video synthesis with improved controllability in both spatial and temporal perception. For this purpose, we decompose a video into three kinds of representative factors, i.e., textual condition, spatial conditions and the crucial temporal conditions, and then train a latent diffusion model to recompose the input video conditioned by them. In particular, we introduce the video-specific motion vector as a kind of temporal guidance during video synthesis to explicitly capture the inter-frame dynamics, thereby providing direct control over the internal motions. To ensure temporal consistency, we additionally present a unified STC-encoder that captures the spatio-temporal relations within sequential input utilizing cross-frame attention mechanisms, leading to an enhanced cross-frame consistency of the output videos. Moreover, STC-encoder serves as an interface that allows for efficient and unified utilization of the control signals from various condition sequences. As a result, VideoComposer is
capable of flexibly composing a video with diverse conditions while simultaneously maintaining the synthesis quality, as shown in Fig. 1. Notably, we can even control the motion patterns with simple hand-crafted motions, such as an arrow indicating the moon's trajectory in Fig. 1d, a feat that is nearly impossible with current methods. Finally, we demonstrate the efficacy of VideoComposer through extensive qualitative and quantitative results, and achieve exceptional creativity in the various downstream generative tasks.
Image synthesis with diffusion models. Recently, research efforts on image synthesis have shifted from utilizing GANs [18], VAEs [31], and flow models [14] to diffusion models [9, 19, 23, 32, 34, 54, 59, 74, 80, 81] due to more stable training, enhanced sample quality, and increased flexibility in a conditional generation. Regarding image generation, notable works such as DALL-E 2 [46] and GLIDE [40] employ diffusion models for text-to-image generation by conducting the diffusion process in pixel space, guided by CLIP [44] or classifier-free approaches. Imagen [50] introduces generic large language models, i.e., T5 [45], improving sample fidelity. The pioneering work LDMs [48] uses an autoencoder [15] to reduce pixel-level redundancy, making LDMs computationally efficient. Regarding image editing, pix2pix-zero [42] and prompt-to-prompt editing [21] follow instructional texts by manipulating cross-attention maps. Imagic [29] interpolates between an optimized embedding and the target embedding derived from text instructions to manipulate images. DiffEdit [11] introduces automatically generated masks to assist text-driven image editing. To enable conditional synthesis with flexible input, ControlNet [79] and T2I-Adapter [38] incorporate a specific spatial condition into the model, providing more fine-grained control. One milestone,Composer [28], trains a multi-condition diffusion model that broadly expands the control space and displays remarkable results. Nonetheless, this compositionality has not yet been proven effective in video synthesis, and VideoComposer aims to fill this gap.
Video synthesis with diffusion models. Previous methods [13, 58, 67] usually adopt GANs for video synthesis. Recent research has demonstrated the potential of employing diffusion models for high-quality video synthesis [5, 20, 25, 30, 36, 62, 75]. Notably,ImagenVideo [24] and Make-A-Video [53] both model the video distribution in pixel space, which limits their applicability due to high computational demands. In contrast, MagicVideo [83] models the video distribution in the latent space, following the paradigm of LDMs [48], significantly reducing computational overhead. With the goal of editing videos guided by texts, VideoP2P [33] and vid2vid-zero [66] manipulate the crossattention map, while Dreamix [37] proposes an image-video mixed fine-tuning strategy. However, their generation or editing processes solely rely on text-based instructions [44, 45]. A subsequent work, Gen-1 [16], integrates depth maps alongside texts using cross-attention mechanisms to provide structural guidance. Both MCDiff [8] and LaMD [27] target motion-guided video generation; the former focuses on generating human action videos and encodes the dynamics by tracking the keypoints and reference points, while the latter employs a learnable motion latent to improve quality. Nevertheless, incorporating the guidance from efficient motion vectors or incorporating multiple guiding conditions within a single model is seldom explored in the general video synthesis field.
Motion modeling. Motion cues play a crucial role in video understanding fields, such as action recognition [1, 4, 6, 43, 60, 63, 64], action detection [10, 68, 77, 82], human video generation [39, 41, 67], etc. Pioneering works [1, 6, 39, 43, 63, 67] usually leverage hand-crafted dense optical flow [76] to embed motion information or design various temporal structures to encode long-range temporal representations. Due to the high computational demands of optical flow extraction, several attempts in compressed video recognition [7, 52, 69, 78] have begun to utilize more efficient motion vectors as an alternative to represent motions and have shown promising performance. In contrast to these works, we delve into the role of motions in video synthesis and demonstrate that motion vectors can enhance temporal controllability through a well-designed architecture.
3 VideoComposer
In this section, we will comprehensively present VideoComposer to showcase how it can enhance the controllability of video synthesis and enable the creation of highly customized videos. Firstly, we in brief introduce Video Latent Diffusion Models (VLDMs) upon which VideoComposer is designed, given their impressive success in various generative tasks. Subsequently, we delve into the details of
Figure 2: Overall architecture of VideoComposer. First, a video is decomposed into three types of conditions, including textual condition, spatial conditions and temporal conditions. Then, we feed these conditions into the unified STC-encoder or the CLIP model to embed control signals. Finally, the resulting conditions are leveraged to jointly guide VLDMs for denoising.
VideoComposer's architecture, including the composable conditions and unified Spatio-Temporal Condition encoder (STC-encoder) as illustrated in Fig. 2. Finally, the concrete implementations, including the training and inference processes, will be analyzed.
Compared to images, processing video requires substantial computational resources. Intuitively, adapting image diffusion models that process in the pixel space [40, 46] to the video domain impedes the scaling of VideoComposer to web-scale data. Consequently, we adopt a variant of LDMs that operate in the latent space, where local fidelity could be maintained to preserve the visual manifold.
Perceptual video compression. To efficiently process video data, we follow LDMs by introducing a pre-trained encoder [15] to project a given video  $\pmb{x} \in \mathbb{R}^{F \times H \times W \times 3}$  into a latent representation  $\pmb{z} = \mathcal{E}(\pmb{x})$ , where  $\pmb{z} \in \mathbb{R}^{F \times h \times w \times c}$ . Subsequently, a decoder  $\mathcal{D}$  is adopted to map the latent representations back to the pixel space  $\bar{\pmb{x}} = \mathcal{D}(\pmb{z})$ . We set  $H / h = W / w = 8$  for rapid processing.
Diffusion models in the latent space. To learn the actual video distribution  $\mathbb{P}(x)$ , diffusion models [23, 54] learn to denoise a normally-distributed noise, aiming to recover realistic visual content. This process simulates the reverse process of a Markov Chain of length  $T$ .  $T$  is set to 1000 by default. To perform the reverse process on the latent, it injects noise to  $\pmb{z}$  to obtain a noise-corrupted latent  $\pmb{z}_t$  following [48]. Subsequently, we apply a denoising function  $\epsilon_{\theta}(\cdot ,\cdot ,t)$  on  $\pmb{z}_t$  and selected conditions  $\pmb{c}$ , where  $t\in \{1,\dots,T\}$ . The optimized objective can be formulated as:
$$
\mathcal {L} _ {V L D M} = \mathbb {E} _ {\mathcal {E} (x), \epsilon \in \mathcal {N} (0, 1), \boldsymbol {c}, t} \left[ \| \epsilon - \epsilon_ {\theta} (\boldsymbol {z} _ {t}, \boldsymbol {c}, t) \| _ {2} ^ {2} \right] \tag {1}
$$
To exploit the inductive bias of locality and temporal inductive bias of sequentiality during denoising, we instantiate  $\epsilon_{\theta}(\cdot ,\cdot ,t)$  as a 3D UNet augmented with temporal convolution and cross-attention mechanism following [25, 49, 62].
3.2 VideoComposer
Videos as composable conditions. We decompose videos into three distinct types of conditions, i.e., textual conditions, spatial conditions and crucially temporal conditions, which can jointly determine the spatial and temporal patterns in videos. Notably, VideoComposer is a generic compositional framework. Therefore, more customized conditions can be incorporated into VideoComposer depending on the downstream application and are not limited to the decompositions listed above.
Textual condition. Textual descriptions provide an intuitive indication of videos in terms of coarse-grained visual content and motions. In our implementation, we employ the widely used pre-trained text encoder from OpenCLIP ViT-H/14 to obtain semantic embeddings of text descriptions.
Spatial conditions. To achieve fine-grained spatial control and diverse stylization, we apply three spatial conditions to provide structural and stylistic guidance: i) Single image. Video is made up of consecutive images, and a single image usually reveals the content and structure of this video. We select the first frame of a given video as a spatial condition to perform image-to-video generation. ii) Single sketch. We extract sketch of the first video frame using PiDiNet [55] as the second spatial condition and encourage VideoComposer to synthesize temporal-consistent video according to the structure and texture within the single sketch. iii) Style. To further transfer the style from one image to the synthesized video, we choose the image embedding as the stylistic guidance, following [3, 28]. We apply a pre-trained image encoder from OpenCLIP ViT-H/14 to extract the stylistic representation.
Temporal conditions. To accomplish finer control along the temporal dimension, we introduce four temporal conditions: i) Motion vector. Motion vector as a videospecific element is represented as two-dimension vectors, i.e., horizontal and vertical orientations. It explicitly encodes the pixel-wise movements between two adjacent frames, as visualized by red arrows in Fig. 3. Due to the natural properties of motion vector, we treat this condition as a motion control signal for temporal-smooth synthesis. Following [52, 69], we extract motion vectors in standard
Figure 3: Examples of motion vectors.
MPEG-4 format from compressed videos. ii) Depth sequence. To introduce depth information, we utilize the pre-trained model from [47] to extract depth maps of video frames. iii) Mask sequence. To facilitate video regional editing and inpainting, we manually add masks. We introduce tube masks [17, 57] to mask out videos and enforce the model to predict the masked regions based on observable information. iv) Sketch sequence. Compared with the single sketch, sketch sequence can provide more control details and thus achieve precisely customized synthesis.
STC-encoder. Sequential conditions contain rich and complex space-time dependencies, posing challenges for controllable guidance. In order to enhance the temporal awareness of input conditions, we design a Spatio-Temporal Condition encoder (STC-encoder) to incorporate the space-time relations, as shown in Fig. 2. Specifically, a light-weight spatial architecture consisting of two 2D convolutions and an average pooling layer is first applied to the input sequences, aiming to extract local spatial information. Subsequently, the resulting condition sequence is fed into a temporal Transformer layer [61] for temporal modeling. In this way, STC-encoder facilitates the explicit embedding of temporal cues, allowing for a unified condition interface for diverse inputs, thereby enhancing inter-frame consistency. It is worth noting that we repeat the spatial conditions of a single image and single sketch along the temporal dimension to ensure their consistency with temporal conditions, hence facilitating the condition fusion process.
After processing the conditions by STC-encoder, the final condition sequences are all in an identical spatial shape to  $\boldsymbol{z}_t$  and then fused by element-wise addition. Finally, we concatenate the merged condition sequence with  $\boldsymbol{z}_t$  along the channel dimension as control signals. For textual and stylistic conditions organized as a sequence of embeddings, we utilize the cross-attention mechanism to inject textual and stylistic guidance.
Two-stage training strategy. Although VideoComposer can initialize with the pre-training of LDMs [48], which mitigates the training difficulty to some extent, the model still struggles in learning to simultaneously handle temporal dynamics and synthesize video content from multiple compositions. To address this issue, we leverage a two-stage training strategy to optimize VideoComposer. Specifically, the first stage targets pre-training the model to specialize in temporal modeling through text-to-video generation. In the second stage, we optimize VideoComposer to excel in video synthesis controlled by the diverse conditions through compositional training.
Inference. During inference, DDIM [80] is employed to enhance the sample quality and improve inference efficiency. We incorporate classifier-free guidance [22] to ensure that the generative results
Figure 4: Compositional image-to-video generation. We showcase six examples, each displaying two generated videos. The upper video is generated using a given single frame as the spatial condition and a textual condition describing the scene. The lower video is generated by incorporating an additional sequence of temporal conditions to facilitate finer control over the temporally evolving structure.
adhere to specified conditions. The generative process can be formalized as:
$$
\hat {\epsilon} _ {\theta} \left(\boldsymbol {z} _ {t}, \boldsymbol {c}, t\right) = \epsilon_ {\theta} \left(\boldsymbol {z} _ {t}, \boldsymbol {c} _ {1}, t\right) + \omega \left(\epsilon_ {\theta} \left(\boldsymbol {z} _ {t}, \boldsymbol {c} _ {2}, t\right) - \epsilon_ {\theta} \left(\boldsymbol {z} _ {t}, \boldsymbol {c} _ {1}, t\right)\right) \tag {2}
$$
where  $\omega$  is the guidance scale;  $c_{1}$  and  $c_{2}$  are two sets of conditions. This guidance mechanism extrapolates between two condition sets, placing emphasis on the elements in  $(c_{2} \setminus c_{1})$  and empowering flexible application. For instance, in text-driven video inpainting,  $c_{2}$  represents the expected caption and a masked video, while  $c_{1}$  is an empty caption and the same masked video.
Datasets. To optimize VideoComposer, we leverage two widely recognized and publicly accessible datasets: WebVid10M [2] and LAION-400M [51]. WebVid10M [2] is a large-scale benchmark scrapped from the web that contains 10.3M video-caption pairs. LAION-400M [51] is an image-caption paired dataset, filtered using CLIP [44].
Evaluation metrics. We utilize two metrics to evaluate VideoComposer:  $i$  ) To evaluate video continuity, we follow Gen-1 [16] to compute the average CLIP cosine similarity of two consecutive frames, serving as a frame consistency metric; ii) To evaluate motion controllability, we adopt end-point-error [56, 73] as a motion control metric, which measures the Euclidean distance between the predicted and the ground truth optical flow for each pixel.
4.2 Composable video generation with versatile conditions
In this section, we demonstrate the ability of VideoComposer to tackle various tasks in a controllable and versatile manner, leveraging its inherent compositionality. It's important to note that the conditions employed in these examples are customizable to specific requirements. We also provide additional results in the supplementary material for further reference.
Figure 5: Compositional video inpainting. By manually adding masks to videos, VideoComposer can perform video inpainting, facilitating the restoration of the corrupted parts according to textual instructions. Furthermore, by incorporating temporal conditions specifying the visual structure, VideoComposer can perform customized inpainting that conforms to the prescribed structure.
Compositional Image-to-video generation. Compositional training with a single image endows VideoComposer with the ability of animating static images. In Fig. 4, we present six examples to demonstrate this ability. VideoComposer is capable of synthesizing videos conformed to texts and the initial frame. To further obtain enhanced control over the structure, we can incorporate additional temporal conditions. We observe resultant videos consistently adhere to the given conditions.
**Compositional video inpainting.** Jointly training with masked video endows the model with the ability of filling the masked regions with prescribed content, as shown in Fig. 5. VideoComposer can replenish the mask-corrupted regions based on textual descriptions. By further incorporating temporal conditions, i.e., depth maps and sketches, we obtain more advanced control over the structure.
Compositional sketch-to-video generation. Compositional training with single sketch empowers VideoComposer with the ability of animating static sketches, as illustrated in Fig. 6. We observe that VideoComposer synthesizes videos conforming to texts and the initial sketch. Furthermore, we observe that the inclusion of mask and style guidance can facilitate structure and style control.
4.3 Comparative experimental results
Depth-to-video controllability comparison. In Fig. 7, we compare our VideoComposer with Text2Video-Zero [30] and existing state-of-the-art Gen-1 [16]. We observed that Text2Video-Zero suffers from appearance inconsistency and structural flickering due to the lack of temporal awareness. Meanwhile, Gen-1 produces a video with color inconsistency and structure misalignment (revealed by the orientation of the bird head). The video generated by VideoComposer is faithful to the structure of the input depth sequence and maintains a continuous appearance. This shows the superiority of our VideoComposer in terms of controllability.
Text-to-video generation performance. Although VideoComposer is not specifically tailored for text-to-video generation, its versatility allows VideoComposer to perform the traditional text-to-video generation task effectively. In Tab. 1, we follow the evaluation settings in Video LDM [5] to adopt
Figure 6: Compositional sketch-to-video generation. In the first example, the upper video is generated using text and a single sketch as the conditions, while the lower is generated by using an additional mask sequence for finer control over the temporal patterns. For the last two examples, the upper video is generated using a single sketch and a textual condition, while the lower is generated with an additional style from a specified image.
Table 1: Text-to-video generation performance on MSR-VTT.
Method	Zero-shot	FVD ↓	CLIPSIM ↑
GODIVA [70]	No	-	0.2402
Nüwa [71]	No	-	0.2439
CogVideo (Chinese) [26]	Yes	-	0.2614
CogVideo (English) [26]	Yes	1294	0.2631
MagicVideo [83]	Yes	1290	-
Make-A-Video [53]	Yes	-	0.3049
Video LDM [5]	Yes	-	0.2929
Text-to-video pre-training (First stage)	Yes	803	0.2876
VideoComposer	Yes	580	0.2932