Title: MotionBooth: Motion-Aware Customized Text-to-Video Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: custimized generation

video motion generation
Abstract: In this work, we present MotionBooth, an innovative framework designed for animating customized subjects with precise control over both object and camera movements. By leveraging a few images of a specific object, we efficiently fine-tune a text-to-video model to capture the object's shape and attributes accurately. Our approach presents subject region loss and video preservation loss to enhance the subject's learning performance, along with a subject token cross-attention loss to integrate the customized subject with motion control signals. Additionally, we propose training-free techniques for managing subject and camera motions during inference. In particular, we utilize cross-attention map manipulation to govern subject motion and introduce a novel latent shift module for camera movement control as well. MotionBooth excels in preserving the appearance of subjects while simultaneously controlling the motions in generated videos. Extensive quantitative and qualitative evaluations demonstrate the superiority and effectiveness of our method. Models and codes will be made publicly available.

MotionBooth: Motion-Aware Customized

Abstract

In this work, we present MotionBooth, an innovative framework designed for animating customized subjects with precise control over both object and camera movements. By leveraging a few images of a specific object, we efficiently finetune a text-to-video model to capture the object's shape and attributes accurately. Our approach presents subject region loss and video preservation loss to enhance the subject's learning performance, along with a subject token cross-attention loss to integrate the customized subject with motion control signals. Additionally, we propose training-free techniques for managing subject and camera motions during inference. In particular, we utilize cross-attention map manipulation to govern subject motion and introduce a novel latent shift module for camera movement control as well. MotionBooth excels in preserving the appearance of subjects while simultaneously controlling the motions in generated videos. Extensive quantitative and qualitative evaluations demonstrate the superiority and effectiveness of our method. Models and codes will be made publicly available.

1 Introduction

Generating videos for customized subjects, such as specific scenarios involving a particular dog's type or appearance, has gained research attention [55, 25, 42]. This customized generation field originated from text-to-image (T2I) generation methods, which learn a subject's appearance from a few images and generate diverse images of that subject [12, 42, 30]. Following them, subject-driven text-to-video (T2V) generation has seen increasing interest, which has found a wide range of applications in personal shorts or film production [55, 25, 42, 57, 13]. Can you imagine your toy riding along the road from a distance to the camera or your pet dog dancing on the street from the left to the right? However, rendering such lovely imaginary videos is a challenging task. It often involves subject learning and motion injection while maintaining the generative capability to generate diverse scenes. Notably, VideoBooth [25] trains an image encoder to embed the subject's appearance into the model, generating a short clip of the subject. However, the generated videos often display minimal or missing motion, resembling a "moving image." This approach underutilizes the motion diversity of pre-trained T2V models. Another line of works [57, 61, 13] fine-tunes the customized model on specific videos, requiring motion learning for each specific camera or subject motion type. Their pipelines restrict the type of motion and require fine-tuning a new adapter for each motion type, which is inconvenient and computationally expensive.

The key lies in the conflict between subject learning and video motion preservation . During subject learning, training on limited images of the specific subject significantly shifts the distribution of the base T2V model, leading to significant degradation (e.g., blurred backgrounds and static video). Therefore, existing methods often need additional motion learning for specific motion control. In this paper, we argue that the base T2V model already has diverse motion prior, and the key is to preserve video capability during subject learning and digging out the motion control during inference.

To ensure subject-driven video generation with universal and precise motion control, we present MotionBooth, which can perform motion-aware customized video generation . The videos generated by MotionBooth are illustrated in Fig. 1. MotionBooth can take any combination of subject, subject motion, and camera motion as inputs and generate diverse videos, maintaining quality on par with pre-trained T2V models.

MotionBooth learns subjects without hurting video generation capability, enabling a training-free motion injection for subject-driven video generation. First, during subject learning, we introduce subject region loss and video preservation loss, which enhance both subject fidelity and video quality. In addition, we present a subject token cross-attention loss to connect the customized subject with motion control signals. During inference, we propose training-free techniques to control the camera and subject motion. We directly manipulate the cross-attention maps to control the subject motion. We also propose a novel latent shift module to govern the camera movement. It shifts the noised latent to move the camera pose. Through quantitative and qualitative experiments, we demonstrate the superiority and effectiveness of the proposed motion control methods, and they can be applied to different base T2V models without further tuning.

Our contributions are summarized as follows: 1) We propose a unified framework, MotionBooth, for motion-aware customized video generation. To our knowledge, this is the first framework capable of generating diverse videos by combining customized subjects, subject motions, and camera movements as input. 2) We propose a novel loss-augmented training architecture for subject learning. This includes subject region loss, video preservation loss, and subject token cross-attention loss, significantly enhancing subject fidelity and video quality. 3) We develop innovative, training-free methods for controlling subject and camera motions. Extensive experiments demonstrate that MotionBooth outperforms existing state-of-the-art video generation models.

2 Related Work

Text-to-video generation. T2V generation leverages deep learning models to interpret text input and generate corresponding video content. It builds upon earlier breakthroughs in text-to-image generation [44, 41, 19, 21, 37, 48, 60, 62] but introduces more complex dynamics by incorporating motion and time [46, 20, 18, 2, 68, 59]. Recent advancements particularly leverage diffusion-based architectures. Notable models such as ModelScopeT2V [51] and LaVie [54] integrate temporal layers within spatial frameworks. VideoCrafter1 [6] and VideoCrafter2 [7] address the scarcity of video

Training

Self Attn.

Cross Attn.

Self Attn.

Cross Attn.

Cross-Attention Loss

ℒ𝑠𝑡𝑐𝑎

'A

[V] dog

'

Extract cross-attention maps

𝐀

at [V] dog

Video Preservation Loss

ℒ𝑣𝑖𝑑

Subject Region Loss

ℒ𝑠𝑢𝑏

Cross Attn. Map

Trainable

Inference

·  ·  ·

·  ·  ·

·  ·  ·

Self Attn.

Cross Attn.

Self Attn.

Cross Attn.

·  ·  ·

·  ·  ·

·  ·  ·

'A

[V] dog

running on the beach'

Latent Shift

Step

t

Latent

Camera Control

Camera

Direction

Shifted Latent

Subject Motion Control

Bounding Box Sequence

[V]

<end>

Edit Cross-Attention Maps

Generated Video

<start>

a

·  ·  ·

amplify

suppress

dog

No

Change

No

Change

Extra Inputs

Camera Direction

Bounding Box

Sequence

Image/Video Outputs

𝜎1~𝜎2

𝒄𝑐𝑎𝑚

data by utilizing high-quality image datasets. Latte [36] and W.A.L.T [14] adopt Transformers as backbones [49]. VideoPoet [29] explores generating videos autoregressively to produce consistent long videos. Recent Sora [3] excels in generating videos with impressive quality, stable consistency, and varied motion. Despite these advancements, controlling video content through text alone remains challenging, highlighting a continuing need for research into more refined control signals.

Customized generation. Generating images and videos with customized subjects is attracting growing interest. Most works concentrate on learning a specific subject with a few images from the same subject [23, 8, 10, 43, 47, 39] or specific domains [15, 16, 50]. Textual Inversion [12] proposes to train a new word to capture the feature of an object. In contrast, DreamBooth [42] fine-tunes the whole U-Net, resulting in a better IP preservation ability. Following them, many works explore more challenging tasks, such as customizing multiple objects [30, 55, 33, 5], developing common subject adapter [58, 25, 65, 11, 67], and simultaneously controlling their positions [11, 33]. However, the customization of video models from a few images often results in overfitting. The models fail to incorporate significant motion dynamics. A recent work, DreamVideo [57], addresses this by learning specific motion types from video data. Yet, this method is restricted to pre-defined motion types and lacks the flexibility of text-driven input. In contrast, our work introduces MotionBooth to control both the subject and camera motions without needing pre-defined motion prototypes.

Motion-aware video generation. Recent works explore incorporating explicit motion control in video generation. This includes camera and object motions. To control camera motion, existing works like AnimateDiff [13], VideoComposer [53], CameraCtrl [17], Direct-A-Video [66], and MotionCtrl [56] design specific modules to encode the camera movement or trajectory. These models usually rely on training on large-scale datasets [1, 9], leading to high computational costs. In contrast, our MotionBooth framework builds a training-free camera motion module that can be easily integrated with any T2V model, eliminating the need for re-training. For object motion control, recent works [63, 31, 32, 24, 66, 4, 69, 27, 22] propose effective methods to manipulate attention values during the inference stage. Inspired by these approaches, we connect subject text tokens to the subject position using a subject token cross-attention loss. This allows for straightforward control over the motion of a customized object by adjusting cross-attention values.

3 Method

3.1 Overview

Task formulation. We focus on generating motion-aware videos featured by a customized subject. To customize video subjects, we fine-tune the T2V model on a specific subject. This process can be accomplished with just a few (typically 3-5) images of the same subject. During inference, the fine-tuned model generates motion-aware videos of the subject. The motion encompasses both camera and subject movements, which are freely defined by the user. For camera motion, the user

inputs the horizontal and vertical camera movement ratios, denoted as c cam = [ c x , c y ] . For subject motion, the user provides a bounding box sequence [ B 1 , B 2 , ..., B L ] to indicate the desired positions of the subject, where L represents the video length. Each bounding box specifies the x-y coordinates of the top-left and bottom-right points for each frame. By incorporating these conditional inputs, the model is expected to generate videos that include a specific subject, along with predefined camera movements and subject motions.

Overall pipeline. The overall pipeline of MotionBooth is illustrated in Fig. 2. During the training stage, MotionBooth learns the appearance of the given subject by fine-tuning the T2V model. To prevent overfitting, we introduce video preservation loss and subject region loss in Section 3.2. Additionally, we propose a subject token cross-attention (STCA) loss in Section 3.2 to explicitly connect the subject tokens with the subject's position on cross-attention maps, facilitating the control of subject motion. Camera and subject motion control are performed during the inference stage. We manipulate the cross-attention maps by amplifying the subject tokens and their corresponding regions while suppressing other tokens in Section 3.3. This ensures that the generated subjects appear in the desired positions. By training on the cross-attention map, the STCA loss enhances the subjects' motion control. For camera movement, we introduce a novel latent shift module to shift the noised latent directly, achieving smooth camera movement in the generated videos in Section 3.4.

3.2 Subject Learning

Given a few images of a subject, previous works have demonstrated that fine-tuning a diffusion model on these images can effectively learn the appearance of the subject [42, 23, 8, 10, 43, 47]. However, two significant challenges remain. First, due to the limited size of the dataset, the model quickly overfits the input images, including their backgrounds, within a few steps. This overfitting of the background impedes the generation of videos with diverse scenes, a problem also noted in previous works [42, 12]. Second, fine-tuning T2V models using images can impair the model's inherent ability to generate videos, leading to severe background degradation in the generated videos. To illustrate these issues, we

(b) Subject

(c) Region

Video

(a) Pre-trained text-to-video

a dog running on the grass

a [V] dog running on the grass

(e) Region

Video

(f) Region

Video

(d) Region

Video

conducted a toy experiment. As depicted in Fig. 3, without any modifications, the model overfits the background to the subject image. To address this, we propose computing the diffusion reconstruction loss solely within the subject region. However, even with this adjustment, the background in the generated videos remains over-smoothed. This degradation likely results from tuning a T2V model exclusively with images, which damages the model's original weights for video generation. To mitigate this, we propose incorporating video data as preservation data during the training process. Although training with video data but without subject region loss still suffers from overfitting, our approach, MotionBooth, can generate videos with detailed and diverse backgrounds.

Preliminary. T2V diffusion models learn to generate videos by reconstructing noise in a latent space [42, 30, 55, 12]. The input video is first encoded into a latent representation z 0 . Noise ϵ is added to this latent representation, resulting in a noised latent z t , where t represents the timestamp. This process simulates the reverse process of a fixed-length Markov Chain [41]. The diffusion model ϵ θ is trained to predict this noise. The training loss, which is a reconstruction loss, is given by:

where c is the conditional input used in classifier-free guidance methods, which can be text or a reference image. During inference, a pure noise z T is gradually denoised to a clean latent z ′ 0 , where T is the length of the Markov Chain. The clean latent is then decoded back into RGB space to generate the video X ′ .

Subject region loss. To address the challenge of overfitting backgrounds in training images, we propose a subject region loss. The core idea is to calculate the diffusion reconstruction loss exclusively

within the subject region, thereby preventing the model from learning the background. Specifically, we first extract the subject mask for each image. This can be done manually or through automatic methods, such as a segmentation model. In practice, we use SAM [28] to collect all the masks. The subject region loss is then calculated as follows:

where M represents the binary masks for the training images. These masks are resized to the latent space to compute the dot product. c i is a fixed sentence in the format "a [V] [class name]," where "[V]" is a rare token and "[class name]" is the class name of the subject [42]. We have found that with the subject region loss, the trained model effectively avoids the background overfitting problem.

Video preservation loss. Image customization datasets like DreamBooth [42] and CustomDiffusion [30] provide excellent examples of multiple images from the same subject. However, in the customized video generation task, directly fine-tuning the video diffusion model on images leads to significant background degradation. Intuitively, this image-based training process may harm the original knowledge embedded in video diffusion models. To address this, we introduce a video preservation loss designed to maintain video generation knowledge by joint training with video data. Unlike the class-specific preservation data used in previous works [42, 55], we utilize common videos with captions denoted as c v . Our experiments in Section 4 demonstrate that common videos are more effective for subject learning and preserving video generation capabilities. The loss function is formulated as follows:

Subject token cross-attention loss. To control the subject's motion, we directly manipulate the cross-attention maps during inference. Since we introduce a unique token, '[V]', in the training stage and associate it with the subject, we need to link this special token to the subject's position within the cross-attention maps. As illustrated in Fig. 4, fine-tuning the model does not effectively connect the unique token to the cross-attention maps. Therefore, we propose a Subject Token Cross-Attention (STCA) loss to guide this process explicitly. First, we extract the crossattention map, A , at the tokens '[V] [class name]'. We then apply a Binary Cross-Entropy Loss to ensure that the corresponding attention map is larger at the subject's position and smaller outside this region. This process incorporates the subject mask and can be expressed as:

(a) Input Images

(b) w/o STCA loss

(c) w/ STCA loss

During training, the overall loss function is defined as:

where λ 1 and λ 2 are hyperparameters that control the weights of the different loss components.

3.3 Subject Motion Control

We chose bounding boxes as the motion control signal for subjects because they are easy to draw and manipulate. In contrast, providing object masks for every frame is labor-intensive, requiring consideration of the subject's shape transformation between frames. In practice, we find that bounding boxes are sufficient for precisely controlling the positions of subjects. Previous works like GLIGEN [31] attempt to control object positions by training an extra condition module with large-scale image data. However, these training methods fix the models and cannot easily align with customized models fine-tuned for specific subjects. Therefore, we adopt an alternative approach that

directly edits the cross-attention maps during inference in a training-free manner [66, 27, 4]. This cross-attention editing method is plug-and-play and can be used with any customized model.

In cross-attention layers, the query features Q are extracted from the video latent and represent the vision features. The key and value features K and V are derived from input language tokens. The calculation process of the edited cross-attention layer can be formulated as follows:

where d is the feature dimension of Q and serves as a normalization term. QK ⊤ √ d is the normalized production between Q and K , representing the attention scores between vision and language features. We manipulate the production by adding a new term α S , where S has positive values on the subject region provided in bounding boxes and large negative values outside the desired positions. α is a hyperparameter to control the editing strength. The editing matrix S is set as follows:

where i , j , and k indicate the vision token, language token, and frame indexes, respectively. P represents the indexes for subject language tokens in the text prompt. In this work, we choose '[V]' and '[class name]' as subject tokens. The SCTA loss in Section 3.2 binds the two tokens with cross-attention maps. t is the denoising timestamp and τ is a hyperparameter defining a timestamp threshold. Since diffusion models tend to form the approximate object layout in earlier denoising steps and refine the details in later steps [63], we apply stronger attention amplification in earlier steps and no amplification in later steps. Note that attention suppression outside the bounding box regions persists throughout the generation. | B k | and | Q | are the areas of the box and query, respectively. Following previous works [27, 66], smaller boxes should have larger amplifications, and we do not apply any editing on the <start> and <end> tokens.

Discussion. An important aspect is how the necessary information from other language tokens is integrated into the generated outputs, given that tokens such as verbs and background nouns are assigned minimal values. We propose that this information is extracted through the <start> and <end> tokens. Given that Transformer-based language encoders like CLIP [40] are typically trained on classification tasks, they often encode the overall context of a sentence into these special tokens. Thus, despite the suppression of other tokens during the softmax calculation, the model can still access relevant information about verbs, background elements, and other components necessary for the generation process. To support this explanation, we conducted an experiment in which we examined the softmax outputs using a naive text-to-video pipeline. The results showed that the <start> token consistently held the highest softmax value, close to 1, while the <end> token had the second-largest value. The remaining tokens, including those representing nouns, adjectives, verbs, and conjunctions, were distributed among the remaining softmax values.

3.4 Camera Movement Control

Simply editing the cross-attention map can efficiently control the motion of the subject. This suggests that the latent can be considered a "shrunk image," which maintains the same visual geographic distribution as the generated images. For camera movement control, an intuitive approach is to directly shift the noised latent during inference based on the camera movement signal c cam = [ c x , c y ] . The latent shift pipeline is illustrated in Table 2. The key challenge with this idea is filling in the missing parts caused by the latent shift (the question mark region in Step

'A

waterfall in a beautiful forest with fall foliage'

Noised Latent

𝑧𝜎1

Step 1:

Shift

x-axis

y-axis

Step 2:

Sample Tokens

Step 3:

Fill in the Missing Part

Denoising

𝜎1 -𝜎2

steps

Shifted Noised Latent

𝑧𝜎2

Input Camera

Movement

1). To address this issue, we propose sampling tokens from the original noised latent and using them to fill the gap. This is based on the prior knowledge that when a camera moves in a video, the new scene it captures is semantically close to the previous one. For example, in a video with forest scenes, when the camera pans left, it is highly likely to capture more trees similar to those in the original scene. Another assumption is that in a normally angled video, a visual element is more likely to be semantically close to elements along the same x-axis or y-axis rather than other elements. For instance, in the waterfall video in Fig. 5, trees are at the top and bottom, spreading horizontally, while the waterfall spans the middle x-axis area. Experimentally, we over that sampling tokens horizontally and vertically provides better initialization and results in smoother video transitions. Randomly sampling tokens degrades the generated video quality. The latent shift process for timestamp t can be formulated as follows:

where h x and h y are sampled tokens along the x and y axes, respectively. Crop ( · ) removes the tokens outside the camera view after the shift. B is the subject bounding box. We filter out the tokens belonging to the subjects because they are not likely to occur in the new scenes. In addition, to avoid a drastic change in latent in one shift, we spread the latent shift over multiple timestamps, with each step only shifting a small number of tokens. Note that the latent shift needs to be applied after the subject's approximate layout is fixed but before the video details are completed. We set a pair of hyperparameters σ 1 and σ 2 . The latent shift only applies in the timestamp range [ σ 1 , σ 2 ] .

4 Experiments

4.1 Experimental Setup

Datasets. For customization, we collect a total of 26 objects from DreamBooth [42] and CustomDiffusion [30]. These objects include pets, plushies, toys, cartoons, and vehicles. To evaluate camera and object motion control, we built a dataset containing 40 text-object motion pairs and 40 text-camera motion pairs, ensuring that the camera and object motion patterns are consistent with the text prompts. This dataset evaluates the videos generated for each subject in various scenarios and motions.

Implementation details. We train MotionBooth for 300 steps using the AdamW optimizer, with a learning rate of 5e-2 and a weight decay of 1e-2. We collect 500 preservation videos from the Panda-70M [9] training set, chosen randomly. Each batch consists of one batch for images and one for videos, with batch sizes equal to the number of training images and 1 for images and videos, respectively. The loss weight parameters λ 1 and λ 2 are set to 1.0 and 0.01. We use Zeroscope and LaVie as base models. During inference, we perform 50-step denoising using the DDIM scheduler and set the classifier-free guidance scale to 7.5. The generated videos are 576x320x24 and 512x320x16 for Zeroscope and LaVie, respectively. The training process finishes in around 10 minutes in a single NVIDIA A100 80G GPU. Additional implementation details can be found in Appendix A.1.

Baselines. Since we are pioneering motion-aware customized video generation, we compare our methods with closely related works, including DreamBooth [42], CustomVideo [55], and DreamVideo [57]. Dreambooth customizes subjects for text-to-image generation. We follow its practice with class preservation images and fine-tune T2V models for generating videos. CustomVideo is a recent video customizing method. We adopt its parameter-efficient training procedure. DreamVideo learns motion patterns from video data. To provide such data, we sample videos from Panda-70M, which are most relevant to the evaluation motions. Since these methods cannot control motions during inference, we apply our camera and object motion control technologies for a fair comparison. Additionally, we compare our camera control method with training-based methods, AnimateDiff [13] and CameraCtrl [17], focusing on camera motion control without subject customization. Since AnimateDiff is trained with only basic camera movement types and cannot take user-defined camera movement c cam = [ c x , c y ] as input, we use the closest basic movement type for evaluation.

Evaluation metrics. We evaluate motion-aware customized video generation from four aspects: region subject fidelity, temporal consistency, camera motion fidelity, and video quality. 1) To ensure the subject is well-preserved and accurately generated in the specified motion, we introduce region CLIP similarity (R-CLIP) and region DINO similarity metrics (R-DINO). These metrics utilize the CLIP [40] and DINOv2 [38] models to compute the similarities between the subject images and frame regions indicated by bounding boxes. Additionally, we use CLIP image-text similarity (CLIP-T) to measure the similarity between entire frames and text prompts. 2) We evaluate temporal consistency by computing CLIP image features between each consecutive frame. 3) We use VideoFlow [45] to predict the optical flow of the generated videos. Then, we calculate the flow error by comparing the predicted flow with the ground-truth camera motion provided in the evaluation dataset. 4) We randomly select 1000 videos from the MSRVTT dataset [64], predict their camera motion sequences with VideoFlow [45], and compute the FVD metric for camera motion control only.

4.2 Main Results

Quantitative results. We conduct quantitative comparisons with baseline models on both motionaware customized video generation and camera movement control. The results for motion-aware customized video generation are shown in Table 1. The results demonstrate that MotionBooth outperforms all baselines on both Zeroscope and LaVie models, indicating that our proposed technologies can be extended to different T2V models. Thanks to the training-free architecture of subject and camera motion control methods, MotionBooth is expected to be adaptable to more open-sourced models in the future, such as Sora [3]. Notably, DreamVideo [57] achieves the second-best scores in T-Cons. and flow error, which aligns with our observation that incorporating video data as auxiliary training data enhances video generation performance. On the other hand, CustomVideo [55] shows inferior performance in R-DINO scores, indicating a poorer ability to generate subjects in given positions. This may be attributed to its approach of only fine-tuning the text embeddings and cross-attention layers of the diffusion models, which is insufficient for learning the subjects.

For camera movement control, we compare our method with two training-based methods, AnimateDiff [13] and CameraCtrl [17]. The results are shown in Table 2. Remarkably, MotionBooth achieves superior results compared to the two baselines with our training-free latent shift module. Specifically, MotionBooth outperforms the recent method CameraCtrl by 0.617, 0.015, and 0.009 in flow error, CLIP-T, and T-Cons. metrics with Zeroscope, and 0.511, 0.004, and 0.024 for the LaVie model. These results demonstrate that the latent shift method is simple yet effective.

Qualitative results. The qualitative comparison results for video generation with customized objects and controlled subject motions are presented in Fig. 6. Our observations reveal that MotionBooth excels in subject motion alignment, text prompt alignment, and overall video quality. In contrast,

MotionBooth

(Ours)

CustomVideo

DreamBooth

DreamVideo

a [V] cat jumping off the stairs

+

Subject

+

Motion

a [V} toy riding a bike on the road with a view of mountains

+

Zeroscope

Ours (Zeroscope)

AnimateDiff

CameraCtrl

A playful puppy frolicking in flowers

Up Right

Ours (Zeroscope)

AnimateDiff

CameraCtrl

A squirrel gathering acorns

Up

Ours (LaVie)

CameraCtrl

A waterfall in a beautiful forest

with fall foliage

Down Left

AnimateDiff

Ours (LaVie)

CameraCtrl

clown fish swimming in a coral reef

Right

AnimateDiff

DreamBooth and CustomVideo produce videos with vague backgrounds, highlighting that generated backgrounds deteriorate when training is conducted without video data. Additionally, CustomVideo and DreamVideo struggle to capture the subjects' appearances, likely because their approach tunes only part of the diffusion model, preventing the learning process from fully converging.

We also conduct qualitative experiments focused on camera movement control, with results shown in Fig. 7. AnimateDiff, limited to basic movements, does not support user-defined camera directions. Although the CameraCtrl method can accept user input, it generates videos with subpar aesthetics and objects that exhibit flash movements. In contrast, our MotionBooth model outperforms both the Zeroscope and Lavie models. The proposed latent method generates videos that adhere to user-defined camera movements while maintaining time consistency and high video quality.

4.3 Ablation Studies

Training technologies. We analyze the technologies proposed during the subject learning stage. The ablation results are shown in Table 3. Clearly, without the proposed modules, the quantitative

metrics drop accordingly. These results demonstrate that the proposed subject region loss, STCA loss, and video preservation loss are beneficial for subject learning and generating motion-aware customized videos. Specifically, the R-DINO metric decreases significantly by 0.256 without the subject region loss, highlighting its core contribution in filtering out image backgrounds during training. Additionally, the "w/ class video" experiment, which uses class-specific videos instead of randomly sampled common videos, yields worse results. This approach restricts the scenes and backgrounds in class-specific videos, hindering the models' ability to generalize effectively.

4.4 Human Preference Study

To evaluate our approach to understanding user preferences, we conducted a user study experiment. We collected 30 groups of videos generated by MotionBooth and baseline methods. We then asked 7 colleagues to select the best videos based on the following criteria: subject motion alignment, camera movement alignment, subject appearance alignment, and temporal consistency. For each group of videos, the annotators selected only the best one. For the subject appearance alignment, the annotators were provided with corresponding subject images. As

16.4

3.4

1.5

9.3

0.8

4

3.6

6

7.4

16

13.3

9.2

75.4

75.6

81.6

75.5

0

20

40

60

80

100

Motion Alignment

Camera Alignment

Subject Alignment

Temporal Consistency

Preference Rate (%)

DreamBooth

CustomVideo

DreamVideo

MotionBooth (Ours)

shown in Fig. 8, MotionBooth was the most preferred method across all models and evaluation aspects, particularly in subject appearance alignment. These results further demonstrate the effectiveness of our method.

4.5 Limitations and Future Work

In Fig. 9, we illustrate several failure cases of MotionBooth. One significant limitation of MotionBooth is its struggle with generating videos involving multiple objects. As shown in Fig. 9(a), the subject's appearance can sometimes merge with other objects, resulting in visually confusing outputs. This issue might be resolved by incorporating advanced training technologies for multiple subjects.

Another limitation is the model's capability to depict certain motions indicated by the text prompt. As depicted in Fig. 9(b), MotionBooth may fail to accurately represent motions that are unlikely to be performed by the subject. For example, it is hard to imagine a scene where a wolf plushie is riding a bike. These failure cases highlight the need for further improvement in the model's subject separation and motion understanding capabilities to enhance the realism and accuracy of the generated videos. Utilizing more powerful T2V models may eliminate these

a [V] dog and a horse walking on the grass

Subject

(a) Failure case on multiple objects

Subject

a [V] plushie riding a bike on the road

(b) Failure case on hard motions

drawbacks. Future work could focus on refining these aspects to address the current limitations and provide more robust performance in complex scenarios.

5 Conclusion

This paper introduces MotionBooth, a novel framework for motion-aware, customized video generation. MotionBooth fine-tunes a T2V diffusion model to learn specific subjects, utilizing subject region loss to focus on the subject area. The training procedure incorporates video preservation data to prevent background degradation. Additionally, an STCA loss is designed to connect subject tokens with the cross-attention map. During inference, training-free technologies are proposed to control both subject and camera motion. Extensive experiments demonstrate the effectiveness and generalization ability of our method. In conclusion, MotionBooth can generate vivid videos with given subjects and controllable subject and camera motions.

Acknowledgement. This work is supported by the National Key Research and Development Program of China (No. 2023YFC3807600).

References

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV , 2021. 3

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR , 2023. 2

Tim Brooks, Bill Peebles, Connor Holmes, Yufei Guo Will DePue, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Sora: Creating video from text, 2024. 3, 8

Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. Motionzero: Zero-shot moving object control framework for diffusion-based video generation. arXiv preprint arXiv:2401.10150 , 2024. 3, 6, 14, 15, 16, 17

Hong Chen, Xin Wang, Guanning Zeng, Yipeng Zhang, Yuwei Zhou, Feilin Han, and Wenwu Zhu. Videodreamer: Customized multi-subject text-to-video generation with disen-mix finetuning. arXiv preprint arXiv:2311.00990 , 2023. 3

Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512 , 2023. 2

Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047 , 2024. 2

Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation. arXiv preprint arXiv:2305.03374 , 2023. 3, 4

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402.19479 , 2024. 3, 7

Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. NeurIPS , 2024. 3, 4

Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. CVPR , 2024. 3

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022. 2, 3, 4

Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR , 2024. 2, 3, 7, 8

Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 , 2023. 3

Yue Han, Jiangning Zhang, Junwei Zhu, Xiangtai Li, Yanhao Ge, Wei Li, Chengjie Wang, Yong Liu, Xiaoming Liu, and Ying Tai. A generalist facex via learning unified facial representation. arXiv preprint arXiv:2401.00551 , 2023. 3

Yue Han, Junwei Zhu, Keke He, Xu Chen, Yanhao Ge, Wei Li, Xiangtai Li, Jiangning Zhang, Chengjie Wang, and Yong Liu. Face adapter for pre-trained diffusion models with fine-grained id and attribute control. arXiv preprint arXiv:2405.12970 , 2024. 3

Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101 , 2024. 3, 7, 8

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022. 2

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020. 2

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In NeurIPS , 2022. 2

Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICML , 2023. 2

Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Motionmaster: Training-free camera motion transfer for video generation. arXiv preprint arXiv:2404.15789 , 2024. 3

Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, and Qian He. Dreamtuner: Single image is enough for subject-driven generation. arXiv preprint arXiv:2312.13691 , 2023. 3, 4

Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusion. arXiv preprint arXiv:2312.07509 , 2023. 3

Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In CVPR , 2024. 2, 3

Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In ICCV , 2023. 8, 14, 16, 18

Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV , 2023. 3, 6

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023. 5

Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125 , 2023. 3

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In CVPR , 2023. 2, 3, 4, 5, 7, 14

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR , 2023. 3, 5

Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. arXiv preprint arXiv:2309.15091 , 2023. 3

Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones 2: Customizable image synthesis with multiple subjects. arXiv preprint arXiv:2305.19327 , 2023. 3

Wan-Duo Kurt Ma, J. P. Lewis, and W. Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. arXiv preprint arXIv:2401.00896 , 2024. 14, 16, 17

Wan-Duo Kurt Ma, J. P. Lewis, Avisek Lahiri, Thomas Leung, and W. Bastiaan Kleijn. Directed diffusion: Direct control of object placement through attention guidance. In AAAI , 2024. 14, 15, 16, 17

Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 , 2024. 3

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , 2021. 2

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023. 8

Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast identity-preserved personalization. arXiv preprint arXiv:2312.06354 , 2023. 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICCV , 2021. 6, 8

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022. 2, 4

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR , 2023. 2, 3, 4, 5, 7, 8, 14

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06949 , 2023. 3, 4

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. NeurIPS , 2022. 2

Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. In ICCV , 2023. 8

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022. 2

James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora. arXiv preprint arXiv:2304.06027 , 2023. 3, 4

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020. 2

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017. 3

Chaoyang Wang, Xiangtai Li, Lu Qi, Henghui Ding, Yunhai Tong, and Ming-Hsuan Yang. Semflow: Binding semantic segmentation and image synthesis via rectified flow. arXiv preprint arXiv:2405.20282 , 2024. 3

Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023. 2

Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXIv:2402.01566 , 2024. 14, 15

Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. NeurIPS , 2024. 3

Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103 , 2023. 2, 16

Zhao Wang, Aoxue Li, Enze Xie, Lingting Zhu, Yong Guo, Qi Dou, and Zhenguo Li. Customvideo: Customizing text-to-video generation with multiple subjects. arXiv preprint arXiv:2401.09962 , 2024. 2, 3, 4, 5, 7, 8

Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641 , 2023. 3, 8, 14, 16

Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with customized subject and motion. arXiv preprint arXiv:2312.04433 , 2023. 2, 3, 7, 8

Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In ICCV , 2023. 3

Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, et al. Towards language-driven video inpainting via multimodal large language models. CVPR , 2024. 2

Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, and Dacheng Tao. Towards open vocabulary learning: A survey. T-PAMI , 2024. 2

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV , 2023. 2

Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Mosaicfusion: Diffusion models as data augmenters for large vocabulary instance segmentation. arXiv preprint arXiv:2309.13042 , 2023. 2

Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV , 2023. 3, 6

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR , 2016. 8

Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In CVPR , 2023. 3

Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprint arXiv:2402.03162 , 2024. 3, 6

Haoyu Zhao, Tianyi Lu, Jiaxi Gu, Xing Zhang, Zuxuan Wu, Hang Xu, and Yu-Gang Jiang. Videoassembler: Identity-consistent video generation with reference entities using diffusion model. arXiv preprint arXiv:2311.17338 , 2023. 3

Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022. 2

Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self-attention for long-range image and video generation. arXiv preprint arXiv:2405.01434 , 2024. 3

骨

A Appendix

Overview. The supplementary includes the following sections:

Appendix A.1. Implementation details of the experiments.

Appendix A.2. Discussions about our method and the differences with related methods.

Appendix A.3. Comparison with more baselines.

Appendix A.4. Ablation studies.

Appendix A.5. Social impacts.

Appendix A.6. More qualitative results.

Video Demo. We also present a video in a separate supplementary file, which shows the results in video format.

A.1 Implementation Details

Hyperparameters. For the LaVie model, we set α = 10.0, τ = 0.7 T , σ 1 = 0.8 T , and σ 2 = 0.6 T . For the Zeroscope model, we set α = 10.0, τ = 0.9 T , σ 1 = 0.9 T , and σ 2 = 0.7 T .

Evaluation dataset. We collect a total of 26 subjects from DreamBooth [42] and CustomDiffusion [30]. We show one image for each subject in Fig. 10. The subjects contain a wide variety of types, including pets, plushie toys, cartoons, and vehicles, which can provide us with a thorough analysis of the model's effectiveness.

User study interface. We show the application interface for human preference study in Fig. 11. During user study, we ask the annotators to select the best video based on the question, e.g., 'Which video do you think has the best temporal consistency?'

Pseudo-code of latent shift. To present the latent shift module more clearly, we show the

User Study

?

Select

pseudo-code of the algorithm in Fig. 14. Our latent shift module can control the camera movement in videos in a training-free manner at minimal costs.

A.2 Discussions

Differences with related works. Our proposed subject and camera motion control methods differ in several ways from previously established approaches, such as TrailBlazer [34], Directed Diffusion [35], Boximator [52], Motion-Zero [4], MotionCtrl [56], and Text2Video-Zero [26].

TrailBlazer [34] and Directed Diffusion [35] use a training-free approach for controlling object motion by manipulating the cross-attention maps. Specifically, TrailBlazer [34] adjusts both spatial and

Frame 1

Frame 12

Frame 24

Zoom In

Zoom Out

(a) Examples of enlarging or shrinking the subject bounding box.

Frame 1

Frame 8

Frame 16

1x camera speed

2x camera speed

3x camera speed

（

fail

）

Down Left

(b) Examples of gradually enlarging the camera mo- tion speed.

Frame 1

Frame 8

Frame 16

(c) Examples of camera motion control with large foreground objects.

Frame 1

Frame 12

Frame 24

Left

(d) Example of combining zoom-in and camera mo- tion control.

A [v] dog running on grass

Subject

Generated Video

(e) Example of masking images during training.

A wooden barrel drifting on the river

Straight-Sample

(Ours)

Random Sample

Right

Loop

Reflect

(Text2Video-Zero)

(f) Ablations of the filling method in latent shift.

No Camera Motion

Down

Down Left

Down Right

Left

Right

Up

Up Left

Up Right

A playful puppy frolicking in flowers

Latent Shift

(Ours)

Text Guidance

Up Right

Right

A Villa in a garden

(g) Examples of controlling both subject and camera motion. (h) Comparison of latent shift and text guidance to control camera motion.

temporal cross-attention by scaling the attention maps with a hyper-parameter of less than 1. Directed Diffusion [35] focuses on image generation in a similar manner. Our approach, in contrast, targets only the spatial cross-attention, setting attention values outside the object's bounding box to zero. This targeted manipulation simplifies implementation and significantly enhances the performance in generating motion-aware customized videos.

Boximator [52] relies on a training-based technique, requiring box coordinates as inputs to a newly trained self-attention layer. In comparison, our method is training-free, thus providing a more accessible and user-friendly solution for controlling subject and camera motion.

Motion-Zero [4] also operates without additional training, but it utilizes a test-time tuning technique to adjust the latent space using cross-attention map loss during the denoising process. This technique, however, increases the generation time and memory requirements considerably-from approximately 15 seconds to several minutes per video. Additionally, our experiments indicate that Motion-Zero often produces poor-quality outputs, with videos collapsing into unrecognizable elements. This outcome is likely due to the detrimental effects of test-time tuning on parameters and latent distributions in customized scenarios. In contrast, our method directly manipulates the cross-attention map, adding only 0.3 seconds to the generation process and consistently yielding more reliable visual outcomes.

MotionCtrl [56] uses a training-based approach that requires inputting point trajectories to control camera poses and object motion. Our method does not involve additional training, offering a simpler alternative for subject and camera motion control.

Text2Video-Zero [26] builds on a pre-trained text-to-image (T2I) model and extends it to video generation by utilizing consistent noise across frames. However, this approach is unsuitable for text-to-video (T2V) models, which typically use different noise for each frame. Additionally, Text2Video-Zero uses latent shifting for overall scene movement and mirrors latent information to fill in missing regions. In comparison, our approach employs random sampling along the x and y axes, resulting in more concise and coherent video generation for camera motion control.

We conducted quantitative experiments to evaluate the performance of these methods, as reported in Table 2 and Table 4b. Some methods are not included due to the unavailability of their code. Nonetheless, our results indicate that our method generally outperforms the other alternatives. In particular, we observed significant limitations in the test-time tuning strategy of Motion-Zero [4] when applied to customized video generation, which further emphasizes the strengths of our approach.

Zoom-in/out effect with subject motion control. Our subject motion control technique effectively manages changes in bounding box size, such as gradual enlargement or reduction. Specifically, enlarging the bounding box results in an appropriate scaling of the subject, creating a zoom-in effect in the generated video. Conversely, reducing the bounding box size produces a zoom-out effect. This zoom-in and zoom-out behavior is demonstrated in Fig. 12a, where examples illustrate these effects in response to bounding box adjustments. Fig. 12d shows an example of combining zoom-in and camera motion control, demonstrating the method's flexibility.

Large camera motion speeds. We evaluate the performance of our camera motion control technique under varying levels of camera movement intensity. As depicted in Fig. 12b, the method was first tested with a camera movement vector of [ c x , c y ] = [ -0 . 5 , 0 . 45] , corresponding to a movement of half the video width to the left and nearly half downward. Under these conditions, our method successfully managed the camera movement, producing accurate results. When the camera motion speed was doubled, the model continued to perform well, demonstrating its ability to handle highspeed scenarios. However, at a speed three times the initial setting, i.e., [ c x , c y ] = [ -1 . 5 , 1 . 35] , the method exhibited limitations, resulting in only a downward tiling effect. These findings indicate that while our method can effectively manage camera movements spanning the entire video width, its performance diminishes under extremely high-speed conditions.

Camera motion control with large foreground objects. We evaluate the camera control capability in scenes containing substantial foreground objects. The results, presented in Fig. 12c, show an experiment involving a scene dominated by a large candy house, which nearly occupies the entire frame. Despite the significant presence of the foreground object, our technique effectively managed to pan the camera to the right, suggesting that the method is capable of handling complex scenes with large foreground elements. This robustness is attributed to the method's reliance on latent shifts, which simultaneously move both foreground and background elements, thereby ensuring that the presence of large foreground objects does not significantly impair performance.

Model efficiency. Training our model by fine-tuning a specific subject takes approximately 10 minutes. For LaVie [54], the naive text-to-video (T2V) pipeline takes about 15.0 seconds per video. When incorporating subject control, the inference time is 15.3 seconds per video; with camera control, it increases to 20.6 seconds per video; and when applying both camera and subject control, the inference time is 21.5 seconds per video.

A.3 Comparison with More Baselines

(a) Comparison of the latent shift method and text guidance for camera motion control. (b) Comparison of subject motion control with more baselines.

𝜏

Subject

Position

a [V] cat running

on the grass

𝛼

5.0

7.5

10.0

15.0

0.95

𝐓

0.90

𝐓

0.85

𝐓

0.80

𝐓

a child blowing

bubbles in the park

(𝜎1, 𝜎

2 )

(0.9

𝐓

, 0.7

𝐓)

Pan Right

(0.8

𝐓

, 0.6

𝐓)

(0.7

𝐓

, 0.5

𝐓)

First Frame

Last Frame

Camera Direction

(a) Ablation study on subject motion control. Only the first frame is shown. Experiments on Zeroscope.

(b) Ablation study on latent shift.

σ

Experiments on means an earlier denoising step.

Comparison with text guidance for camera motion control. we compare our latent shift method with text guidance for controlling camera motion with the results presented in Table 4a and Fig. 12h. Specifically, we utilize simple text prompts, such as 'camera pan right' and 'camera pan up right,' to influence camera movement. The generated videos reflect these instructions to some extent, especially for straightforward motions like panning to the right. However, we observe that such text prompts often lack sufficient specificity, particularly in terms of conveying critical details such as the speed or distance of the camera movement. As a result, this approach yielded sub-optimal outcomes when compared to our proposed method. This indicates that our approach offers a more precise and stable mechanism for controlling camera motion than is possible with text prompts alone.

Comparison of subject motion control with more baselines. Table 4b presents quantitative experiments comparing our method with several baseline approaches, including TrailBlazer [34], Directed Diffusion [35], and Motion-Zero [4]. The results demonstrate that our approach generally outperforms these alternatives.

A.4 Ablation Studies

Subject motion control hyperparameters. We conduct ablation studies on the hyperparameter for subject motion control, with the results presented in Fig. 13a. We examined the effects of varying the factor α of S and the maximum cross-attention manipulation timestamp τ . The findings indicate that increasing α and extending the controlling steps lead to stronger control. With lower control strengths, the subject does not appear in the desired position, or only part of its body aligns with the intended spot. Conversely, when the control strengths are too high, the generated subjects tend to appear unnaturally square in shape.

Latent shift hyperparameters. We experiment with the influence of σ 1 and σ 2 in the latent shift module. The results are shown in Fig. 13b. The results indicate that applying latent shift in the earlier steps of the denoising process results in incomplete camera movement, as evidenced by the trees in the background. Conversely, shifting the latent in the later steps degrades video quality and introduces artifacts, highlighted by the red boxes in the last row. Empirically, setting σ 1 and σ 2 to middle values provides optimal control over camera movement.

Number of preservation videos. We conduct an ablation study on the number of preservation videos. As shown in Table 5, ranging the preservation videos from 100 to 900 does not bring large changes to the quantitative scores. We conclude that the key is to use video data to preserve the video generation ability of the pre-trained T2V models. The number of video data can be flexible.

Table 6: More ablation studies.

(a) Ablation of controlling single motion type. (b) Ablation of masking the training images. (c) Ablation of the latent filling method in latent shift.

Controlling Single Motion Type. We conduct an additional experiment focusing specifically on the scenario where only subject motion is controlled, and the results are summarized in Table 6a. Notably, we observe that the inclusion of both subject and camera motion controls leads to a slight decrease in some metrics. For instance, using the Zeroscope model, there is a 0.1 drop in the R-CLIP score and a 0.01 decrease in the T-Cons metric when both types of motion are controlled compared to the scenario where only subject motion is controlled. However, there are also instances where metrics such as R-DINO and CLIP-T show a slight improvement under combined control conditions. We consider this trade-off acceptable within the context of motion-aware customized video generation. Qualitative results in Fig. 12g show that our method can successfully handle controlling both subject and camera motion.

Masking the training images. We conduct an ablation study to evaluate the effect of masking the background regions directly in pixel space of the training images, compared to masking the diffusion loss. As illustrated in Fig. 12e, directly masking the training images significantly impairs the model's capacity to learn the subject effectively. The quantitative metrics shown in Table 6b indicate a marked decrease in performance when the training images are masked. We hypothesize that this decline arises from the unnatural distribution created by the masked images, which disrupts the learning process. Consequently, we conclude that masking the diffusion loss, rather than masking the training images directly, is a more effective strategy for preserving the integrity of the learned representations.

Latent Filling Methods. We test several latent filling approaches and present the qualitative results in Fig. 12f and the quantitative results in Table 6c.

Random Init: This method involves filling the hole with random values. Our experiments revealed that this technique leads to severe artifacts due to the disruption of the natural horizontal and vertical distribution of latent pixels, ultimately degrading the overall visual quality.

Random Sample: In this approach, values are randomly sampled in the latents. Similar to Random Init , this method produced significant artifacts, leading to poor visual quality in the generated video.

Loop: This method reuses the moved-out-of-scene values to fill the missing region, thereby creating a looping background effect. While this technique was found to preserve video quality better than the random methods, it introduced a limitation in terms of flexibility for camera movements, resulting in repetitive looping effects. Therefore, it is not suitable for more diverse camera controls.

Reflect: This approach is employed in Text2Video-Zero [26], where the missing region is filled by reflecting the surrounding content. However, in our Text-to-Video (T2V) scenario, this method collapsed, failing to maintain the desired visual quality.

The quantitative results presented in Table 6c corroborate these findings. Random Init and Random Sample lead to significant artifacts, whereas the Loop method provides better visual quality but at the cost of limiting camera movement diversity. The Reflect method, despite its success in other applications, did not yield satisfactory results in our T2V context.

In conclusion, our proposed straight-sample method consistently maintained high visual quality without introducing significant artifacts or limiting camera movement flexibility, demonstrating its superiority in maintaining the desired video fidelity.

A.5 Social Impacts

Positive societal impacts. MotionBooth allows for precise control over customized subjects and camera movements in video generation, opening new avenues for artists, filmmakers, and content creators to produce unique and high-quality visual content without extensive resources or professional equipment.

Potential negative societal impacts. The ability to generate realistic customized videos could be misused to create deepfakes, leading to potential disinformation campaigns, privacy violations, and reputational damage. This risk is particularly significant in the context of political manipulation and social media. If the underlying models are trained on biased datasets, the generated content might reinforce harmful stereotypes or exclude certain groups. Ensuring diversity and fairness in training data is crucial to mitigate this risk.

Mitigation strategies. Developing and adhering to strict ethical guidelines for the use and dissemination of video generation technologies can help mitigate misuse. This includes implementing usage restrictions and promoting transparency about the generated content.

A.6 More Qualitative Results

We show more qualitative results in Fig. 15.

Subject

Subject motion

Camera motion

Generated Videos

A dog running on the beach

A motorbike moving closer on the beach with

seashore in the background

A plushie panda running on the grass

Up Right

A robot toy walking in sunrise

A plushie swimming around coral reef

Right

A wolf plushie running on the grass

Up Left

A dog jumping down the stairs

Down Left

A car driving on the highway

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We include the main claims in the abstract and introducion.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our method in Section 4.5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the implementation details in Section 4 and Appendix A.1 that are needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We are not able to provide the code at submission time. But we are making sure that our code and models will be released publically in the future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the implementation details in Section 4 and Appendix A.1 that are needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We did not conduct experiments with error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the implementation details in Section 4 and Appendix A.1 that are needed to reproduce the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conducted in the paper confirm, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the social impacts of MotionBooth in Appendix A.5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We describe the safeguards in Appendix A.5.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use existing assets with properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We introduce new assets in Section 4 and Appendix A.1.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose MotionBooth, a method to fine-tune a pre-trained text-to-video model on a collection of images of a specific object to enable the model to generate controlled videos of that object. Fine-tuning incorporates three losses: diffusion loss on image data, restricted to the object's region; video preservation loss, i.e diffusion loss on sample video data, to prevent overfitting to static images; subject token cross-attention loss, to improve controllability at generation time. During inference the motion of the object is controlled via editing the cross-attention maps to amplify the attention to the object tokens within the object's bounding boxes. The camera is controlled via shifting the noised latent and filling the new regions with latents sampled from the background of the original noised latent.\n\nSTRENGTHS:\nThe problem of controllable video generation is of high significance nowadays. The method proposed in the paper is relatively simple and the demonstrated visual results confirm the improved controllability of MotionBooth over prior work. The contributions in the paper are supported with well-designed figures aiding the clarity of the presentation. The limitations of the proposed method are discussed in the appendix.\n\nWEAKNESSES:\n1) **Novelty**: \n    - Similar training-free object motion control was proposed in [1] and [2] for image and video models respectively, where cross-attention amplification within object's bounding box was used to control its location in the generated frame. Similar training-free camera motion control was used in [3], where a series of noised latent shifts were used to control the global scene and camera motion. \nDiscussion and comparisons with those works are missing in the paper.\n    - Some missing references: [4, 5].\n\n2) **Method**: \n    - Based on the formula for subject motion control, the attention is suppressed (set to -infinity) for all query-key pairs other than the queries from the bounding box and the keys from the object tokens. This way it seems that the rest of the prompt is ignored. Moreover, it is stated in the paper that \"the attention suppression outside the bounding box regions persists throughout the generation\". Details clarifying this are missing in the paper.\n\n3) **Evaluation**: \n    - The paper lacks quantitative evaluation. Comparing CLIP features between the frames doesn't measure temporal consistency. Metrics like FVD, or optical flow warpping error could be better choices for this purpose.\n    - Details on how exactly the metrics are calculated are missing. Formal definitions of newly introduced metrics are required for better understanding. E.g. how exactly is the flow error calculated in the case when not all the motion except for the object's motion corresponds to camera motion? Having those would help with understanding the significance of the improvements reported in the tables. \n    - The evaluation dataset consists of limited number of different object classes, mostly dogs, cats and toys. Evaluation on more general collection of object classes would better support the claims made in the paper.\n    - More ablation studies would better illustrate the contributions. E.g. what would happen if the missing region in the shifted noised latent was not filled with samples from the same noised latent? Or if camera control like in [3] was used instead?\n\n4) **Results**:\n    - Figure 14 shows that bounding boxes often don't fully contain the object of interest. Given that the attention is suppressed outside of bounding boxes, this looks like a flaw in the proposed control and needs to be investigated.\n\n[1] Ma et al., Directed diffusion: Direct control of object placement through attention guidance, AAAI 2024.\n\n[2] Ma et al., TrailBlazer: Trajectory Control for Diffusion-Based Video Generation, arxiv 2024.\n\n[3] Khachatryan et al., Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators, ICCV 2023.\n\n[4] Wang et al., Boximator: Generating rich and controllable motions for video synthesis, arxiv 2024.\n\n[5] Chen et al. Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based\nVideo Generation, arxiv 2024.\n\nQUESTIONS:\n1) Could the authors discuss the connection between the proposed MotionBooth and the prior work with similar training-free controls? Some ablation studies would be ideal. \n\n2) Could the authors provide more details about their subject motion control attention suppression technique? Especially regarding the issue with attention to the prompt tokens other than the subject language tokens.\n\n3) The authors claim that their fine-tuning is efficient (line 3 in the abstract). Could the authors provide more details on this? E.g. what is the training time to adapt the model to a new subject?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitation and the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a video generated method - for customizing a subject, along with its motion, provided in the form of bounding boxes, and camera movement provided in the form of camera pose. To do this, the training method has a subject region loss to prevent bias from background and other details in the image, and a video regularization loss function to ensure that video generation properties are not forgotten. At inference, in order to control camera movements, latents are shifted. Cross-attention maps are edited with the provided bounding boxes, to fix the subject motion.\n\nSTRENGTHS:\nThe paper is written well. I particularly like the introduction and related work, where the paper clearly motivates the problem, describes the challenges and goes on to describe how it solves the problem. \n\nThe method comprises of various components, and each of the components is designed very well - they are intuitive, make sense and well formulated. The method very systematically addresses various challenges involved in the problem by designing various relevant techniques. The paper is also written very well to motivate and describe the method.\n\nWEAKNESSES:\nMy main concerns are with the experiments, which are very non-convincing. While the paper does provide detailed quantitative results for comparisons and ablations on various metrics validating the solution, the qualitative results do not support that.\n\n1. I watched the supplementary video, which has the videos for the results presented in the paper. First of all, the number of qualitative results provided is severely limited, raising questions about cherry picking. Second, the method seems to perform reasonably well on cases that either have camera motion or subject motion - the results look good. In cases involving both camera as well as subject motion, which is the theme of the paper, the results are highly suboptimal - for instance, at 1:48 of the video, the last video showing the dog does not show the subject motion appropriately.\n\n2. I would like to see more detailed results that segregate the various kinds of motion (camera and subject), to see how effective the method is for just camera motion, just subject motion and a combination of both. This will give a clearer picture of where the method is failing.\n\nQUESTIONS:\nI am unable to judge the quality of this work with the limited number of qualitative results that have been provided. Moreover, the provided qualitative results are sub-optimal, esp in the case of subject + camera motion, questioning the working of the method. Given that the limit on the size of the supplementary material is considerably large, it would have been nice if the authors had provided more results showcasing their method.\n\nSince the rebuttal does not allow for more qualitative results, I am rejecting the paper at this stage. \n\nThe method, while pretty good, probably needs a little more work to get it working for the subject + camera motion case. I recommend that the authors continue working on it to strengthen the paper.\n\nPost rebuttal: it would have been nice to see more qualitative results and videos, but i understand that is not possible in the 1 page rebuttal pdf. Based on the submitted rebuttal, I am happy to increase the score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of how to control the identity and the motion of the subject while generating video, in a text-to-video setup. Specifically, in addition to the standard text prompt, it allows the user to control: the subject's identity by providing a few images of it (e.g. 5 photos of my dog); the subject's position in the video, as provided by a bounding box sequence; and the camera motion, as provided by a sequence of delta-x & delta-y.\n\nIdentity preservation is achieved by two changes to the loss function, during fine-tuning on images of the subject. First a \"subject region loss\" zeros out gradients that lie outside the subject's bounding box during fine-tuning, and second a video preservation loss (similar to the class-specific preservation loss introduced by DreamBooth) to ensures that the ability to generate dynamic motion is not lost due to over-fitting on a few static images. \n\nSubject and camera motion are controlled at inference time, by modifying the cross-attentional map between the prompt and the latent feature, by shifting the latent feature, respectively.\n\nFinally, a third loss \"subject token cross-attention\" links the two, enabling the inference motion controls by ensuring that the cross-attention between the subject token (in the text prompt) and the diffusion model's U-Net features is strong where the object should be and weak elsewhere.\n\nThese proposed changes are validated quantitatively through several metrics: CLIP and DINO scores between the source images and the desired video locations of the subject, CLIP score between the text prompt and video frames, and a difference in optical flow between the generated video and ideal video (according to the subject and camera motion prompts).\n\nExamples of generated videos are included in the supplemental.\n\nSTRENGTHS:\nThis paper addresses a very significant and timely problem -- how to effect significantly more fine-grained control in text to video models, and allow them to be more than a novelty.\n\nThe results, as shown in the supplemental video, are very impressive: the subject identity, subject motion, and camera motions very much follow the user's input.\n\nThe additional loss terms are intuitively sensible and effective. The inference modifications, although they seem a bit \"hacky\" (if you'll excuse me), do get the job done quite effectively.\n\nI appreciate that the transferability of the method was demonstrated by implementing it on two different text-to-video models.\n\nWEAKNESSES:\nThese are relatively minor weaknesses in my opinion:\n\nThis region metrics measure whether the desired object is translating around the frame as per the bounding box guidance, but because they are image metrics, can't evaluate the actual \"animation quality\" of the subject. I believe the entire suite of metrics could be optimized by a static image crop of the subject sliding across the video frame according to bounding box motion and flow. Can you confirm whether this is correct?\n\nThe set of camera motions that can be models is limited to 2d translations in x and y, missing zooms, rotations, etc.\n\nAlthough this paper presents its techniques as transferable across many latent diffusion models, they require that the \"latent can be considered a \"shrunk image,\" which maintains the same visual geographic distribution as the generated images.\"  This should generally be true for most latent video diffusion models, but may not work with new latent encoders being developed.\n\nQUESTIONS:\nRe \"Subject Region Loss\": How certain are you that the binary mask, when applied to the latent (rather than the pixels), actually does mask out the background? Given that the latent is produced by a stack of many convolutional or transformer layers, the receptive fields of each latent location almost certainly include information from the nearby background. Have you tried masking the training images directly in pixel space, rather than masking the diffusion loss? How does that compare?\n\nRe \"Ablation studies\": w/o video and w/ class video give quite-nearly identical numbers. Is this a copy-paste error? If not, it's worth discussing in more detail.\n\nIn Figure 3, what is the difference between c, d, e, and f? Are they different samples from the model, or from different models?\n\nHave you tried to achieve the desired camera motions via the text prompt, and considered that as a baseline? I have seen it work well for many basic types of motions in some recent t2v models that you cite.\n\nLIMITATIONS:\nThe limitations described in the appendix are important enough that they should be included in the main paper.\n\nOtherwise, yes, they have addressed limitations and impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper primarily addresses the problem of motion-aware customized video generation. Traditional customized generation methods either tend to lose the motion information of the video or require additional training of motion modules. To enhance subject fidelity and video dynamics, this paper introduces subject region loss and video preservation loss. Additionally, a training-free subject- and camera-motion control method is proposed, enabling the model to flexibly control the motion information of the video. Extensive experiments demonstrate that the proposed method outperforms existing state-of-the-art methods.\n\nSTRENGTHS:\n1.\tThis paper achieves better automized generation results than the state-of-the-method\n2.\tThe writing is clear and easy to follow\n\nWEAKNESSES:\n1.\tSome details need to be added to more clearly distinguish the differences. For example, in line 169, existing methods use class-specific data, whereas this paper uses common videos. Do the common videos need to include customized subjects? How are these common videos obtained?\n2.\tThe assumptions made by the method, such as sampling operations on the x- and y-axes when controlling camera motion, may limit its effectiveness. For instance, in scenarios with significant changes from left to right or top to bottom, or in close-up videos with large foreground objects, this operation may result in uncoordinated outcomes. However, the paper does not seem to discuss these limitations.\n3.\tMethods that control subject and camera motion through object trajectories and camera parameters have also been proposed, such as motionCtrl, TrailBlazer, Boximator, and motion-zero. However, this paper does not provide an in-depth discussion or experimental comparison with these methods. Although some of these methods are training-based, it is still worthwhile to explore the existing gaps.\n\nQUESTIONS:\n1.\tHow many subject samples and/or common videos are used for training each customized model?\n2.\tThe proposed mainly explores the camera motion on panning left or right, is it possible to control other motion types, like zoom-in/out effects? \n3.\twhat is the difference between the proposed subject region loss, subject token cross attention loss, and those losses in Mix-of-show [1] for customized image generation\nref:\n[1] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models\n\nLIMITATIONS:\n1. This paper needs to discuss more methods addressing the same tasks, like motion control, and specify the differences between the proposed loss functions and those of existing methods.\n2. More limitations or failure cases needed to be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets at animating customized subjects with both motion and camera control. To achieve that, they first customize t2v models with subject images finetuning without hurting video generation capability. Then, they propose a training-free approach to manage subject and camera motions. Extensive experiments demonstrate the superiority of their method.\n\nSTRENGTHS:\n•\tThe paper introduces the first unified video generation framework for subject customization with motion and camera movement control, which is also an interesting problem setup.\n\n•\tThe paper proposes a new subject region loss and video preservation loss for video customization, and a training-free method to control both subject motion and camera motion.\n\n•\tThe paper is well-written with clear motivations.\n\nWEAKNESSES:\n•\tThe subject motion control module is interesting by directly amplifying in the bounding box region and suppressing outside the box region. However, how to define the value of parameter alpha remains unclear. Is it only defined based on experimental performance? And does the authors keep this alpha parameter the same for all customization experiments?\n\n•\tThe proposed method only control the camera movement in 2D such as up, down. However, camera movement is quite important for 3D, such as moving into the scene, rotation, etc. The proposed camera control module is not able to handle such camera motions and can be a weakness for camera control.\n\nQUESTIONS:\n•\tFor subject motion control module, if the bounding box size is changing, e.g., gradually enlarging bounding box, will the method enlarge the subject accordingly?\n\n•\tSubject motion is composed of both position translation and subject movements, such as walking, dancing, jumping, etc. The paper only control the subject position for subject motion. The reviewer wonders is it a bit overclaiming for controlling subject motions, since the subject movements cannot be customized or controlled?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}