MotionBooth: Motion-Aware Customized Text-to-Video Generation
Authors: Jianzong Wu
Xiangtai Li
Yanhong Zeng
Jiangning Zhang
Qianyu Zhou
Yining Li
Yunhai Tong
Kai Chen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: custimized generation
video motion generation
Primary area: diffusion_based_models

Abstract: In this work, we present MotionBooth, an innovative framework designed for animating customized subjects with precise control over both object and camera movements. By leveraging a few images of a specific object, we efficiently fine-tune a text-to-video model to capture the object's shape and attributes accurately. Our approach presents subject region loss and video preservation loss to enhance the subject's learning performance, along with a subject token cross-attention loss to integrate the customized subject with motion control signals. Additionally, we propose training-free techniques for managing subject and camera motions during inference. In particular, we utilize cross-attention map manipulation to govern subject motion and introduce a novel latent shift module for camera movement control as well. MotionBooth excels in preserving the appearance of subjects while simultaneously controlling the motions in generated videos. Extensive quantitative and qualitative evaluations demonstrate the superiority and effectiveness of our method. Models and codes will be made publicly available.

MotionBooth: Motion-Aware Customized Text-to-Video Generation
Jianzong Wu Xiangtai Li2.3
Yanhong Zeng", Jiangning Zhang', Qianyu Zhou²,
Yining Lis, Kai Chen Yunhai Tong*
PKU LS-Lab, NTU Shanghai Al Laboratory *ZJU SSTUU
Project Page: https: innnzongg/ anzongwu.
gi thub. proboioo/ooooeeeooooee jttp://s.eeeeeeeooeoooootsstsses //actts/mooia tionbooth
Code: https: / gi thub. waaaioagggg///ooiooonnao
Email: jzwu stu.pku. st..pu.edu.co edu. xiangtai94@ oe
Subject
Subject mation
Camera motion
Generated Videos
jamping
wonsster boy wallzing Squars
jumpking SherSSeewa
Right
Sartosw menninn Ca palxting forset
Figure 1: Motion-aware customized video generation results of MotionBooth. Our method animates customized object with controllable subject and camera motions.
Abstract
In this work, present MotionBooth, innovative framework designed for animating customized subjects with precise control over both object and camera movements. By leveraging a few images of specific object, we efficiently fine- tune text-to-video model to capture the object's 's shape and attributes accurately. Our approach presents subject region loss and video preservation loss to enhance the subject's learning performance, along with a subject token cross-attention loss integrate the customized subject with motion control signals. Additionally, we propose training-free techniques for managing subject and camera motions during inference. In particular, we utilize cross-attention map manipulation to govern subject motion and introduce novel latent shift module for camera movement control as well. MotionBooth excels in preserving the appearance of subjects while simultaneously controlling the motions in generated videos. Extensive quantitative and qualitative evaluations demonstrate the superiority and effectiveness of our method. Models and codes will be made publicly available.
Work done when Jianzong is an intern at Shanghai Al Laboratory. t: Project Lead. Corresponding: Xiangtai Li and Yunhai Tong.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Generating videos for customized subjects, such as specific scenarios involving particular dog's type or appearance, has gained research attention [55, 25, 42]. This customized generation field originated from text-to-image (T2I) generation methods, which learn a subject's appearance from few images and generate diverse images of that subject [12, 42, 30J. Following them, subject-driven text-to-video (T2V) generation has seen increasing interest, which has found wide range of applications in personal shorts or film production [55, 25, 42, 57, 13]. Can you imagine your toy riding along the road from a distance to the camera or your pet dog dancing the street from the left to the right? However, rendering such lovely imaginary videos is a challenging task. It often involves subject learning and motion injection while maintaining the generative capability to generate diverse scenes. Notably, VideoBooth [25] trains an image encoder to embed the subject's appearance into the model, generating short clip of the subject. However, the generated videos often display minimal missing motion, resembling a "moving image. "This approach underutilizes the motion diversity of pre-trained T2V models. Another line of works [57, 61, 13] fine-tunes the customized model specific videos, requiring motion learning for each specific camera or subject motion type. Their pipelines restrict the type of motion and require fine-tuning new adapter for each motion type, which is inconvenient and computationally expensive.
The key lies in the conflict between subject learning and video motion preservation. During subject learning, training on limited images of the specific subject significantly shifts the distribution of the base T2V model, leading to significant degradation (e.g., blurred backgrounds and static video). Therefore, existing methods often need additional motion learning for specific motion control. In this paper, we argue that the base T2V model already has diverse motion prior, and the key preserve video capability during subject learning and digging out the motion control during inference.
To ensure subject-driven video generation with universal and precise motion control, we present Mo- tionBooth, which can perform motion-aware customized video generation. The videos generated by MotionBooth are illustrated in Fig. 1. MotionBooth can take any combination of subject, subject motion, and camera motion as inputs and generate diverse videos, maintaining quality on par with pre-trained T2V models.
learns subjects without hurting video generation capability, enabling training- free motion injection for subject-driven video generation. First, during subject learning, we introduce subject region loss and video preservation loss, which enhance both subject fidelity and video quality. In addition, we present a subject token cross-attention loss to connect the customized subject with motion control signals. During inference, we propose training- ffre techniques to control the camera and subject motion. We directly manipulate the cross-attention maps to control the subject motion. We also propose a novel latent shift module to govern the camera movement. It shifts the noised latent to move the camera pose. Through quantitative and qualitative experiments, we demonstrate the superiority and effectiveness of the proposed motion control methods, and they can be applied to different base T2V models without further tuning.
MotionBooth
Our contributions are summarized as follows: 1) We propose a unified framework, MotionBooth, for motion-aware customized video generation. To our knowledge, this is the first framework capable of generating diverse videos by combining customized subjects, subject motions, and camera movements as input. 2) We propose novel loss-augmented training architecture for subject learning. This includes subject region loss, video preservation loss, and subject token cross-attention loss, significantly enhancing subject fidelity and video quality. 3) We develop innovative, training-fiee methods for controlling subject and camera motions. Extensive experiments demonstrate that MotionBooth outperforms existing tfate-ofthh--an video generation models.
2 Related Work
Text-to-video generation. T2V generation leverages deep learning models to interpret text input and generate corresponding video content. It builds upon earlier breakthroughs in text-to-image generation [44, 41, 19, 21, 37, 48, 60, 62] but introduces more complex dynamics by incorporating motion and time [46, 20, 18, 2, 68, 59]. Recent advancements particularly leverage diffusion- based architectures. Notable models such as ModelScope [51] and La [54] integrate temporal layers within spatial frameworks. VidcoCrafterl [6] and VideoCrafter2 [7] address the scarcity of video
2

---

Training
Inference
Vidro Preservation Loss
[V] dog"
EVI dog naxning
inpuss
Carsoa Dracter
Rouid Rix
Sequancc
Cioua Aith
Coop ARn
Cooas Amh
irugefacee Cutza
Siited
d, 1
Latent Shift
Eenvexted Video
Camera Foom
imputtmagg
Fection
Entract or osi antercion maps A [V] dos
Subject Mocio Control
cdtart
DV
60gg
Subject Reglon Lost
Subject Tokrn
Change
Cross Attinsiinonoonn Loss Liten
Trainable
amplity
suppress
Sutject Masic
C/OSS Attn Map
Figure 2: The overall pipeline of MotionBooth. We first fine-tune T2V model on the subject. This procedure incorporates subject region loss, video preservation loss, and subject token cross-attention loss. During inference, we control the camera movement with a novel latent shift modulc. At the same time, we manipulate the cross-attention maps to govern the subject motion.
data by utilizing high-quality image datasets. Latte [36] and W.A.L.T [14] adopt Transformers as backbones [49]. VideoPoet [29] explores generating videos autoregressively to produce consistent long videos. Recent Sora [3] excels in generating videos with impressive quality, stable consistency, and varied motion. Despite these advancements, controlling video content through text alone remains challenging. highlighting continuing need for research into more refined control signals.
Customized generation. Generating images and videos with customized subjects is attracting growing interest. Most works concentrate learning specific subject with few images from the same subject [23, 8, 10, 43, 47, 39] or specific domains [15, 16, 50]. Textual Inversion [12] proposes train new word capture the feature of object. In contrast, DreamBooth [42] fine-tunes the whole U-Net, resulting in a better IP preservation ability. Following them, many works explore more challenging tasks, such customizing multiple objects [30, 55, 33, 5], developing common subject adapter (58, 25, 65, 11, 67], and simultaneously controlling their positions [11, 33J. However, the customization of video models from a few images often results in overfitting. The models fail to incorporate significant motion dynamics. A recent work, Dream Video [57]. addresses this by learning specific motion types from video data. Yet, this method is restricted to pre-defined motion types and lacks the flexibility of text-driven input. In contrast, our work introduces MotionBooth to control both the subject and camera motions without needing pre-defined motion prototypes.
Motion-aware video generation. Recent works explore incorporating explicit motion control in video generation. This includes camera and object motions. To control camera motion, existing works like AnimateDiff [13], VideoComposer [53], CameraCtrl [17], Direct-A-Vides [66], and MotionCtrl [56] design specific modules encode the camera movement trajectory. These models usually rely on training on large-scale datasets [1, 9], leading to high computational costs. In contrast, our MotionBooth framework builds training-frre camera motion module that can be casily integrated with any T2V model, climinating the need for re-training. For object motion control, recent works [63, 31, 32, 24, 66, 4, 69, 27, 22] propose effective methods to manipulate attention values during the inference stage. Inspired by these approaches, we connect subject text tokens to the subject position using subject token cross-atee coss-atttetion loss. This allows for straightforward control over the motion of a customized object by adjusting cross-attention values.
3 Method
3.1 Overview
Task formulation. We focus generating motion-aware videos featured by customized subject. To customize video subjects, we fine-tune the T2V model on specific subject. This process can be accomplished with just few (typically 3-5) images of the same subject. During inference, the fine-tuned model generates motion-aware videos of the subject. The motion encompasses both camera and subject movements, which are freely defined by the user. For camera motion, the user
3

---

inputs the horizontal and vertical camera movement ratios, denoted as Ceam = [Cz, Cyl. For subject motion, the user provides a bounding box sequence [B, Bg.... [
Br,] r.] indicate the desired positions of the subject, where I. represents the video length. Each bounding box specifies the x-y coordinates of the top-left and bottom-right points for each frame. By incorporating these conditional inputs, the model is expected to generate videos that include specific subject, along with predefined camera movements and subject motions.
Overall pipeline. The overall pipeline of MotionBooth is illustrated in Fig. 2. During the training stage, MotionBooth learns the appearance of the given subject by fine-tuning the T2V model. To prevent overfitting, we introduce video preservation loss and subject region loss in Section 3.2. Additionally, we propose subject token cross-attention (STCA) loss in Section 3.2 to explicitly connect the subject tokens with the subject's 's position on cross- cross-attention maps, facilitating the control of subject motion. Camera and subject motion control are performed during the inference stage. We manipulate the cross-attention maps by amplifying the subject tokens and their corresponding regions while suppressing other tokens Section 3.3. This ensures that the generated subjects appear in the desired positions. By training on the cross-attention map, the STCA loss enhances the subjects' motion control. For camera movement, we introduce a novel latent shift module to shift the noised latent directly, achieving smooth camera movement in the generated videos in Section 3.4.
3.2 Subject Learning
Given few images of a subject, previous works have demonstrated that fine-tuning a diffusion model on these images can effectively learn the
dag munaling on grass
DVd runniig S4 the grass
appearance of the subject [42, 23, 8, 10, 43, 47]. However, two significant challenges remain. First, due to the limited size of the dataset, the model quickly overfits the input images, includ- ing their backgrounds, within a few steps. This overfitting of the background impedes the gen- eration of videos with diverse scenes, a problem also noted in previous works [42, 12]. Second, fine-tuning T2V models using images can impair the model's s inherent ability to generate videos, leading to severe background degradation in the generated videos. To illustrate these issues, we conducted a toy experiment, As depicted in Fig. 3, without any modifications, the model overfits the background the subject image. To address this, we propose computing the diffusion reconstruction loss solely within the subject region. However, even with this adjustment, the background in the generated videos remains over-smoothed. This degradation likely results from tuning a T2V model exclusively with images, which damages the model's 's original weights for video generation. To mitigate this, we propose incorporating video data as preservation data during the training process. Although training with video data but without subject region loss still suffers from overfitting, our approach, MotionBooth, can generate videos with detailed and diverse backgrounds.
Jal Pre-rr ned boo to des
Ic) Regio X Video X
Jes Region / Video X
1e1 X Video
Video V
Figure 3: Case study on subject learning. "Region" " indicates subject region loss. "Video" indicates video preservation loss. The images are extracted from generated videos.
Preliminary. T2V diffusion models learn to generate videos by reconstructing noise in a latent space [42, 30, 55, 12]. The input video is first encoded into a latent representation Zg. Noise is added to this latent representation, resulting in a noised latent 2,, where represents the timestamp. This process simulates the reverse process of a fixed-length Markov Chain [41]. The diffusion model eg is trained to predict this noise. The training loss, which is reconstruction loss, is given by:
F... [[k-- co c, 1)
where e is the conditional input used in classifier-free guidance methods, which can be text or a reference image. During inference, a pure noise Zy is gradually denoised to a clean latent 2/1 where T is the length of the Markov Chain. The clean latent then decoded back into RGB space to generate the video X'.
Subject region loss. To address the challenge of overfitting backgrounds in training images, we propose region loss. The core idea is to calculate the diffusion reconstruction loss exclusively

---

within the subject region, thereby preventing the model from learning the background. Specifically, we first extract the subject mask for each image. This can be done manually or through automatic methods, such as segmentation model. In practice, we use SAM [28] to collect all the masks. The subject region loss is then calculated as follows:
Leub = No,...
C,, M
where M represents the binary masks for the training images. These masks resized the latent space to compute the dot product. C; is a fixed sentence in the format "a [V] [class name]," where "[V]" is rare token and "[class name]" is the class name of the subject [42]. We have found that with the subject region loss, the trained model effectively avoids the background overfitting problem.
Video preservation loss. Image customization datasets like DrcamBooth [42] and CustomDiffu- sion [30] provide excellent examples of multiple images from the same subject. However, in the customized video generation task, directly fine-tuning the video diffusion model on images leads to significant background degradation. Intuitively, this image-based training process may harm the original knowledge embedded in video diffusion models. To address this, introduce video preservation loss designed to maintain video generation knowledge by joint training with video data. Unlike the class-specific preservation data used in previous works [42, 55J, we utilize common videos with captions denoted as C.. Our experiments in Section 4 demonstrate that common videos are more effective for subject learning and preserving video generation capabilities. The loss function is formulated as follows:
Lvid =
[lle- - co (Ze, Cu, t)1
(3)
Subject token cross-attention loss. To control the sub- ject's motion, we directly manipulate the tross-attention maps during inference. Since we introduce a unique token, "[V]", in the training stage and associate it with the subject, we need to link this special token to the subject's 's position within the cross-attention maps. As illustrated in Fig. 4, fine-tuning the model does not effectively connect the unique token the cross-attention maps. Therefore, we propose a Subject Token Attentntttntion (STCA) loss to guide this process explicitly. First, we extract the cross- attention map. A, at the tokens "[V] [class name]". We then apply Binary Cross-Entropp. Loss to ensure that the corresponding attention map is larger at the subject's 's position and smaller outside this region. This process incorporates the subject mask and can be expressed as:
(a) Input Images
(b) w/o loss
(c) w/ STCA loss
Figure 4: Case study on subject token cross-attention maps. (b) and (c) vi- sualization of cross-attention maps on tokens "[V]" and "dog".
Estea - log(( A) + (1 M) log(1 - A)].
(4)
During training, the overall loss function is defined as:
L = Leub + Au + AgCstca:
(5)
where D and A2 are hyperparameters that control the weights of the different loss components.
3.3 Subject Motion Control
We chose bounding boxes as the motion control signal for subjects because they are easy to draw and manipulate. In contrast, providing object masks for every frame is aabor-intensive, requiring consideration of the subject's shape transformation between frames. In practice, we find that bounding boxes are sufficient for precisely controlling the positions of subjects. Previous works like GLIGEN [31] attempt to control object positions by training extra condition module with large-scale image data. However, these training methods fix the models and cannot easily align with customized models fine-tuned for specific subjects. Therefore, adopt alternative approach that
5

---

directly edits the cross atss-nattnnion maps during inference in training free manner [66, 27, 4]. This cross-attention editing method plug-and aongg-and-layy and can be used with any customized model.
In cross-attention layers, the query features Q are extracted from the video latent and represent the vision features. The key and value features K and V are derived from input language tokens. The calculation process of the edited cross-attention layer can be formulated as follows:
QK vd
EditedCrossAttn(( (Q, K, V) =Softmx
+ as V.
(6)
where di is the feature dimension of Q and serves as a normalization term.
va is the normalized production between Q and K, representing the attention scores between vision and language features. We manipulate the production by adding a new term aS, where S has positive values on the subject region provided in bounding boxes and large negative values outside the desired positions. is hyperparameter to control the editing strength. The editing matrix S is set as follows:
QK
Byl
if E Bş and j E P and 2 if E Bg and j E and < otherwise
S.(i. =
0,
where i, j. and & indicate the vision token, language token, and frame indexes, respectively. P represents the indexes for subject language tokens in the text prompt. this work, we choose "[V]" and "[class name]" as subject tokens. The SCTA loss in Section 3.2 binds the two tokens with cross-attention maps. is the denoising timestamp and J is hyperparameter defining timestamp threshold. Since diffusion models tend to form the approximate object layout in earlier denoising steps and refine the details later steps [63], we apply stronger attention amplification in earlier steps and amplification in later steps. Note that attention suppression outside the bounding box regions persists throughout the generation. [B&] and [Q are the areas of the box and query, respectively. Following previous works [27, 66], smaller boxes should have larger amplifications, and we do not apply any editing on the <start> and <end> tokens.
Discussion. An important aspect is how the necessary information from other language tokens is integrated into the generated outputs, given that tokens such as verbs and background nouns are assigned minimal values. We propose that this information is extracted through the <start> and <end> tokens. Given that Transformer-baat based language encoders like CLIP [40] are typically trained on classification tasks, they often encode the overall context of sentence into these special tokens. Thus, despite the suppression of other tokens during the softmax calculation, the model can still access relevant information about verbs, background elements, and other components necessary for the generation process. To support this explanation, we conducted an experiment in which we examined the softmax outputs using naive text-to-video pipeline. The results showed that the <start> token consistently held the highest softmax value, close to 1, while the <end> token had the second-largest value. The remaining tokens, including those representing nouns, adjectives, verbs, and conjunctions, were distributed among the remaining softmax values.
3.4 Camera Movement Control
Simply editing the cross-attention map can effi- ciently control the motion of the subject. This suggests that the latent can be considered "shrunk image," which maintains the same vi- sual geographic distribution the generated im- ages. For camera movement control, an intuitive approach is to directly shift the noised latent during inference based on the camera movement signal Coom = (cz, C.). The latent shift pipeline is illustrated in Table 2. The key challenge with this idea is filling the missing parts caused by the latent shift (the question mark region in Step
vistorilli bolutifu full fslage
Step L:Sett
& Missing
Movemant
Stitted Lanent Say
Figure Illustration of camera movement control through shifting the noised latent.

---

1). To address this issue, we propose sampling tokens from the original noised latent and using them to fill the gap. This is based on the prior knowledge that when a camera moves in video, the new scene it captures is semantically close to the previous one. For example, a video with forest scenes, when the camera pans left, it is highly likely to capture more trees similar to those in the original scene. Another assumption is that in normally angled video, a visual element is more likely to be semantically close to elements along the same x-axis or y-axis rather than other elements. For instance, in the waterfall video in Fig. 5, trees are at the top and bottom, spreading horizontally, while the waterfall spans the middle x-axis area. Experimentally, we that sampling tokens horizontally and vertically provides better initialization and results in smoother video transitions. Randomly sampling tokens degrades the generated video quality. The latent shift process for timestamp t can be formulated as follows:
h SampleHorizontal(zzz B, cz),
h, = SampleVertical(z, B, Cy),
(8)
Zahift Crop(Shift(z Ca. Gy)),
Z = Fill (zhiit. hg by: Cas Cy):
where h2 and hy are sampled tokens along the x and y axes, respectively. Crop(-) removes the tokens outside the camera view after the shift. B is the subject bounding box. We filter out the tokens belonging to the subjects because they are not likely to occur in the new scenes. In addition, avoid drastic change in latent in one shift, we spread the latent shift over multiple timestamps, with each step only shifting a small number of tokens. Note that the latent shift needs to be applied after the subject's ss upproximate layout is fixed but before the video details are completed. We set a pair of hyperparameters oi and 02. The latent shift only applies in the timestamp range 02].
4 Experiments
4.1 Experimental Setup
Datasets. For customization, we collect a total of 26 objects from DreamBooth [42] and CustomDif- fusion [30]. These objects include pets, plushies, toys, cartoons, and vehicles. To evaluate camera and object motion control, we built a dataset containing 40 text-object motion pairs and 40 text-camera motion pairs, ensuring that the camera and object motion patterns are consistent with the text prompts. This dataset evaluates the videos generated for each subject in various scenarios and motions.
Implementation details. We train MotionBooth for 300 steps using the Adam W optimizer, with learning rate of 5e-2 and weight decay of 1e-2. We collect 500 preservation videos from the Panda-70M [9] training set, chosen randomly. Each batch consists of one batch for images and one for videos, with batch sizes equal to the number of training images and I for images and videos, respectively. The loss weight parameters a and X are set to 1.0 and 0.01. We use Zeroscope and La as base models. During inference, we perform 50-step denoising using the DDIM scheduler and set the classifier-fres guidance scale to 7.5. The generated videos are 576x320x24 and 51 512x320x 16 for Zeroscope and Viee, respectively. The training process finishes in around 10 minutes single NVIDIA A A100 80G GPU. Additional implementation details can be found in Appendix A.I.
Baselines. Since we are pioneering motion-aware customized video generation, we compare our meth- ods with closely related works, including DreamBooth [42], Custom Video [55], and Dream Video [57]. Dreambooth customizes subjects for text-to-image generation. We follow its practice with class preservation images and fine-tune T2V models for generating vidcos. Custom Video is a recent video customizing method. We adopt its parameter-efficien training procedure. Dream Video learns motion patterns from video data. To provide such data, we sample videos from Panda-70M, which are most relevant to the evaluation motions. Since these methods cannot control motions during inference, we apply our camera and object motion control technologies for a fair comparison. Additionally, we compare our camera control method with training-
methods, AnimateDiff [13] and Camer- aCtrl [17], focusing on camera motion control without subject customization. Since AnimateDiff trained with only basic camera movement types and cannot take user-defined camera movement Ceam [ca, Cy] as input, we use the closest basic movement type for evaluation.
7

---

Table 1: Quantitative comparison for motion aware customized video generation.
T2V Model
Method
1-CLIP
R-DINO 1
CLIP-T
T-Cons.
Flow error
DreamBooth [42]
0.608
0.279
0.231
0.951
0.690
Custom Video [55]
0.657
0.267
0,245
0.955
0,516
Zeroscope
Dream Video [57]
0.656
0.238
0.258
0.954
0,349
Motion (Ours)
0.667
0.306
0.258
0.958
0,252
DreamBooth [42]
0.696
0.426
0.238
0.958
1,156
Custom Video [55]
0.634
0.189
0.248
0,911
1,055
LaVic
Dream [57]
0.649
0.216
0.243
0.925
0.691
MotionBooth (Ours)
0.712
0.472
0.247
0.962
0.332
Table 2: Quantitative comparison for camera movement control.
Method
Module Weight Storage
FVD J
CLIP-T
T-Cons.
Flow error
Text2Video-Zero (SD 1.5) [26]
[No Trainingl
1821.72
0.248
0.904
1.854
AnimateDiif [13]
74M
1515.82
0.245
0,925
1.683
CameraCtrl [17]
2.5G
1468.53
0.237
0.939
0.807
MotionCtrl [56]
4.0G
1109.45
0.236
0.935
0.872
MotionBooth (Zerosoope)
[No Training]
905.40
0.252
0.948
0.190
MotionBooth (LaVic)
[No Training]
723.26
0.241
0.963
0.296
Evaluation metrics. We evaluate motion-aware customized video generation from four aspects: region subject fidelity, temporal consistency, camera motion fidelity, and video quality. 1) To ensure the subject well-preserved and accurately generated the specified motion, we introduce region CLIP similarity (R (R-CLIP) and region DINO similarity metrics (R-DINO). These metrics utilize the CLIP [40] and DINOv2 [38] models to compute the similarities between the subject images and frame regions indicated by bounding boxes. Additionally, we use CLIP image-text similarity (CLIP-T) to measure the similarity between entire frames and text prompts. 2) We evaluate temporal consistency by computing CLIP image features between each consecutive frame. 3) We use VideoFlow [45] to predict the optical flow of the generated videos. Then, we calculate the flow error by comparing the predicted flow with the ground-truth camera motion provided in the evaluation dataset. 4) We randomly select 1000 videos from the MSRVTT dataset [64], predict their camera motion sequences with VideoFlow [45], and compute the FVD metric for camera motion control only.
4.2 Main Results
Quantitative results. We conduct quantitative comparisons with baseline models both motion- aware customized video generation and camera movement control. The results for motion-aware customized video generation forms all baselines on both Zeroscope and La Vae models, indicating that our proposed technologies can be extended to different T2V models. Thanks to the training- free architecture of subject and cam- era motion control methods, MotionBooth is expected to be adaptable to more open-sourced models in the future, such Sora [3]. Notably, Dream [57] achieves the second- best scores in T-Cons. and flow error, which aligns with our observation that incorporating video data as auxiliary training data enhances video generation performance. On the other hand, Custom Video [55] shows inferior performance in R-DINO scores, indicating a poorer ability to generate subjects in given positions. This may be attributed to its approach of only fine-tuning the text embeddings and cross-attention layers of the diffusion models, which is insufficient for learning the subjects.
shown in Table 1. The results demonstrate that MotionBooth outper-
For camera movement control, we compare method with two training- based methods, AnimateD- iff [13] and CameraCtrl [17]. The results are shown in Table 2. Remarkably, MotionBooth achieves superior results compared to the two baselines with our training- free latent shift module. Specifically, MotionBooth outperforms the recent method CameraCtrl by 0,617, 0.015, and 0.009 in flow error, CLIP-T, and T-Cons. metries with Zeroscope, and 0.511. 0.004, and 0.024 for the Vie model. These results demonstrate that the latent shift method is simple yet effective.
Qualitative results. The qualitative comparison results for video generation with customized objects and controlled subject motions are presented in Fig. Our observations reveal that MotionBooth excels in subject motion alignment, text prompt alignment, and overall video quality. In contrast,
8

---

Subject
Base Model Zeroscope
Base Model LaVie
DreamBooth
CustomVVdee
DreamVideo
[Ours)
salt junsping off the stelrs
IVD riding kikee the votul with view mounttinn
Figure 6: Qualitative comparison customizing objects and controlling their motions.
Dean
piagffil partng Sawars
in
Figure 7: Qualitative comparison of camera motion control, Lines and points are used to help the readers track the camera movement more easily.
Table 3: Ablation study for training technologies. "mask" means subject region loss. "STCA" means subject token cross-attention loss. "video" means video preservation loss. "w/ class video" means utilizing class-specific videos in video preservation loss. The results evaluated La
Method
R-CLIP t
R-DINO
CLIP-T
T-Cons. t
Flow error
w/o mask 0.673
0.216
0.245
0.943
0.451
w/o STCA
0.710
0.453
0.244
0.962
0,363
w/o video
0.677
0.364
0.244
0.953
0.344
w/ class video
0.677
0.364
0.244
0.953
0.345
MotionBooth 0.712
0.472
0.247
0.962
0.332
DreamBooth and Custom Video produce videos with vague backgrounds, highlighting that generated backgrounds deteriorate when training is conducted without video data. Additionally, Custom Video and Dream Video struggle to capture the subjects' appearances, likely because their approach tunes only part of the diffusion model, preventing the learning process from fully converging.
We also conduct qualitative experiments focused on camera movement control, with results shown in Fig. 7. DnimateDiff, limited to basic movements, does not support user-defined camera directions. Although the CameraCtrl method can accept user input, it generates videos with subpar aesthetics and objects that exhibit flash movements. In contrast, our MotionBooth model outperforms both the Zeroscope and Lavie models. The proposed latent method generates videos that adhere to user-defined camera movements while maintaining time consistency and high video quality.
4.3 Ablation Studies
Training technologies. We analyze the technologies proposed during the subject learning stage. The ablation results are shown in Table 3. Clearly, without the proposed modules, the quantitative

---

metrics drop accordingly. These results demonstrate that the proposed subject region loss, STCA loss, and video preservation loss are beneficial for subject learning and generating motion-aware customized videos. Specifically, the R-DINO metric decreases significantly by 0.256 without the subject region loss, highlighting its core contribution in filtering out image backgrounds during training. Additionally, the "w/ class video" experiment, which uses class-specific videos instead of randomly sampled common videos, yields worse results. This approach restricts the scenes and backgrounds in class-specife videos, hindering the models' ability to generalize effectively.
4.4 Human Preference Study
To evaluate approach to understanding preferences, we conducted a user study exper- iment. We collected 30 groups of videos gen- erated by MotionBooth and baseline methods. We then asked 7 colleagues videos based on the following criteria: subject motion alignment, camera movement alignment, subject appearance alignment, and temporal con- sistency. For each group of videos, the annota- tors selected only the best one. For the subject appearance alignment, the annotators were pro- vided with corresponding subject images. As shown Fig. 8, MotionBooth was the most preferred method across all models and evaluation aspects, particularly in subject appearance alignment. These results further demonstrate the effectiveness of our method.
Preforrme Rato [S]
$3.5
75.5
select the best
20
0.8
16
15
3/6
23
9.2
Mition Al grmment
Carrera Alggmment
Subject Al Aiggment
CamperalCmmatenny
DreamSooth
IN CustoniVidoo
Dreamiiide
Booth [Dus)
Figure 8: Human preference study, Our Motion- Booth achieves the best human preference scores in all the evaluation aspects.
4.5 Limitations and Future Work
In Fig. 9, we illustrate several failure cases of MotionBooth, One significant limitation of Motion- Booth is its struggle with generating videos involving multiple objects. As shown in Fig. 9a), the subject's appearance can sometimes merge with other objects, resulting in visually confusing outputs. This issue might be resolved by incorporating advanced training technologies for multiple subjects. Another limitation is the model's capability to depict certain motions indicated by the text prompt. As depicted in Fig. 9b)., MotionBooth may fail to accurately represent motions that are unlikely to be performed by the subject. For example, it is hard to imagine a scene where wolf plushie is riding bike. These failure cases highlight the need for further improvement in the model's s subject separation and motion un- derstanding capabilities to enhance the realism and accuracy of the generated videos. Utilizing more powerful T2V models may eliminate these drawbacks. Future work could focus refining these aspects to address the current limitations and provide more robust performance complex scenarios.
Subject
A [V] dog and kerse walkiing ew, the [o] Failure objects
Subject
pluskle viding Wiz road [b) Fallure case on hard mations
Figure 9: Failure cases of MotionBooth.
5
Conclusion
This paper introduces MotionBooth, a novel framework for motion-aware, customized video gen- eration. MotionBooth fine-tunes a T2V diffusion model to learn specific subjects, utilizing subject region loss to focus on the subject area. The training procedure incorporates video preservation data to prevent background degradation. Additionally, an STCA loss is designed to connect subject tokens with the cross-attention map. During inference, training-free technologies are proposed to control both subject and camera motion. Extensive experiments demonstrate the effectiveness and generalization ability of our method. In conclusion, MotionBooth can generate vivid videos with given subjects and controllable subject and camera motions.
10

---

Acknowledgement. This work is supported by the National Key Research and Development Program of China (No. 2023YFC3807600..
References
[1] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint vidco and image encoder for end-to-end retrieval. In /CCV, 2021. 3
[2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn. Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR,
2023. 2
[3] Tim Brooks, Bill Peebles. Connor Holmes, Yufei Guo Will DePue, Li Jing, David Schnurr. Joe Taylor. Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Sora: Creating video from
text, 2024. 3,8 8
[4] Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He. Changbo Wang. and Yang Li. Motion- zero: Zero-shot moving object control framework for diffusiionbbsed vidco generation. arXiv preprint arXiv:2401.101 2024, 3, 6, 14, 15, 16, 17
[5] Hong Chen, Xin Wang, Guanning Zeng, Yipeng Zhang, Yuwei Zhou, Feilin Han, and Wenwu Zhu. Videodreamer: Customized multi-subject text-to-video generation with disen-mix finetuning. arXiv preprint Xrv:333 1.00900, 2023. 3
[6] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023. 2
[7] Haoxin Chen. Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videvcrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint
2Xiv:2401.090477 2024, 2
[8] Hong Chen, Yipeng Zhang. Xin Wang. Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation. arXiv preprint arXiv:2305.033 000.33345..33 2023. 3.4
[9] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, wsi Chao, Byung Eun Jeon. Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming-Ilsua Yang. et al. Panda-70ms Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:2402. 0..402..9479, 19479, 2024. 3, 7
[10] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wes Wei Chang. and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. NeurIPS, 2024. 3,4
[11] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot
object-level image customization. CVPR, 2024. 3
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion.
arXiv preprint arXiv:2208. 01618, 2022. 2, 4
[13] Yuwci Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024. 2, 7,
[14] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic vidco generation with diffusion models. arXiv preprint 231 2023. 3
[15] Yue Han, Jiangning Zhang, Junwei Zhu, Xiangtai Li, Yanhao Ge. Wei Li, Chengjie Wang. Yong Liu, Xiaoming Liu, and Ying Tai. A generalist facex via learning unified facial representation. arXiv preprint arXiv:240J. 00551,, 2023. 3
[16] Yue Han, Junwei Zhu, Keke He, Xu Chen, Yanhao Ge, Wei Li, Xiangtai Li, Jiangning Zhang, Chengjie Wang. and Yong Liu. Face adapter for pre-trained diffusion models with fine-grained id and attribute control. arXiv preprint arXiv:2405.1291
2024. 3
[17] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai. Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 2441v2240..22101, 2024. 3, 7, [18] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole. Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprini arXiv:2210.02300 2022. 2
[19] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In NeurlPS, 2022.
[21] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICMI., 2023. 2
[22] Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang. Jieyu Weng. Yabiao Wang. and Lizhuang Ma. Motionmaster: Training- free camera motion transfer for video generation. arXiv preprint 20X7:2404.11789, 2024.
11

---

[23] Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, and Qian He. Dreamtuner: Single image is enough for subject-driven generation. arXiv preprint arXiv:2312. 13691, 2023. 3,4
[24] Yash Jain, Anshul Nasery. Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via
masked-diffusion. arXiv preprint urXiv:2312.07509, 2023. 3
[25] Yuming Jiang. Tianxing Wu, Shuai Yang. Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei
Liu. Vidcobooth: Diffusion- based video generation with image prompts. In CVPR, 2024. 2,
[26] Levon Khachatryan, Andramik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2videozerro: Text-to-image diffusion models are zero-shot
video generators. In ICCV, 2023. 8, 14, 16, 18
[27] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Wug Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In /CCV, 2023. 3,6
[28] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete
Xiao. Spencer Whitehea,, Alexander C Berg, Wan Lo, el al. Segment anything. arXiv preprint arXiv:2304.00 02643, 2023. 5
[29] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoct: A large language model for zero-shot
vidco generation. arXiv preprint arXiv: 14/25, 2023. 3
[30] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Zhu, Multi-concept customization of text-to-image diffusion. In CVPR, 2023. 2, 3, 4, 5, 7, 14
[31] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuam Li, and Yong Jac Lec. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023. 3,5 5
[32] Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene vidco generation via Ilm-guided planning. arXiv preprint arXiv:2309.15000 2023.
[33] Zhiheng Liu, Yifei Zhang. Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng. Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones 2: Customizable image synthesis with multiple subjects. arXiv preprint
arXiv:2305. 193277, 2023. 3
[34] Wan-Duo Kurt Ma, J. P. Lewis, and W. Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based
vidco generation. arXiv preprint arXIv:2401.000 00096, 2024, 14, 16, 17
[35] Wan-Duo Kurt Ma, P. Lewis, Avisck Lahiri, Thomas Leung, and W. Bastiaan Kleijn. Directed diffusion:
Direct control of object placement through attention guidance. In AAAI, 2024, 14, 15, 16, 17
[36] Xin Ma. Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint -xXv:2441.00044 2024.
[37] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In
JCML, 2021.
[38] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 077993,,, 2023. 8
[39] Xu Peng. Junwei Zhu, Boyuan Jiang. Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin. Chengjie Wang, and Rongrong Ji. Portraitbooth: A versatile portrait model for fast Aeentity-presenvy personalization. arXiv preprint arXiv:2312 06354, 2023. 3
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CCV, 2021. 6, 8
[41] Robin Rombach, Andreas Blatmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022. 2,
[42] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 2, 4, 5,
7, 7,8,1 14
[43] Nataniel Ruiz. Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image -maae models. arXiv preprint arXiv:2307.0m 064429307.069499, 2023. 3,4
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. NeurlPS, 2022. 2
[45] Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. In ICCV, 2023. 8
[46] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-vided Text-to- tovio generation without vixt-video data. arXiv preprint arXiv:2209. 14792, 2022. 2
[47] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusson with corar. arXiv preprint arXiv:2304. 06027, 2023, 3.4
12

---

[48] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint
arXiv:2010.02500 2020. 2
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NewrlPS, 2017. 3
[50] Chaoyang Wang. Xiangtai Li, Lu Qi. Henghui Ding, Yunhai Tong. and Ming- Yang. Semflow: Binding semantic segmentation and image synthesis via rectified flow. arXiv preprint arXiv:2405.20288 2024. 3
[51] Jiuniu Wang. Hangjie Yuan. Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang, Modelscope
text-to-video technical report. arXiv preprint arXiv:2308.066 065.8,657111.. 2023. 2
[52] Jiawei Wang. Yuchen Zhang, Jiaxin Zou, Yan Zeng. Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint urXIv:2402.01 0266..000..6...0. 2024. 14,
15
[53] Xiang Wang. Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. NeurIPS, 2024.
[54] Yaohui Wang. Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang. Ceyuan Yang. Yinan
He. Jiashuo Yu, Peiqing Yang. et al. Lavie: High-quality video generation with cascaded latent diffusion
models, arXiv preprint arXiv:2309.15 15/03, 2023. 2, 16
[55] Zhao Wang, Aoxue Li, Enze Xie, Lingting Zhu, Yong Guo, Qi Dou, and Zhenguo Li. Customvideo: Customizing text-to-vidoo generation with multiple subjects. arXiv preprint arXiv:2401.0980 09992,, 2024, 2, 3,
4, 5, 7, 8
[56] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for vidco generation. arXiv preprint arXiv:2312.03641,
2023. 3, 8, 14, 16
[57] Yujic Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream vidcos with customized subject and motion. arXiv preprint arXiv:2312.0444
2023. 2, 7,8
[58] Yuxiang Wei, Yabo Zhang. Zhilong Ji, Jinfeng Bai, Lci Zhang, and Wangmeng Zuo. Elite: Encoding visual
concepts into textual embeddings for customized text-to-image generation. In ICCV, 2023. 3
[59]
Jianzong Wu, Xiangtai Li, Chenyang Si, Shangehen Zhou, Jingkang Yang. Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, al. Towards language-driven vidco inpainting via multimodal large language models. CVPR, 2024. 2
[60] Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, and Dacheng Tao. Towards open vocabulary lcarning: A survey. T-PAMI, 2024. 2
[61] Jay Zhangjic Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufci Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuming of image diffusion models for text-to-video generation. In ICCV, 2023. 2
[62] Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Mosaicfusion: Diffusion models as data augmenters for large vocabulary instance segmentation. arXiv preprint arXiv:2309. 13042,
2023. 2
[63] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV, 2023. 3, 6 [64] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vit: A large video description dataset for bridging video and language. In CVPR, 2016. 8
[65] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar- based image editing with diffusion models. In CVPR, 2023. 3
[66] Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. arXiv preprint arXiv:2402.03: 122. 2024. 3, 6
[67] Haoyu Zhao, Tianyi Lu, Jiaxi Gu, Xing Zhang, Zuxuan Wu, Hang Xu, and Yu-Gang Jiang. Videoassem-
bler: clentity-consistent video generation with reference entities using diffusion model. arXiv preprint arXiv:2311.
2023.
[68] Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11012 2022.
[69] Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self-attention for long-range image and video generation. arXiv preprint
004444 2024. 3
13

---

Figure 10: The evaluation dataset. We present picture for each subject.
A
Appendix
Overview. The supplementary includes the following sections:
Appendix A.1. Implementation details of the experiments.
Appendix A.2. Discussions about our method and the differences with related methods.
Appendix A.3. Comparison with more baselines.
Appendix A.4. Ablation studies.
Appendix A.5. Social impacts.
Appendix A.6. More qualitative results.
Video Demo. We also present video in a separate supplementary file, which shows the results in video format.
A.1 Implementation Details
Hyperparameters. For the LaVie model, we set a 0 0.STT, and d2 = 0.6T. For the Zeroscope model, we a 10.0, 0.9T, σ1 0.9T, and 02 0.7T.
Evaluation dataset. collect a total of 26 subjects from DreamBooth [42] and CustomDiffu- sion [30]. We show one image for each subject in Fig. 10. The subjects contain a wide variety of types, including pets, plushie toys, cartoons, and vehicles, which can provide us with a thorough analysis of the model's effectiveness.
User study interface. We show the application interface for human preference study in Fig. 11. During user study, we ask the annotators to se- lect the best video based the question, e.g., "Which video do you think has the best temporal consistency?"
Use Study
Pseudo-code of latent shift. To present the Figure 1:: The application interface for user study. latent shift module more clearly, we show the pseudo-code of the algorithm Fig. 14. Our latent shift module can control the camera movement in videos in training-free manner at minimal costs.
A.2 Discussions
Differences with related works. Our proposed subject and camera motion control methods dif- fer several ways from previously established approaches, such as TrailBlazer [34], Directed Diffusion [35], Boximator [52], Motion-Zero [4], MotionCtrl [56], and Text2Video-Zero [26].
TrailBlazer [34] and Directed Diffusion [35] use training- approach for controlling object motion by manipulating the cross-attention maps. Specifically, TrailBlazer [34] adjusts both spatial and
14

---

Fome
Frome 12
Fraire
(d) Example of combining zoom-in and camera mo- tion control.
(a) Examples of enlarging shrinking the subject bounding box.
Supject
cemmnna spred Down Let
[v] naxinn grass
Zx mora speed
(c) Example of masking images during training.
carmrria spoed (fall)
Frame
(b) Examples of gradually enlarging the camera mo- tion speed.
Stright [Dunl
Frame
Frame
Frame 16
orifting
(c) Examples of camera motion control with large foreground objects.
(f) Ablations the filling method latent shift.
Tri aclacce
Apaeftd fotabing foress
(g) Examples of controlling both subject and camera (h) Comparison of latent shift and text guidance to control camera motion.
motion.
Figure 12: More qualitative results.
temporal cross-attention by scaling the attention maps with a hyper-parameter less than Directed Diffusion [35] focuses image generation in similar manner. Our approach, in contrast, targets only the spatial cross-attention, setting attention values outside the object's bounding box to zero. This targeted manipulation simplifies implementation and significantly enhances the performance in generating motion-aware customized videos.
Boximator [52] relies training- based technique, requiring box coordinates as inputs to newly trained self-attention layer. In comparison, our method training-frree thus providing a more accessible and user-friendly solution for controlling subject and camera motion.
Motion-Zero [4] also operates without additional training, but it utilizes test-time tuning technique to adjust the latent space using cross-attentis map loss during the denoising process. This technique, however, increases the generation time and memory requirements considerably from approximately 15 seconds to several minutes video. Additionally, our experiments indicate that Motion- often produces poor-quality outputs, with videos collapsing into unrecognizable elements. This outcome is likely due to the detrimental effects of test-time tuning parameters and latent distributions in customized scenarios. In contrast, our method directly manipulates the cross-attention map, adding only 0.3 seconds to the generation process and consistently yielding more reliable visual outcomes.
15

---

MotionCtrl [56] uses training-basse -ased approach that requires inputting point trajectories to control camera poses and object motion. Our method does not involve additional training, offering simpler alternative for subject and camera motion control.
Texi2Viieo-ZZen [26] builds on pre-trained text-to-image (T2I) model and extends it to video generation by utilizing consistent noise across frames. However, this approach is unsuitable for text-to-video (T2V) models, which typically use different noise for each frame. Additionally, Texi2Video-Zero uses latent shifting for overall scene movement and mirrors latent information to fill in missing regions. In comparison, our approach employs random sampling along the x and y axes. resulting in more concise and coherent video generation for camera motion control.
We conducted quantitative experiments to evaluate the performance of these methods, reported in Table 2 and Table 4b. Some methods are not included due to the unavailability of their code. Nonetheless, our results indicate that method generally outperforms the other alternatives. In particular, we observed significant limitations in the test-time tuning strategy of Motion-Zero [4] when applied to customized video generation, which further emphasizes the strengths of our approach.
Zoom-in/out effect with subject motion control. Our subject motion control technique effectively manages changes in bounding box size, such as gradual enlargement or reduction. Specifically, enlarging the bounding box results in an appropriate scaling of the subject, creating a zoom-in effect in the generated video. Conversely, reducing the bounding box size produces zoom-out effect. This zoom-in and zoom-out behavior is demonstrated in Fig. 12a, where examples illustrate these effects in response bounding box adjustments. Fig. 12d shows an example of combining zoom-in and camera motion control, demonstrating the method's ' flexibility.
Large camera motion speeds. We evaluate the performance of our camera motion control technique under varying levels of camera movement intensity. As depicted in Fig. 12b, the method was first tested with camera movement vector of [Ca, Cy] 0.5, 0.45). corresponding to a movement of half the video width to the left and nearly half downward. Under these conditions, our method successfully managed the camera movement, producing accurate results. When the camera motion speed was doubled, the model continued to perform well, demonstrating its ability to handle high- speed scenarios. However, at speed three times the initial setting, i.c., [cz, Cyl = I|-1.5, 1.35], the method exhibited limitations, resulting in only a downward tiling effect. These findings indicate that while our method can effectively manage camera movements spanning the entire video width, its performance diminishes under extremely high-speed conditions.
Camera motion control with large foreground objects. We evaluate the camera control capability in scenes containing substantial foreground objects. The results, presented in Fig. 12c, show an experiment involving a scene dominated by a large candy house, which nearly occupies the entire frame. Despite the significant presence of the foreground object, our technique effectively managed to pan the camera to the right, suggesting that the method is capable of handling complex scenes with large foreground elements. This robustness is attributed to the method's s reliance on latent shifts, which simultancously move both foreground and background elements, thereby ensuring that the presence of large foreground objects does not significantly impair performance.
Model efficiency. Training our model by fine-tuning a specific subject takes approximately 10 minutes. For LiVie [54], the naive text-to-video (T2V) pipeline takes about 15.0 seconds per video. When incorporating subject control, the inference time is 15.3 seconds per video; with camera control, increases to 20.6 seconds per video; and when applying both camera and subject control, the inference time is 21.5 seconds per video.
A.3 Comparison with More Baselines
Table 4: Comparison with More Baselines.
(a) Comparison of the latent shift method and text (b) Comparison of subject motion control with more baselines.
guidance for camera motion control.
Method
CLIP-T
T.Cons f
Flow error
Medwel
R.C1IP
R. B-NO
CHIP-T
TiClxx 1
Mosion Zero Ii]
Callapo 0.359
Guiddance
0.256
0,957
0.416
Latent Shift (Ours)
0.252
0.918
0.190
TediBllve [4]
0.35
6.67
Oua Zeooeegeo
6.677
0.345
01342
(loVoo
6/36
0347
0.34
16

---

(o 02)
(0.97, (0.37,0.71) 0.7T)
Camera Dexctonn
75
(0.8T,
Selett
a eköual blowiigg
322
bubbles En the pario
(0.7T,0.5T)
Notder
(+] ei noteleg crethe
153
First Frame
Last
(a) Ablation study subject motion control. Only the first frame is shown. Experiments on Zeroscope.
(b) Ablation study latent shift. Experiments on LaVie. higher a means an carlier denoising step.
Figure 13: Ablation study on motion control hyperparameters.
Table 5: Ablation study for the number of video preservation data.
Videos
R-CLIP 1
R-DINO
CLIP-T1
T-Cons. f
Flow error |
100
0.714
0.488
0.245
0.964
0.324
300
0.711
0.486
0.245
0.962
0.330
500
0.712
0.472
0.247
0.962
0.332
700
0.712
0,473
0.247
0.963
0.340
900
0.712
0,480
0.244
0.963
0.328
Comparison with text guidance for camera motion control. compare our latent shift method with text guidance for controlling camera motion with the results presented in Table 4a and Fig. 12h. Specifically, utilize simple text prompts, such "camera pan right" and "camera pan up right," to influence camera movement. The generated videos reflect these instructions to some extent, especially for straightforward motions like panning to the right. However, observe that such text prompts often lack sufficient specificity, particularly in terms of conveying critical details such as the speed or distance of the camera movement. As result, this approach yielded sub-optimal outcomes when compared to our proposed method. This indicates that our approach offers a more precise and stable mechanism for controlling camera motion than is possible with text prompts alone.
Comparison subject motion control with more baselines. Table 4b presents quantitative experiments comparing method with several baseline approaches, including TrailBlazer [34], Directed Diffusion [35], and Motion-Zer Zero [4]. The results demonstrate that our approach generally outperforms these alternatives.
A.4 Ablation Studies
Subject motion control hyperparameters. We conduct ablation studies the hyperparameter for subject motion control, with the results presented in Fig. 13a. We examined the effects of varying the factor of S and the maximum cross-atie cross-attention manipulation timestamp т. The findings indicate that increasing a and extending the controlling steps lead to stronger control. With lower control strengths, the subject does not appear in the desired position, only part of its body aligns with the intended spot. Conversely, when the control strengths are too high, the generated subjects tend to appear unnaturally square in shape.
Latent shift hyperparameters. We experiment with the influence of 0 and d2 in the latent shift module. The results shown in Fig. 13b. The results indicate that applying latent shift in the carlier steps of the denoising process results in incomplete camera movement, as evidenced by the trees in the background. Conversely, shifting the latent in the later steps degrades video quality and introduces artifacts, highlighted by the red boxes the last row. Empirically, setting di and 02 to middle values provides optimal control over camera movement.
Number of preservation videos. We conduct an ablation study on the number of preservation videos. As shown in Table 5, ranging the preservation videos from 100 to 900 does not bring large changes to the quantitative scores. We conclude that the key is to use video data to preserve the video generation ability of the pre-trained T2V models. The number of video data can be flexible.
17

---

Table 6: More ablation studies.
(a) Ablation of controlling single (b) Ablation of masking the training (c) Ablation of the latent filling method in latent shift.
motion type.
images.
Moliel
Telms:
OUAT
6210 Soogle
500
652
Macl Los (7am
6:12
6.72
630
6.43
900
6.60
Controlling Single Motion Type. We conduct an additional experiment focusing specifically on the scenario where only subject motion is controlled, and the results are summarized in Table 6a. Notably, we observe that the inclusion of both subject and camera motion controls leads to slight decrease in some metrics. For instance, using the Zeroscope model, there is a 0.1 drop in the R-CLIP score and a 0.01 decrease in the T-Cons metric when both types of motion the scenario where only subject motion is controlled. However, there are also instances where metrics such R-DINO and CLIPP-T show slight improvement under combined control conditions. We consider this trade-off acceptable within the context of motion-aware customized video generation. Qualitative results Fig. 12g show that our method can successfully handle controlling both subject and camera motion.
controlled compared to
Masking the training images. We conduct ablation study evaluate the effect of masking the background regions directly in pixel space of the training images, compared to masking the diffusion loss. As illustrated in Fig. 12c, directly masking the training images significantly impairs the model's capacity to learn the subject effectively. The quantitative metrics shown Table 6b indicate a marked decrease in performance when the training images masked. We hypothesize that this decline arises from the unnatural distribution created by the masked images, which disrupts the learning process. Consequently,
conclude that masking the diffusion loss, rather than masking the training images directly, is a more effective strategy for preserving the integrity of the learned representations.
Latent Filling Methods. We test several latent filling approaches and present the qualitative results in Fig. 12f and the quantitative results in Table 6c.
Random Ini:: This method involves filling the hole with random values. Our experiments revealed that this technique leads to severe artifacts due the disruption the natural horizontal and vertical distribution of latent pixels, ultimately degrading the overall visual quality.
Random Sample: In this approach, values randomly sampled in the latents. Similar Random Int, this method produced significant artifacts, leading to poor visual quality in the generated video.
Loop: This method reuses the moved-out-of-scenee values to fill the missing region, thereby creating a looping background effect. While this technique was found to preserve video quality better than the random methods, it introduced a limitation in terms of flexibility for camera movements, resulting in repetitive looping effects. Therefore, is not suitable for more diverse camera controls.
[26], where the missing region is filled by reflecting the surrounding content. However, in our Text-to- Video (T2V) scenario, this method collapsed, failing to maintain the desired visual quality.
Reflect: This approach is employed in
The quantitative results presented in Table 6c corroborate these findings. Random Init and Random Sample lead to significant artifacts, whereas the Loop method provides better visual quality but at the cost of limiting camera movement diversity. The Reflect method, despite its success in other applications, did not yield satisfactory results in our T2V context.
In conclusion, proposed senrghhtssampl method consistently maintained high visual quality without introducing significant artifacts or limiting camera movement flcxibility, demonstrating its superiority in maintaining the desired video fidelity.
A.5 Social Impacts
Positive societal impacts. MotionBooth allows for precise control over customized subjects and camera movements in video generation, opening new avenues for artists, filmmakers, and content creators to produce unique and high-quality visual content without extensive resources or professional equipment.
18

---

Potential negative societal impacts. The ability to generate realistic customized videos could be misused to create deepfakes. leading potential disinformation campaigns. privacy violations, and reputational damage. This risk is particularly significant in the context of political manipulation and social media. If the underlying models are trained on biased datasets, the generated content might reinforce harmful stereotypes or exclude certain groups. Ensuring diversity and fairness in training data crucial to mitigate this risk.
Mitigation strategies. Developing and adhering to strict ethical guidelines for the use and dissemina- tion of video generation technologies can help mitigate misuse. This includes implementing usage restrictions and promoting transparency about the generated content.
A.6 More Qualitative Results
We show more qualitative results in Fig. 15.
19

---

def shift shift_latent latent, one step(latent, Cx, cy, bbox, num wit__shfftsss_____
latent: noised latent for timestamp x-axis speed
camera movement
cy: y-axis speed for camera movement
bbox: the bounding box for the customized subject
num_shift_steps: d1 02, the total steps needed to complete latent shift
the latent shape
batch_ size, channels, frames, height, width latent. shape
divide the camera speed by simsshift_steps.. for each step, we move part of the total distance 5x cx / num_shh wum_shiftsstepp
cy wum_shift_steps
for f in :
it define latent shift distance for each frame sfx = int(sx f)
sfy = int(sy
# define a obj_mask mask to avoid sampling tokens within the subject region obj_nask mask torch. ones ceathhooosslleeeeaeeeeeeeeee (0, obj 2oj_nask[bbox] False
# sampling tokens horizontally
sfx != 0:
fill_x norch.zeros
for in range(height):
included indices [x for x in range(O, width) if obj obj_mask[i,x]] sampled_indices indices randon choice(included indices,, size=abs fill i latent[;, sampledd_ # sampling tokens vertically
sfy 1=
fill_y torch. 11keelatent[:,. (.bb
for in range(width):
included indices [y for y range(0, height) if ebj_mask[y,]] sampled [ampilediddicess indices randon cheiceeincluded Sinncuuudd_dddess,__ sizemabs filly[isi,,,j] latent[; ;, sampled [ampled__nniiess_]]
# shift the original latent and fill in the hole with sampled tokens sfx
temp Latent[:;;,,,
latent[:,; torch.cat([temp, fill_x], []]__x], dim=-1)
elif sfx < <: temp 1,tent[:,;,,,;;, torch.cat(Ifill) temp], dim=-1)
if sfy 8: temp ff
latent[:,;,
torch.cat([teng fill_y), dim--2)
elif sfy < 8:
temp = : :ssyy latent[:,;,,, torch.cat([fill] temp], dim=-2)
return latents
Figure 14: Pseudo-code the latent shift algorithm.
20

---

Subject
Subject motion
Camera motion
Generated Videos
Right
plussile panda nowning the grass
Down Left
doa jumping the staiks
A robot toy walkiing in sunvise
A slog namning the teach
A ixctorbile moving closer ov. the beack with seaskore in the background
ear driving on the kigkway
Right
plushie swimming ground ooval reef
Up Left
A wolf plushie ruuning ow the
Figure 15; More qualitative results of MotionBooth.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We include the main claims in the abstract and introducion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in Section 4.5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details in Section and Appendix A. that are needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[No]
Justification: We are not able to provide the code at submission time. But we are making sure that our code and models will be released publically in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nip) cc/
public/guides http///eeuiissiioooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wnips.ccpppil wttpcccppppccccccccocccccccccccc guwdee/ Cooeeubmii www.inooliiieo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details in Section and Appendix A.1 that are needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We did conduct experiments with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the implementation details in Section 4 and Appendix A. that are needed to reproduce the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https /nerrpss.....// cc wctppcccc///ccccccccccccccccctct Etthi csluidelineeees
Answer: [Yes]
Justification: We conducted in the paper confirm, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impacts of MotionBooth in Appendix A.5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe the safeguards in Appendix A.5.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use existing assets with properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We introduce new assets in Section 4 and Appendix A.I.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "rGVKWSmxLM", "reviewer": "Reviewer_aAio", "summary": "This paper targets at animating customized subjects with both motion and camera control. To achieve that, they first customize t2v models with subject images finetuning without hurting video generation capability. Then, they propose a training-free approach to manage subject and camera motions. Extensive experiments demonstrate the superiority of their method.", "questions": "•\tFor subject motion control module, if the bounding box size is changing, e.g., gradually enlarging bounding box, will the method enlarge the subject accordingly?\n\n•\tSubject motion is composed of both position translation and subject movements, such as walking, dancing, jumping, etc. The paper only control the subject position for subject motion. The reviewer wonders is it a bit overclaiming for controlling subject motions, since the subject movements cannot be customized or controlled?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThe paper introduces the first unified video generation framework for subject customization with motion and camera movement control, which is also an interesting problem setup.\n\n•\tThe paper proposes a new subject region loss and video preservation loss for video customization, and a training-free method to control both subject motion and camera motion.\n\n•\tThe paper is well-written with clear motivations.", "weaknesses": "•\tThe subject motion control module is interesting by directly amplifying in the bounding box region and suppressing outside the box region. However, how to define the value of parameter alpha remains unclear. Is it only defined based on experimental performance? And does the authors keep this alpha parameter the same for all customization experiments?\n\n•\tThe proposed method only control the camera movement in 2D such as up, down. However, camera movement is quite important for 3D, such as moving into the scene, rotation, etc. The proposed camera control module is not able to handle such camera motions and can be a weakness for camera control.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the review! Here are our replies.\n\n**Note: Please refer to the one-page PDF for the mentioned figures and tables with an \"R.\" in their names.**\n\n\n# The definition of hyper-parameters\n\nWe use different sets of hyper-parameters for the Zeroscope and LaVie models, as detailed in `Appendix A.3` of the paper. The selection of these hyper-parameters, including the value of parameter alpha, is primarily based on experimental performance. Once determined, we maintain these hyper-parameters consistently across all our experiments to ensure comparability of results.\n\nFurthermore, we conduct a thorough ablation study of these hyper-parameters, including alpha, as presented in `Figure 12 (a)` and `(b)` of the main paper. This study analyze the impact of different settings, and the findings are discussed in detail in Appendix A.4. We hope this clarifies the basis and consistency of our parameter choices.\n\n\n# Enlarging and shrinking subjects through subject motion control\n\nThank you for the question. Our subject motion control technique indeed accommodates changes in the bounding box size, such as gradual enlargement or reduction. When the bounding box is enlarged, the method appropriately scales the subject, resulting in a zoom-in effect in the generated video. Conversely, shrinking the bounding box causes the subject to appear smaller, producing a zoom-out effect.\n\nThis behavior is demonstrated in `Figure R.1 (a)`, where examples illustrate the zoom-in and zoom-out phenomena corresponding to the bounding box adjustments. Additionally, Figure 14 in the main paper includes an example that showcases the enlargement of an object through the progressive increase of the bounding box.\n\n\n# The 3D controlling ability of camera motion control technique\n\nWe acknowledge that the proposed latent shift module primarily facilitates control over 2D camera movements, such as up and down panning. However, it is important to note that our work is pioneering in utilizing this approach for camera control. While our current method does not inherently support 3D camera motions like moving into the scene or full rotations, we have integrated subject motion control techniques to achieve zoom-in and zoom-out effects. By combining these techniques with 2D camera motion control, we can provide a more versatile camera control experience.\n\nAs illustrated in `Figure R.1 (d)`, we demonstrate an example where the camera pans left while simultaneously zooming in. This showcases our system's ability to achieve a complex, coordinated camera movement, highlighting the potential for more dynamic and flexible camera control in future developments.\n\n\n# The expalanation to control the position of subjects as subject motion control\n\nThank you for your feedback. In our paper, when we refer to controlling subject motion, we are specifically addressing a limitation observed in recent text-to-video (T2V) models. Many of these models fail to generate realistic motion that involves actual positional changes of the subject within the scene. For example, given a prompt like \"a dog running in the forest,\" some models, such as LaVie, produce animations where the dog appears to be running in place, without any change in its overall position in the video.\n\nOur approach focuses on overcoming this limitation by explicitly controlling the subject's position, thereby enhancing the perceived realism of the motion. For instance, we ensure that the dog not only appears to be running but actually moves from one side of the scene to the other. This positional control is crucial for creating a more convincing representation of motion.\n\nWhile it is true that our method does not allow for granular control over specific subject movements like walking, dancing, or jumping, we believe that our focus on controlling positional changes significantly contributes to the overall quality and realism of motion in generated videos. Additionally, specific subject movements are already guided by text prompts, which direct the model to generate appropriate actions. Therefore, we maintain that our claim of controlling subject motion is justified, as it addresses a critical aspect of motion that many existing models overlook.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G9A6pDaJny", "reviewer": "Reviewer_ZPXz", "summary": "This paper primarily addresses the problem of motion-aware customized video generation. Traditional customized generation methods either tend to lose the motion information of the video or require additional training of motion modules. To enhance subject fidelity and video dynamics, this paper introduces subject region loss and video preservation loss. Additionally, a training-free subject- and camera-motion control method is proposed, enabling the model to flexibly control the motion information of the video. Extensive experiments demonstrate that the proposed method outperforms existing state-of-the-art methods.", "questions": "1.\tHow many subject samples and/or common videos are used for training each customized model?\n2.\tThe proposed mainly explores the camera motion on panning left or right, is it possible to control other motion types, like zoom-in/out effects? \n3.\twhat is the difference between the proposed subject region loss, subject token cross attention loss, and those losses in Mix-of-show [1] for customized image generation\nref:\n[1] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models", "limitations": "1. This paper needs to discuss more methods addressing the same tasks, like motion control, and specify the differences between the proposed loss functions and those of existing methods.\n2. More limitations or failure cases needed to be discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper achieves better automized generation results than the state-of-the-method\n2.\tThe writing is clear and easy to follow", "weaknesses": "1.\tSome details need to be added to more clearly distinguish the differences. For example, in line 169, existing methods use class-specific data, whereas this paper uses common videos. Do the common videos need to include customized subjects? How are these common videos obtained?\n2.\tThe assumptions made by the method, such as sampling operations on the x- and y-axes when controlling camera motion, may limit its effectiveness. For instance, in scenarios with significant changes from left to right or top to bottom, or in close-up videos with large foreground objects, this operation may result in uncoordinated outcomes. However, the paper does not seem to discuss these limitations.\n3.\tMethods that control subject and camera motion through object trajectories and camera parameters have also been proposed, such as motionCtrl, TrailBlazer, Boximator, and motion-zero. However, this paper does not provide an in-depth discussion or experimental comparison with these methods. Although some of these methods are training-based, it is still worthwhile to explore the existing gaps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the review! Here are our replies.\n\n**Note: Please refer to the one-page PDF for the mentioned figures and tables with an \"R.\" in their names.**\n\n\n# Details about video data used for training\n\nThanks for the question, during training, we **randomly** download 500 videos from the Panda-70M dataset. The Panda-70M dataset is a large-scale text-video dataset that contains 70M videos from YouTube. The videos are annotated with captions. We call them common videos because **we do not constrain them to have any specific class or the customized subjects**. The reason to use video data is to preserve the video generation ability of pre-trained T2V models.\n\nThe customization ability is learned through 4-6 images of the same subject.\n\nFor more details on the training data, please refer to `Sec. 4.1 Implementation details`, specifically lines 257-258.\n\n\n# Camera motion control on extreme cases\n\nThank you for your valuable feedback. We conduct analysis to address the limitations you mentioned, focusing on extreme scenarios involving significant camera movements and large foreground objects.\n\n## Large Camera Motion Speed\n\nWe test our camera motion control technique with varying levels of camera movement. The results are shown in `Figure R.1 (b)`. Initially, with camera movement [c_x, c_y] = [-0.5, 0.45], indicating a movement of half the video width to the left and nearly half downward, our technique successfully controls the camera motion.\n\nEven when we double the camera motion speed, the model continues to output accurate results. However, at triple the speed, [c_x, c_y] = [-1.5, 1.35], the method fails, showing only a downward tiling effect.\n\nThese findings indicate that while our method can manage camera movements spanning the entire video width, it does have limitations under extremely high-speed conditions.\n\n## Camera Motion Control with Large Foreground Objects\n\nResults are shown in `Figure R.1 (c)`. The experiment involves a scene dominated by a large candy house, nearly occupying the entire frame. Despite this, our technique effectively pans the camera to the right.\n\nThis outcome suggests that our method can handle camera movements even with large foreground objects. We believe this effectiveness stems from the method's reliance on latent shifts, which moves both foreground and background elements simultaneously. Therefore, the presence of large foreground objects does not significantly impair the performance.\n\n\n# Comparisons with more subject and camera motion control methods\n\nThanks for the suggestion. Due to the response word limit, please see the general response for comparison to related methods.\n\n\n# Zoom-in/out\n\nWe acknowledge that the proposed latent shift module primarily facilitates control over 2D camera movements, such as up and down panning. However, we observe that we can achieve the zoom-in/out effect through gradually enlarging or reducting the bounding box for the subject, with subject motion control.\n\nThis behavior is demonstrated in `Figure R.1 (a)`, where examples illustrate the zoom-in and zoom-out phenomena corresponding to the bounding box adjustments. Additionally, Figure 14 in the main paper includes an example that showcases the enlargement of an object through the progressive increase of the bounding box.\n\nBy combining this technique with 2D camera motion control, we can provide a more versatile camera control experience.\n\nAs illustrated in `Figure R.1 (d)`, we demonstrate an example where the camera pans left while simultaneously zooming in. This showcases our system's ability to achieve a more complex, coordinated camera movement, highlighting the potential for more dynamic and flexible camera control in future developments.\n\n\n# Differences of losses with Mix-of-Show\n\n**Mix-of-Show:** This method focuses on training an ED-LoRA (Efficient and Decentralized Low-Rank Adaptation) model for individual clients. The primary aim is to merge multiple LoRA weights using a multi-concept fusion technique to handle multiple concepts effectively.\n\n**Our Method:** We aim to eliminate the overfitting problem to the background while learning from a few subject images. Our approach includes specific loss functions designed to address this issue and improve the representation of subjects within videos.\n\n1. Subject Region Loss:\n\n**Mix-of-Show:** Does not explicitly focus on subject region loss. Instead, it leverages the general capabilities of the ED-LoRA model to handle multiple concepts.\n\n**Our Method:** We introduce a subject region loss specifically designed to minimize overfitting to the background. This loss ensures that the model learns to focus on the subject itself, rather than the surrounding context, thereby improving subject-specific customization.\n\n2. Subject Token Cross-Attention Loss:\n\n**Mix-of-Show:** The approach does not include a specific loss function that guides the subject token with the subject's position in the video.\n\n**Our Method:** We propose a subject token cross-attention loss to explicitly guide the subject token with the subject's position in the video. This helps in maintaining the spatial consistency of the subject across frames, ensuring that the subject is accurately represented in its correct position throughout the video.\n\nTo summarize, our method distinguishes itself by introducing specialized loss functions that address specific issues in customized image generation, such as overfitting to the background and maintaining subject position consistency. These targeted losses are not present in the Mix-of-Show approach, which focuses more broadly on decentralized training and merging of multiple LoRA weights.\n\nWe will cite Mix-of-Show and incorporate this discussion in the final version of our work. If further specific aspects of our loss functions need to be compared with those in Mix-of-Show, we would be happy to provide a more detailed comparison. Thank you for your attention to this matter.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oDZlA7KlmT", "reviewer": "Reviewer_N7np", "summary": "This paper addresses the challenge of how to control the identity and the motion of the subject while generating video, in a text-to-video setup. Specifically, in addition to the standard text prompt, it allows the user to control: the subject's identity by providing a few images of it (e.g. 5 photos of my dog); the subject's position in the video, as provided by a bounding box sequence; and the camera motion, as provided by a sequence of delta-x & delta-y.\n\nIdentity preservation is achieved by two changes to the loss function, during fine-tuning on images of the subject. First a \"subject region loss\" zeros out gradients that lie outside the subject's bounding box during fine-tuning, and second a video preservation loss (similar to the class-specific preservation loss introduced by DreamBooth) to ensures that the ability to generate dynamic motion is not lost due to over-fitting on a few static images. \n\nSubject and camera motion are controlled at inference time, by modifying the cross-attentional map between the prompt and the latent feature, by shifting the latent feature, respectively.\n\nFinally, a third loss \"subject token cross-attention\" links the two, enabling the inference motion controls by ensuring that the cross-attention between the subject token (in the text prompt) and the diffusion model's U-Net features is strong where the object should be and weak elsewhere.\n\nThese proposed changes are validated quantitatively through several metrics: CLIP and DINO scores between the source images and the desired video locations of the subject, CLIP score between the text prompt and video frames, and a difference in optical flow between the generated video and ideal video (according to the subject and camera motion prompts).\n\nExamples of generated videos are included in the supplemental.", "questions": "Re \"Subject Region Loss\": How certain are you that the binary mask, when applied to the latent (rather than the pixels), actually does mask out the background? Given that the latent is produced by a stack of many convolutional or transformer layers, the receptive fields of each latent location almost certainly include information from the nearby background. Have you tried masking the training images directly in pixel space, rather than masking the diffusion loss? How does that compare?\n\nRe \"Ablation studies\": w/o video and w/ class video give quite-nearly identical numbers. Is this a copy-paste error? If not, it's worth discussing in more detail.\n\nIn Figure 3, what is the difference between c, d, e, and f? Are they different samples from the model, or from different models?\n\nHave you tried to achieve the desired camera motions via the text prompt, and considered that as a baseline? I have seen it work well for many basic types of motions in some recent t2v models that you cite.", "limitations": "The limitations described in the appendix are important enough that they should be included in the main paper.\n\nOtherwise, yes, they have addressed limitations and impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper addresses a very significant and timely problem -- how to effect significantly more fine-grained control in text to video models, and allow them to be more than a novelty.\n\nThe results, as shown in the supplemental video, are very impressive: the subject identity, subject motion, and camera motions very much follow the user's input.\n\nThe additional loss terms are intuitively sensible and effective. The inference modifications, although they seem a bit \"hacky\" (if you'll excuse me), do get the job done quite effectively.\n\nI appreciate that the transferability of the method was demonstrated by implementing it on two different text-to-video models.", "weaknesses": "These are relatively minor weaknesses in my opinion:\n\nThis region metrics measure whether the desired object is translating around the frame as per the bounding box guidance, but because they are image metrics, can't evaluate the actual \"animation quality\" of the subject. I believe the entire suite of metrics could be optimized by a static image crop of the subject sliding across the video frame according to bounding box motion and flow. Can you confirm whether this is correct?\n\nThe set of camera motions that can be models is limited to 2d translations in x and y, missing zooms, rotations, etc.\n\nAlthough this paper presents its techniques as transferable across many latent diffusion models, they require that the \"latent can be considered a \"shrunk image,\" which maintains the same visual geographic distribution as the generated images.\"  This should generally be true for most latent video diffusion models, but may not work with new latent encoders being developed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the review! Here are our replies.\n\n**Note: Please refer to the one-page PDF for the mentioned figures and tables with an \"R.\" in their names.**\n\n\n# Metrics for static image crop of the subject\n\nWe apologize for any confusion. We believe your suggestion involves cropping a video to focus on the subject, according to the bounding box, and then calculating a metric that reflects the \"animation quality\" within that cropped video.\n\nThis is a valuable suggestion. In response, we developed a new metric called \"Region CLIP-text.\" This metric first crops the video to isolate the subject as guided by the bounding box. Then, it calculates the similarity between the cropped video and a text prompt describing the animation, such as running, jumping, etc. We believe this metric better reflects the quality of the subject's animation by focusing specifically on the subject's actions.\n\nThe results, presented in `Table R.1. (f)`, compare our method with other approaches in terms of subject motion control. The \"Region CLIP-text\" metric shows a trend similar to CLIP-T but with a stronger emphasis on the subject region. It more accurately captures the alignment of the subject's animation with the descriptive text. Our method outperforms the baselines, demonstrating the effectiveness of our subject motion control technique.\n\nRegarding a desired video metrics, please refer to the general response due to the word limit.\n\n\n# Capability of camera motion control\n\nWe acknowledge that the proposed latent shift module primarily facilitates control over 2D camera movements, such as up and down panning. However, we observe that we can achieve the zoom-in/out effect through gradually enlarging or reducting the bounding box for the subject, with subject motion control.\n\nThis behavior is demonstrated in `Figure R.1 (a)`, where examples illustrate the zoom-in and zoom-out phenomena corresponding to the bounding box adjustments. Additionally, Figure 14 in the main paper includes an example that showcases the enlargement of an object through the progressive increase of the bounding box.\n\nBy combining this technique with 2D camera motion control, we can provide a more versatile camera control experience.\n\nAs illustrated in `Figure R.1 (d)`, we demonstrate an example where the camera pans left while simultaneously zooming in. This showcases our system's ability to achieve a more complex, coordinated camera movement, highlighting the potential for more dynamic and flexible camera control in future developments.\n\n\n# The assumption of \"shrunk image\"\n\nThank you for your insightful feedback. Models such as ModelScope, Zeroscope, LaVie, and VideoCrafter all adhere to the principle where the latent can be considered a \"shrunk image,\" preserving the same visual geographic distribution as the generated output.\n\nRegarding newer models, we anticipate that they will likely follow a similar architecture, especially given the examples of video editing capabilities demonstrated by Sora using SDEdit technology. The requirement for the latent to act as a \"shrunk image\" appears to be a foundational aspect of such technologies. Therefore, we believe our techniques will remain applicable and useful even as new latent encoders are developed.\n\n\n# Mask the diffusion loss vs mask the training images\n\nThank you for your insightful suggestion. We conduct an ablation study to compare the effect of masking the background regions in the training images directly in pixel space, as opposed to masking the diffusion loss.\n\nAs demonstrated in `Figure R.1 (e)`, masking the training images significantly impairs the model's ability to learn the subject. The video example clearly shows that this approach results in a substantial degradation in the model's performance. Furthermore, the quantitative metrics, as presented in `Table R.1 (b)`, reflect a marked decrease in performance when the training images are masked.\n\nWe hypothesize that this decline is due to the masked images creating an unnatural distribution, which disrupts the learning process. Therefore, we conclude that masking the diffusion loss, rather than the training images, provides a more effective approach for maintaining the integrity of the learned representations.\n\n\n# Numbers in Table 3\n\nIt is not a copy-paste error. This result demonstrates that using class-specific videos is comparable with not using any videos.\n\n\n# Differences between c, d, e, and f in Figure 3\n\nFigure 3 shows an ablation about learning loss techniques. They are all sampled from a customized Zeroscope model on the white dog. (c) means the model is trained without using subject region loss and without video loss. (e) means the model is trained with subject region loss but without video loss, etc.\n\n\n# Control the camera motions through text prompt\n\nThank you for your insightful suggestion. We conduct an ablation study and presented the results in `Figure R.1 (h)`. We add simple text prompts such as \"camera pan right\" and \"camera pan up right\" to control the camera motion. The generated videos do follow these instructions to some extent, particularly for simpler motions like panning right. However, we find that these text prompts are often too vague, lacking the ability to specify important details such as camera motion speed or distance. Consequently, this approach yields sub-optimal results compared to our proposed method.\n\nQuantitative results supporting this observation are shown in `Table R.1 (d)`. While text guidance results in slightly higher CLIP scores and temporal consistency, the flow error—which measures the precision of camera is significantly lower than what we achieved using our method. This demonstrates that our approach provides more precise and stable camera motion control than can be achieved with text prompts alone.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "az3c9ufBYi", "reviewer": "Reviewer_yPkE", "summary": "The paper proposes a video generated method - for customizing a subject, along with its motion, provided in the form of bounding boxes, and camera movement provided in the form of camera pose. To do this, the training method has a subject region loss to prevent bias from background and other details in the image, and a video regularization loss function to ensure that video generation properties are not forgotten. At inference, in order to control camera movements, latents are shifted. Cross-attention maps are edited with the provided bounding boxes, to fix the subject motion.", "questions": "I am unable to judge the quality of this work with the limited number of qualitative results that have been provided. Moreover, the provided qualitative results are sub-optimal, esp in the case of subject + camera motion, questioning the working of the method. Given that the limit on the size of the supplementary material is considerably large, it would have been nice if the authors had provided more results showcasing their method.\n\nSince the rebuttal does not allow for more qualitative results, I am rejecting the paper at this stage. \n\nThe method, while pretty good, probably needs a little more work to get it working for the subject + camera motion case. I recommend that the authors continue working on it to strengthen the paper.\n\nPost rebuttal: it would have been nice to see more qualitative results and videos, but i understand that is not possible in the 1 page rebuttal pdf. Based on the submitted rebuttal, I am happy to increase the score.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is written well. I particularly like the introduction and related work, where the paper clearly motivates the problem, describes the challenges and goes on to describe how it solves the problem. \n\nThe method comprises of various components, and each of the components is designed very well - they are intuitive, make sense and well formulated. The method very systematically addresses various challenges involved in the problem by designing various relevant techniques. The paper is also written very well to motivate and describe the method.", "weaknesses": "My main concerns are with the experiments, which are very non-convincing. While the paper does provide detailed quantitative results for comparisons and ablations on various metrics validating the solution, the qualitative results do not support that.\n\n1. I watched the supplementary video, which has the videos for the results presented in the paper. First of all, the number of qualitative results provided is severely limited, raising questions about cherry picking. Second, the method seems to perform reasonably well on cases that either have camera motion or subject motion - the results look good. In cases involving both camera as well as subject motion, which is the theme of the paper, the results are highly suboptimal - for instance, at 1:48 of the video, the last video showing the dog does not show the subject motion appropriately.\n\n2. I would like to see more detailed results that segregate the various kinds of motion (camera and subject), to see how effective the method is for just camera motion, just subject motion and a combination of both. This will give a clearer picture of where the method is failing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the review! Here are our replies.\n\n**Note: Please refer to the one-page PDF for the mentioned figures and tables with an \"R.\" in their names.**\n\n\n# Separate results for subject motion control, camera motion control, and both\n\nThank you for your feedback and suggestion. It's a valid point to consider the performance of our method across different types of motion scenarios.\n\nIn the main paper, we've already presented quantitative results for scenarios involving only camera motion, as shown in `Table 2`. Additionally, we provided results for cases where both subject and camera motion are controlled, as detailed in `Table 1`. Following your recommendation, we conducted an additional experiment to evaluate our method specifically for scenarios involving only subject motion. The results are presented in `Table R.1 (a)`.\n\nFor a comprehensive analysis, we have compiled the results for all three cases—camera motion, subject motion, and both—into a single table. This allows us to better understand how controlling these aspects influences the performance. Notably, we observed that when both subject and camera motions are controlled, there is a slight decrease in some metrics. For instance, with the Zeroscope model, there's a 0.1 drop in the R-CLIP score and a 0.01 decrease in the T-Cons metric when both motions are controlled, compared to controlling only subject motion. Similarly, the flow error increases slightly when subject motion control is added to camera motion control. However, there are also instances where the R-DINO and CLIP-T metrics show a slight improvement under combined control.\n\nIn conclusion, while there is a partial performance drop when incorporating both control aspects, we believe this trade-off is reasonable and acceptable within the context of the motion-aware customized video generation task. It's natural for some quality loss to occur when adding more control signals, especially with training-free methods. However, the overall quality remains above an acceptable threshold, ensuring that the generated results are still meaningful and effective. Finally, considering the novelty of the proposed subject and motion control methods and the performance gain compared to existing approaches, we believe our work makes a substantial contribution to the field.\n\n\n# Examples of controlling both camera and subject motion\n\nThank you for your feedback and for taking the time to review the supplementary video. We appreciate your concerns regarding the number of qualitative results and potential cherry-picking. We would like to clarify and address your points.\n\nWe clarify that all the qualitative cases compared with baselines are sampled without cherry-picking. We use different subject and camera motions to better demonstrate the model capability. We will release all the qualitative results in the final version.\n\nAs indicated in the quantitative results, our method is designed to effectively control both subject and camera motion. For a more comprehensive understanding, we have provided additional qualitative examples in the revised manuscript.\n\nIn particular, `Figure R.1(g)` illustrates the \"dog running on the beach\" video with eight distinct camera movements. This example demonstrates the capability of our method to manage complex scenarios involving simultaneous subject and camera motion. The dog's movement from the top right to the bottom left of the frame, combined with various camera motions, showcases the robustness of our approach in maintaining both motions' integrity.\n\nAdditionally, `Figure R.1(d)` shows another challenging scenario where the camera pans to the left while zooming in on the subject by enlarging the bounding box. This example further emphasizes our method's ability to handle complex motion dynamics.\n\nWe believe these examples provide clear evidence of our method's effectiveness in different motion scenarios, including cases involving simultaneous subject and camera motion. We will consider your suggestion to include more detailed segregated results to further validate the method's performance across different types of motion.\n\nThank you again for your valuable input. We hope these examples address your concerns and demonstrate the robustness of our approach.\n\n\n# More detailed ablations of our method and more comparisons\n\nIn the one-page PDF, we also conduct many other qualitative and quantitative experiments to thourghly analysis our method, and compare with more baselines.\n\nTo be specific, `Figure R.1 (a)`, shows we can achieve the zoom-in and zoom-out effect by gradually enlarging or shrinking the subject bounding box.\n\n`Figure R.1 (c)` involves a scene dominated by a large candy house, nearly occupying the entire frame. Despite this, our technique effectively pans the camera to the right. This outcome suggests that our method can handle camera movements even with large foreground objects.\n\n`Table R.1 (e)` and `(f)` shows quantitative experiments comparing with more baselines, including TrailBlazer, Directed Diffusion, Boximator, Motion-Zero, MotionCtrl, and Text2Video-Zero.. Our results generally outperforms the alternatives.\n\n`Figure R.1 (f)` and `Table R.1 (c)` explores different methods for filling the missing region in the shifted noised latent.\n\nWe hope the extra results will provide a broader and deeper sight about our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tAKkNslVVi", "reviewer": "Reviewer_q2XJ", "summary": "The authors propose MotionBooth, a method to fine-tune a pre-trained text-to-video model on a collection of images of a specific object to enable the model to generate controlled videos of that object. Fine-tuning incorporates three losses: diffusion loss on image data, restricted to the object's region; video preservation loss, i.e diffusion loss on sample video data, to prevent overfitting to static images; subject token cross-attention loss, to improve controllability at generation time. During inference the motion of the object is controlled via editing the cross-attention maps to amplify the attention to the object tokens within the object's bounding boxes. The camera is controlled via shifting the noised latent and filling the new regions with latents sampled from the background of the original noised latent.", "questions": "1) Could the authors discuss the connection between the proposed MotionBooth and the prior work with similar training-free controls? Some ablation studies would be ideal. \n\n2) Could the authors provide more details about their subject motion control attention suppression technique? Especially regarding the issue with attention to the prompt tokens other than the subject language tokens.\n\n3) The authors claim that their fine-tuning is efficient (line 3 in the abstract). Could the authors provide more details on this? E.g. what is the training time to adapt the model to a new subject?", "limitations": "The authors have adequately addressed the limitation and the potential negative societal impact of their work.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem of controllable video generation is of high significance nowadays. The method proposed in the paper is relatively simple and the demonstrated visual results confirm the improved controllability of MotionBooth over prior work. The contributions in the paper are supported with well-designed figures aiding the clarity of the presentation. The limitations of the proposed method are discussed in the appendix.", "weaknesses": "1) **Novelty**: \n    - Similar training-free object motion control was proposed in [1] and [2] for image and video models respectively, where cross-attention amplification within object's bounding box was used to control its location in the generated frame. Similar training-free camera motion control was used in [3], where a series of noised latent shifts were used to control the global scene and camera motion. \nDiscussion and comparisons with those works are missing in the paper.\n    - Some missing references: [4, 5].\n\n2) **Method**: \n    - Based on the formula for subject motion control, the attention is suppressed (set to -infinity) for all query-key pairs other than the queries from the bounding box and the keys from the object tokens. This way it seems that the rest of the prompt is ignored. Moreover, it is stated in the paper that \"the attention suppression outside the bounding box regions persists throughout the generation\". Details clarifying this are missing in the paper.\n\n3) **Evaluation**: \n    - The paper lacks quantitative evaluation. Comparing CLIP features between the frames doesn't measure temporal consistency. Metrics like FVD, or optical flow warpping error could be better choices for this purpose.\n    - Details on how exactly the metrics are calculated are missing. Formal definitions of newly introduced metrics are required for better understanding. E.g. how exactly is the flow error calculated in the case when not all the motion except for the object's motion corresponds to camera motion? Having those would help with understanding the significance of the improvements reported in the tables. \n    - The evaluation dataset consists of limited number of different object classes, mostly dogs, cats and toys. Evaluation on more general collection of object classes would better support the claims made in the paper.\n    - More ablation studies would better illustrate the contributions. E.g. what would happen if the missing region in the shifted noised latent was not filled with samples from the same noised latent? Or if camera control like in [3] was used instead?\n\n4) **Results**:\n    - Figure 14 shows that bounding boxes often don't fully contain the object of interest. Given that the attention is suppressed outside of bounding boxes, this looks like a flaw in the proposed control and needs to be investigated.\n\n[1] Ma et al., Directed diffusion: Direct control of object placement through attention guidance, AAAI 2024.\n\n[2] Ma et al., TrailBlazer: Trajectory Control for Diffusion-Based Video Generation, arxiv 2024.\n\n[3] Khachatryan et al., Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators, ICCV 2023.\n\n[4] Wang et al., Boximator: Generating rich and controllable motions for video synthesis, arxiv 2024.\n\n[5] Chen et al. Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based\nVideo Generation, arxiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for the review! Here are our replies.\n\n**Note: Please refer to the one-page PDF for the mentioned figures and tables with an \"R.\" in their names.**\n\n\n# Novelty of our method\n\nThanks for the suggestion. We appreciate that most reviewers have acknowledged the novelty of our method and we will add the missing citations in the final version.\n\nDue to the response word limit, please see the general response for comparison to related methods.\n\n\n# Ablations of the latent shift filling method\n\nTo address the reviewer's question regarding additional ablation studies, we have conducted experiments to explore different methods for filling the missing region in the shifted noised latent. We present the qualitative results in `Figure R.1 (f)` and the quantitative results in `Table R.1 (c)`.\n\nIn our experiments, we tested several approaches:\n\n1. Random Init: Filling the hole with random values. This method resulted in severe artifacts due to the disruption of the natural horizontal and vertical distribution of latent pixels.\n2. Random Sample: Randomly sampling values in the latents. Similar to Random Init, this approach also produced significant artifacts and degraded the visual quality.\n3. Loop: This method involves using the moved-out-of-scene values to fill the hole, creating a looping background effect. While this technique preserves video quality better than random initialization or sampling, it limits the flexibility of camera movements by producing only looping effects. Thus, it is not suitable for more varied camera controls.\n4. Reflect: This is the method used in Text2Video-Zero, where the missing region is filled by reflecting the surrounding content. However, in our Text-to-Video (T2V) situation, this approach resulted in the method collapsing, failing to maintain the desired quality.\n\nThe quantitative results in `Table R.1 (c)` corroborate these observations, showing that Random Init and Random Sample lead to significant artifacts, while Loop provides better quality but lacks flexibility. The Reflect method, although effective in other contexts, did not perform well in our scenario.\n\nIn conclusion, our straight-sample method consistently maintained high visual quality without introducing significant artifacts or limiting camera movement flexibility.\n\n\n# Details about attention suppresion\n\nThank you for your insightful question. In our approach, we manipulate the dot product between the query matrix $\\mathbf{Q}$ and the key matrix $\\mathbf{K}$. The final attention map is then calculated using a Softmax function across all tokens.\n\nThe amplification of the subject tokens is during the earlier denoising steps.\n\nIn the later steps, we suppress all the tokens, including the subject tokens all to $-\\infty$. This does not completely ignore any tokens due to the nature of the Softmax function, which normalizes the scores across all tokens.\n\nOur qualitative results, as presented in the paper and the accompanying one-page PDF, demonstrate that the generated videos do incorporate other aspects of the prompt, such as verbs or background elements. This shows that while the subject tokens receive amplified attention, the rest of the prompt tokens are still considered, influencing the overall output.\n\n\n# Video metrics\n\nThanks for the feedback. Due to the word limit, please refer to the general response.\n\n\n# Flow error metrics calculation\n\nWe exclude the subject region when calculating flow error, so the flow error only accounts to camera motion control.\n\nThe flow error $E_{flow}$ for each frame can be expressed as:\n\n$$\nE_{flow} = \\frac{1}{|R|} \\sum_{p \\in R} \\| \\mathbf{F}_p - {\\mathbf{C}} \\|, \n$$\n\nwhere $\\mathbf{F}_p$ is the predicted flow vector at pixel $p$. ${\\mathbf{C}}$ is the ground truth camera motion condition. $R$ represents the region outside the subject bounding box. $|R|$ is the number of pixels in region $R$.\n\n\n# Evaluation datasets\n\nThank you for your feedback. We acknowledge that current open-sourced, state-of-the-art T2V models, such as LaVie and Zeroscope, exhibit limitations in generating a wide variety of object types. For example, Zeroscope struggles to generate humans with its pre-trained weights. In our study, we have followed the precedent set by previous research [1,2,3] in selecting our evaluation dataset, which includes categories like pets, plush toys, cartoons, and vehicles. We believe this selection provides a sufficiently diverse set of objects to comprehensively evaluate our proposed techniques within the capabilities of the existing T2V models.\n\nWe emphasize that our approach, characterized by training-free intrinsic subject and camera motion control methods, offers strong potential for generalizability. We are confident that, with the advent of more advanced T2V models, our methods will demonstrate even broader applicability and effectiveness across a wider range of object classes.\n\n\n# Out-of-bbox problem\n\nOur approach involves controlling the approximate position of the subject primarily during the earlier denoising steps. This method can sometimes result in portions of the subject extending beyond the defined bounding box in the final output. This is a deliberate design choice, as overly restrictive bounding box conditions would lead to rigid, unnatural outputs that may appear as squared or boxed regions, which are not desired.\n\n\n# Model efficiency\n\nTraining:\n\nFine-tuning a subject takes about 10 minutes (Line 263 in paper).\n\nInference:\n\nNaive T2V pipeline: around 15.0s per video\n\nours:\n\n\\+ subject control: 15.3s per video\n\n\\+ camera control: 20.6s per video\n\n\\+ camera & subject control: 21.5s per video\n\n\n\n[1] Ruiz, Nataniel, et al. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" CVPR 2023.\n\n[2] Wei, Yujie, et al. \"Dreamvideo: Composing your dream videos with customized subject and motion.\" CVPR 2024.\n\n[3] Wang, Zhao, et al. \"Customvideo: Customizing text-to-video generation with multiple subjects.\" arXiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks very much for the reviews! In the general response, we mainly reply to the most frequently asked questions.

**Note: Please refer to the one-page PDF for the mentioned figures and tables with an "R." in their names.**


# Novelty of our subject and camera motion control methods

We appreciate that most reviewers have acknowledged the novelty of our method. To further clarify, we are willing to elucidate the differences between our proposed techniques and some related mentioned, including TrailBlazer, Directed Diffusion, Boximator, Motion-Zero, MotionCtrl, and Text2Video-Zero.

**TrailBlazer and Directed Diffusion:** While these methods utilize a training-free approach to control object motion by manipulating cross-attention, they differ from our approach. TrailBlazer adjusts both spatial and temporal cross-attention by scaling the attention maps with a hyper-parameter less than 1, and Directed Diffusion, similarly, focuses on image generation. In contrast, our method exclusively targets spatial cross-attention and sets the attention maps outside the object's bounding box to $-\infty$. This approach not only simplifies the implementation but also enhances performance in generating motion-aware customized videos.

**Boximator:** Unlike our training-free method, Boximator is a training-based technique that requires box coordinates to be input into a newly trained self-attention layer. Since Boximator is not open-sourced, it presents a higher barrier to use. Our method's training-free nature provides a more accessible solution for controlling subject and camera motion.

**Motion-Zero:** This method also operates without additional training but employs a test-time tuning technique that adjusts the latent space using cross-attention map loss during denoising. However, this process increases generation time and memory usage significantly, from approximately 15 seconds to several minutes per video. Our experiments showed that Motion-Zero often produces collapsed videos with unrecognizable visual elements, likely due to the adverse effects of test-time tuning on the parameters and latent distributions in a customization scenario. In contrast, our approach directly manipulates the cross-attention map, adding only 0.3 seconds per video and yielding more reliable outcomes.

**MotionCtrl:** utilizes a training-based approach to control camera poses and object motion by inputting point trajectories. In contrast, our approach is training-free, enabling control over subject and camera motion without requiring additional training processes.

**Text2Video-Zero:** This method extends a pre-trained text-to-image (T2I) model to video generation by using consistent noise across frames, which is unsuitable for text-to-video (T2V) models that work with distinct noises for each frame. Additionally, while Text2Video-Zero employs a latent shifting method for overall scene movement and mirrors the latent to fill missing regions, our technique uses random sampling on the x and y-axis, providing more concise and coherent videos for camera motion control.

To substantiate our claims, we conducted quantitative experiments comparing the performance of these methods, as detailed in `Table R.1 (e)` and `(f)`. Unfortunately, some methods could not be directly compared due to the unavailability of their code. Nevertheless, our results demonstrate that our method generally outperforms the alternatives. Specifically, we observed significant limitations in Motion-Zero's test-time tuning approach for customized video generation, underscoring the strengths of our method.

We hope this response clarifies the distinctions and advantages of our approach. We appreciate your feedback and look forward to further discussions and evaluations.


# Video metrics

We acknowledge the difficulty in identifying an appropriate metric for customized video generation, as standard metrics like FVD and optical flow error typically require a ground truth video for comparison. In the context of customized subjects, such ground truth data is not available. Consequently, we have employed alternative metrics including CLIP text similarity, CLIP image similarity (between the generated image and the subject image), and DINO image similarity. These metrics have been validated in prior works [1,2,3,4] and provide a meaningful assessment of our model's performance.

However, we recognize the value of using FVD and optical flow error in situations where a ground truth or standard reference can be utilized, such as in camera motion control. As demonstrated in `Table 2` of the main paper, we have reported results using optical flow error. To further address the reviewers' concern, we have also calculated FVD by randomly selecting 1000 videos from the MSRVTT dataset. The results, presented in `Table R.1 (e)`, indicate that our method significantly outperforms existing camera control methods, including some that are training-based.

Additionally, we conducted human preference studies to capture the intuitive perception of the generated videos, as shown in `Figure 8` of the main paper. We believe these comprehensive evaluations—combining quantitative metrics and subjective assessments—provide a robust understanding of our method's quality and effectiveness.


[1] Ruiz, Nataniel, et al. "Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation." CVPR 2023.

[2] Wei, Yujie, et al. "Dreamvideo: Composing your dream videos with customized subject and motion." CVPR 2024.

[3] Jiang, Yuming, et al. "Videobooth: Diffusion-based video generation with image prompts." CVPR 2024.

[4] Wang, Zhao, et al. "Customvideo: Customizing text-to-video generation with multiple subjects." arXiv 2024.

Meta-review: This paper, which introduces a unified video generation framework enabling control over motion and camera movements, has received 1 accept, 1 weak accept, 2 borderline accepts, and 1 reject. Reviewers aAio, ZPXz, N7np, and yPkE support the submission, praising the method as intuitive, sensible, and well-formulated. They also note that the experimental results are impressive compared to the state-of-the-art. Concerns raised, such as the zoom in/zoom out effects, have been satisfactorily addressed in the rebuttal. While the camera control is described as relatively coarse, it is deemed acceptable limitation. Reviewer q2XJ expressed a negative view, citing issues with the suppression formula as a basis for rejection, but the authors appear to have adequately addressed this concern in their rebuttal. Considering these factors, the AC recommends acceptance of the paper.
Decision: Accept (spotlight)