MotionBooth: Motion-Aware Customized Text-to-Video Generation

Authors: jianzong  wu
xiangtai  li
yanhong  zeng
jiangning  zhang
qianyu  zhou
yining  li
kai  chen
yunhai  tong
  pku
s-lab  ntu
shanghai ai laboratory
  zju
  sjtu
attn  cross
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:37:51

Abstract: In this work, we present MotionBooth, an innovative framework designed for animating customized subjects with precise control over both object and camera movements. By leveraging a few images of a specific object, we efficiently fine-tune a text-to-video model to capture the object's shape and attributes accurately. Our approach presents subject region loss and video preservation loss to enhance the subject's learning performance, along with a subject token cross-attention loss to integrate the customized subject with motion control signals. Additionally, we propose training-free techniques for managing subject and camera motions during inference. In particular, we utilize cross-attention map manipulation to govern subject motion and introduce a novel latent shift module for camera movement control as well. MotionBooth excels in preserving the appearance of subjects while simultaneously controlling the motions in generated videos. Extensive quantitative and qualitative evaluations demonstrate the superiority and effectiveness of our method. Models and codes will be made publicly available.

Main paper: 
 introduction: 
 Generating videos for customized subjects, such as specific scenarios involving a particular dog's type or appearance, has gained research attention [55, 25, 42] . This customized generation field originated from text-to-image (T2I) generation methods, which learn a subject's appearance from a few images and generate diverse images of that subject [12, 42, 30] . Following them, subject-driven text-to-video (T2V) generation has seen increasing interest, which has found a wide range of applications in personal shorts or film production [55, 25, 42, 57, 13] . Can you imagine your toy riding along the road from a distance to the camera or your pet dog dancing on the street from the left to the right? However, rendering such lovely imaginary videos is a challenging task. It often involves subject learning and motion injection while maintaining the generative capability to generate diverse scenes. Notably, VideoBooth [25] trains an image encoder to embed the subject's appearance into the model, generating a short clip of the subject. However, the generated videos often display minimal or missing motion, resembling a "moving image." This approach underutilizes the motion diversity of pre-trained T2V models. Another line of works [57, 61, 13] fine-tunes the customized model on specific videos, requiring motion learning for each specific camera or subject motion type. Their pipelines restrict the type of motion and require fine-tuning a new adapter for each motion type, which is inconvenient and computationally expensive.The key lies in the conflict between subject learning and video motion preservation. During subject learning, training on limited images of the specific subject significantly shifts the distribution of the base T2V model, leading to significant degradation (e.g., blurred backgrounds and static video). Therefore, existing methods often need additional motion learning for specific motion control. In this paper, we argue that the base T2V model already has diverse motion prior, and the key is to preserve video capability during subject learning and digging out the motion control during inference.To ensure subject-driven video generation with universal and precise motion control, we present Mo-tionBooth, which can perform motion-aware customized video generation. The videos generated by MotionBooth are illustrated in Fig. 1 . MotionBooth can take any combination of subject, subject motion, and camera motion as inputs and generate diverse videos, maintaining quality on par with pre-trained T2V models.MotionBooth learns subjects without hurting video generation capability, enabling a training-free motion injection for subject-driven video generation. First, during subject learning, we introduce subject region loss and video preservation loss, which enhance both subject fidelity and video quality. In addition, we present a subject token cross-attention loss to connect the customized subject with motion control signals. During inference, we propose training-free techniques to control the camera and subject motion. We directly manipulate the cross-attention maps to control the subject motion. We also propose a novel latent shift module to govern the camera movement. It shifts the noised latent to move the camera pose. Through quantitative and qualitative experiments, we demonstrate the superiority and effectiveness of the proposed motion control methods, and they can be applied to different base T2V models without further tuning.Our contributions are summarized as follows: 1) We propose a unified framework, MotionBooth, for motion-aware customized video generation. To our knowledge, this is the first framework capable of generating diverse videos by combining customized subjects, subject motions, and camera movements as input. 2) We propose a novel loss-augmented training architecture for subject learning. This includes subject region loss, video preservation loss, and subject token cross-attention loss, significantly enhancing subject fidelity and video quality. 3) We develop innovative, training-free methods for controlling subject and camera motions. Extensive experiments demonstrate that MotionBooth outperforms existing state-of-the-art video generation models. 
related work: 
 Text-to-video generation. T2V generation leverages deep learning models to interpret text input and generate corresponding video content. It builds upon earlier breakthroughs in text-to-image generation [44, 41, 19, 21, 37, 48, 60, 62] but introduces more complex dynamics by incorporating motion and time [46, 20, 18, 2, 68, 59] . Recent advancements particularly leverage diffusion-based architectures. Notable models such as ModelScopeT2V [51] and LaVie [54] integrate temporal layers within spatial frameworks. VideoCrafter1 [6] and VideoCrafter2 [7] address the scarcity of video data by utilizing high-quality image datasets. Latte [36] and W.A.L.T [14] adopt Transformers as backbones [49] . VideoPoet [29] explores generating videos autoregressively to produce consistent long videos. Recent Sora [3] excels in generating videos with impressive quality, stable consistency, and varied motion. Despite these advancements, controlling video content through text alone remains challenging, highlighting a continuing need for research into more refined control signals.Customized generation. Generating images and videos with customized subjects is attracting growing interest. Most works concentrate on learning a specific subject with a few images from the same subject [23, 8, 10, 43, 47, 39] or specific domains [15, 16, 50] . Textual Inversion [12] proposes to train a new word to capture the feature of an object. In contrast, DreamBooth [42] fine-tunes the whole U-Net, resulting in a better IP preservation ability. Following them, many works explore more challenging tasks, such as customizing multiple objects [30, 55, 33, 5] , developing common subject adapter [58, 25, 65, 11, 67] , and simultaneously controlling their positions [11, 33] . However, the customization of video models from a few images often results in overfitting. The models fail to incorporate significant motion dynamics. A recent work, DreamVideo [57] , addresses this by learning specific motion types from video data. Yet, this method is restricted to pre-defined motion types and lacks the flexibility of text-driven input. In contrast, our work introduces MotionBooth to control both the subject and camera motions without needing pre-defined motion prototypes.Motion-aware video generation. Recent works explore incorporating explicit motion control in video generation. This includes camera and object motions. To control camera motion, existing works like AnimateDiff [13] , VideoComposer [53] , CameraCtrl [17] , Direct-A-Video [66] , and MotionCtrl [56] design specific modules to encode the camera movement or trajectory. These models usually rely on training on large-scale datasets [1, 9] , leading to high computational costs.In contrast, our MotionBooth framework builds a training-free camera motion module that can be easily integrated with any T2V model, eliminating the need for re-training. For object motion control, recent works [63, 31, 32, 24, 66, 4, 69, 27, 22] propose effective methods to manipulate attention values during the inference stage. Inspired by these approaches, we connect subject text tokens to the subject position using a subject token cross-attention loss. This allows for straightforward control over the motion of a customized object by adjusting cross-attention values.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our method in Section 4.5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 Task formulation. We focus on generating motion-aware videos featured by a customized subject.To customize video subjects, we fine-tune the T2V model on a specific subject. This process can be accomplished with just a few (typically 3-5) images of the same subject. During inference, the fine-tuned model generates motion-aware videos of the subject. The motion encompasses both camera and subject movements, which are freely defined by the user. For camera motion, the user inputs the horizontal and vertical camera movement ratios, denoted as c cam = [c x , c y ]. For subject motion, the user provides a bounding box sequence [B 1 , B 2 , ..., B L ] to indicate the desired positions of the subject, where L represents the video length. Each bounding box specifies the x-y coordinates of the top-left and bottom-right points for each frame. By incorporating these conditional inputs, the model is expected to generate videos that include a specific subject, along with predefined camera movements and subject motions.Overall pipeline. The overall pipeline of MotionBooth is illustrated in Fig. 2 . During the training stage, MotionBooth learns the appearance of the given subject by fine-tuning the T2V model. To prevent overfitting, we introduce video preservation loss and subject region loss in Section 3.2. Additionally, we propose a subject token cross-attention (STCA) loss in Section 3.2 to explicitly connect the subject tokens with the subject's position on cross-attention maps, facilitating the control of subject motion. Camera and subject motion control are performed during the inference stage. We manipulate the cross-attention maps by amplifying the subject tokens and their corresponding regions while suppressing other tokens in Section 3.3. This ensures that the generated subjects appear in the desired positions. By training on the cross-attention map, the STCA loss enhances the subjects' motion control. For camera movement, we introduce a novel latent shift module to shift the noised latent directly, achieving smooth camera movement in the generated videos in Section 3.4. Given a few images of a subject, previous works have demonstrated that fine-tuning a diffusion model on these images can effectively learn the appearance of the subject [42, 23, 8, 10, 43, 47] . However, two significant challenges remain. First, due to the limited size of the dataset, the model quickly overfits the input images, including their backgrounds, within a few steps. This overfitting of the background impedes the generation of videos with diverse scenes, a problem also noted in previous works [42, 12] . Second, fine-tuning T2V models using images can impair the model's inherent ability to generate videos, leading to severe background degradation in the generated videos. To illustrate these issues, we conducted a toy experiment. As depicted in Fig. 3 , without any modifications, the model overfits the background to the subject image. To address this, we propose computing the diffusion reconstruction loss solely within the subject region. However, even with this adjustment, the background in the generated videos remains over-smoothed. This degradation likely results from tuning a T2V model exclusively with images, which damages the model's original weights for video generation. To mitigate this, we propose incorporating video data as preservation data during the training process.Although training with video data but without subject region loss still suffers from overfitting, our approach, MotionBooth, can generate videos with detailed and diverse backgrounds.Preliminary. T2V diffusion models learn to generate videos by reconstructing noise in a latent space [42, 30, 55, 12] . The input video is first encoded into a latent representation z 0 . Noise ϵ is added to this latent representation, resulting in a noised latent z t , where t represents the timestamp. This process simulates the reverse process of a fixed-length Markov Chain [41] . The diffusion model ϵ θ is trained to predict this noise. The training loss, which is a reconstruction loss, is given by:EQUATION)where c is the conditional input used in classifier-free guidance methods, which can be text or a reference image. During inference, a pure noise z T is gradually denoised to a clean latent z ′ 0 , where T is the length of the Markov Chain. The clean latent is then decoded back into RGB space to generate the video X ′ .Subject region loss. To address the challenge of overfitting backgrounds in training images, we propose a subject region loss. The core idea is to calculate the diffusion reconstruction loss exclusively within the subject region, thereby preventing the model from learning the background. Specifically, we first extract the subject mask for each image. This can be done manually or through automatic methods, such as a segmentation model. In practice, we use SAM [28] to collect all the masks. The subject region loss is then calculated as follows:EQUATIONwhere M represents the binary masks for the training images. These masks are resized to the latent space to compute the dot product. c i is a fixed sentence in the format "a [V] [class name]," where "[V]" is a rare token and "[class name]" is the class name of the subject [42] . We have found that with the subject region loss, the trained model effectively avoids the background overfitting problem.Video preservation loss. Image customization datasets like DreamBooth [42] and CustomDiffusion [30] provide excellent examples of multiple images from the same subject. However, in the customized video generation task, directly fine-tuning the video diffusion model on images leads to significant background degradation. Intuitively, this image-based training process may harm the original knowledge embedded in video diffusion models. To address this, we introduce a video preservation loss designed to maintain video generation knowledge by joint training with video data.Unlike the class-specific preservation data used in previous works [42, 55] , we utilize common videos with captions denoted as c v . Our experiments in Section 4 demonstrate that common videos are more effective for subject learning and preserving video generation capabilities. The loss function is formulated as follows: Subject token cross-attention loss. To control the subject's motion, we directly manipulate the cross-attention maps during inference. Since we introduce a unique token, "[V]", in the training stage and associate it with the subject, we need to link this special token to the subject's position within the cross-attention maps. As illustrated in Fig. 4 , fine-tuning the model does not effectively connect the unique token to the cross-attention maps. Therefore, we propose a Subject Token Cross-Attention (STCA) loss to guide this process explicitly. First, we extract the crossattention map, A, at the tokens "[V] [class name]". We then apply a Binary Cross-Entropy Loss to ensure that the corresponding attention map is larger at the subject's position and smaller outside this region. This process incorporates the subject mask and can be expressed as:EQUATIONEQUATIONDuring training, the overall loss function is defined as:EQUATIONwhere λ 1 and λ 2 are hyperparameters that control the weights of the different loss components.We chose bounding boxes as the motion control signal for subjects because they are easy to draw and manipulate. In contrast, providing object masks for every frame is labor-intensive, requiring consideration of the subject's shape transformation between frames. In practice, we find that bounding boxes are sufficient for precisely controlling the positions of subjects. Previous works like GLIGEN [31] attempt to control object positions by training an extra condition module with large-scale image data. However, these training methods fix the models and cannot easily align with customized models fine-tuned for specific subjects. Therefore, we adopt an alternative approach that directly edits the cross-attention maps during inference in a training-free manner [66, 27, 4] . This cross-attention editing method is plug-and-play and can be used with any customized model.In cross-attention layers, the query features Q are extracted from the video latent and represent the vision features. The key and value features K and V are derived from input language tokens. The calculation process of the edited cross-attention layer can be formulated as follows:EQUATIONwhere d is the feature dimension of Q and serves as a normalization term. QK ⊤ √ d is the normalized production between Q and K, representing the attention scores between vision and language features. We manipulate the production by adding a new term αS, where S has positive values on the subject region provided in bounding boxes and large negative values outside the desired positions. α is a hyperparameter to control the editing strength. The editing matrix S is set as follows:EQUATIONwhere i, j, and k indicate the vision token, language token, and frame indexes, respectively. P represents the indexes for subject language tokens in the text prompt. In this work, we choose "[V]" and "[class name]" as subject tokens. The SCTA loss in Section 3.2 binds the two tokens with cross-attention maps. t is the denoising timestamp and τ is a hyperparameter defining a timestamp threshold. Since diffusion models tend to form the approximate object layout in earlier denoising steps and refine the details in later steps [63] , we apply stronger attention amplification in earlier steps and no amplification in later steps. Note that attention suppression outside the bounding box regions persists throughout the generation. |B k | and |Q| are the areas of the box and query, respectively. Following previous works [27, 66] , smaller boxes should have larger amplifications, and we do not apply any editing on the <start> and <end> tokens.Discussion. An important aspect is how the necessary information from other language tokens is integrated into the generated outputs, given that tokens such as verbs and background nouns are assigned minimal values. We propose that this information is extracted through the <start> and <end> tokens. Given that Transformer-based language encoders like CLIP [40] are typically trained on classification tasks, they often encode the overall context of a sentence into these special tokens. Thus, despite the suppression of other tokens during the softmax calculation, the model can still access relevant information about verbs, background elements, and other components necessary for the generation process. To support this explanation, we conducted an experiment in which we examined the softmax outputs using a naive text-to-video pipeline. The results showed that the <start> token consistently held the highest softmax value, close to 1, while the <end> token had the second-largest value. The remaining tokens, including those representing nouns, adjectives, verbs, and conjunctions, were distributed among the remaining softmax values."A waterfall in a beautiful forest with fall foliage"Noised Latent 𝑧𝜎 1Step 1: Shiftx-axis y-axisStep 2: Sample Tokens Simply editing the cross-attention map can efficiently control the motion of the subject. This suggests that the latent can be considered a "shrunk image," which maintains the same visual geographic distribution as the generated images. For camera movement control, an intuitive approach is to directly shift the noised latent during inference based on the camera movement signal c cam = [c x , c y ]. The latent shift pipeline is illustrated in Table 2 . The key challenge with this idea is filling in the missing parts caused by the latent shift (the question mark region in Step 1). To address this issue, we propose sampling tokens from the original noised latent and using them to fill the gap. This is based on the prior knowledge that when a camera moves in a video, the new scene it captures is semantically close to the previous one. For example, in a video with forest scenes, when the camera pans left, it is highly likely to capture more trees similar to those in the original scene. Another assumption is that in a normally angled video, a visual element is more likely to be semantically close to elements along the same x-axis or y-axis rather than other elements. For instance, in the waterfall video in Fig. 5 , trees are at the top and bottom, spreading horizontally, while the waterfall spans the middle x-axis area. Experimentally, we over that sampling tokens horizontally and vertically provides better initialization and results in smoother video transitions. Randomly sampling tokens degrades the generated video quality. The latent shift process for timestamp t can be formulated as follows:EQUATIONwhere h x and h y are sampled tokens along the x and y axes, respectively. Crop(•) removes the tokens outside the camera view after the shift. B is the subject bounding box. We filter out the tokens belonging to the subjects because they are not likely to occur in the new scenes. In addition, to avoid a drastic change in latent in one shift, we spread the latent shift over multiple timestamps, with each step only shifting a small number of tokens. Note that the latent shift needs to be applied after the subject's approximate layout is fixed but before the video details are completed. We set a pair of hyperparameters σ 1 and σ 2 . The latent shift only applies in the timestamp range[σ 1 , σ 2 ].4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the social impacts of MotionBooth in Appendix A.5. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental setup: 
 Datasets. For customization, we collect a total of 26 objects from DreamBooth [42] and CustomDiffusion [30] . These objects include pets, plushies, toys, cartoons, and vehicles. To evaluate camera and object motion control, we built a dataset containing 40 text-object motion pairs and 40 text-camera motion pairs, ensuring that the camera and object motion patterns are consistent with the text prompts. This dataset evaluates the videos generated for each subject in various scenarios and motions.Implementation details. We train MotionBooth for 300 steps using the AdamW optimizer, with a learning rate of 5e-2 and a weight decay of 1e-2. We collect 500 preservation videos from the Panda-70M [9] training set, chosen randomly. Each batch consists of one batch for images and one for videos, with batch sizes equal to the number of training images and 1 for images and videos, respectively. The loss weight parameters λ 1 and λ 2 are set to 1.0 and 0.01. We use Zeroscope and LaVie as base models. During inference, we perform 50-step denoising using the DDIM scheduler and set the classifier-free guidance scale to 7.5. The generated videos are 576x320x24 and 512x320x16 for Zeroscope and LaVie, respectively. The training process finishes in around 10 minutes in a single NVIDIA A100 80G GPU. Additional implementation details can be found in Appendix A.1.Baselines. Since we are pioneering motion-aware customized video generation, we compare our methods with closely related works, including DreamBooth [42] , CustomVideo [55] , and DreamVideo [57] .Dreambooth customizes subjects for text-to-image generation. We follow its practice with class preservation images and fine-tune T2V models for generating videos. CustomVideo is a recent video customizing method. We adopt its parameter-efficient training procedure. DreamVideo learns motion patterns from video data. To provide such data, we sample videos from Panda-70M, which are most relevant to the evaluation motions. Since these methods cannot control motions during inference, we apply our camera and object motion control technologies for a fair comparison. Additionally, we compare our camera control method with training-based methods, AnimateDiff [13] and Camer-aCtrl [17] , focusing on camera motion control without subject customization. Since AnimateDiff is trained with only basic camera movement types and cannot take user-defined camera movement c cam = [c x , c y ] as input, we use the closest basic movement type for evaluation. Evaluation metrics. We evaluate motion-aware customized video generation from four aspects: region subject fidelity, temporal consistency, camera motion fidelity, and video quality. 1) To ensure the subject is well-preserved and accurately generated in the specified motion, we introduce region CLIP similarity (R-CLIP) and region DINO similarity metrics (R-DINO). These metrics utilize the CLIP [40] and DINOv2 [38] models to compute the similarities between the subject images and frame regions indicated by bounding boxes. Additionally, we use CLIP image-text similarity (CLIP-T) to measure the similarity between entire frames and text prompts. 2) We evaluate temporal consistency by computing CLIP image features between each consecutive frame. 3) We use VideoFlow [45] to predict the optical flow of the generated videos. Then, we calculate the flow error by comparing the predicted flow with the ground-truth camera motion provided in the evaluation dataset. 4) We randomly select 1000 videos from the MSRVTT dataset [64] , predict their camera motion sequences with VideoFlow [45] , and compute the FVD metric for camera motion control only.Quantitative results. We conduct quantitative comparisons with baseline models on both motionaware customized video generation and camera movement control. The results for motion-aware customized video generation are shown in Table 1 . The results demonstrate that MotionBooth outperforms all baselines on both Zeroscope and LaVie models, indicating that our proposed technologies can be extended to different T2V models. Thanks to the training-free architecture of subject and camera motion control methods, MotionBooth is expected to be adaptable to more open-sourced models in the future, such as Sora [3] . Notably, DreamVideo [57] achieves the second-best scores in T-Cons. and flow error, which aligns with our observation that incorporating video data as auxiliary training data enhances video generation performance. On the other hand, CustomVideo [55] shows inferior performance in R-DINO scores, indicating a poorer ability to generate subjects in given positions. This may be attributed to its approach of only fine-tuning the text embeddings and cross-attention layers of the diffusion models, which is insufficient for learning the subjects.For camera movement control, we compare our method with two training-based methods, AnimateDiff [13] and CameraCtrl [17] . The results are shown in Table 2 . Remarkably, MotionBooth achieves superior results compared to the two baselines with our training-free latent shift module. Specifically, MotionBooth outperforms the recent method CameraCtrl by 0.617, 0.015, and 0.009 in flow error, CLIP-T, and T-Cons. metrics with Zeroscope, and 0.511, 0.004, and 0.024 for the LaVie model. These results demonstrate that the latent shift method is simple yet effective.Qualitative results. The qualitative comparison results for video generation with customized objects and controlled subject motions are presented in Fig. 6 . Our observations reveal that MotionBooth excels in subject motion alignment, text prompt alignment, and overall video quality. In contrast, We also conduct qualitative experiments focused on camera movement control, with results shown in Fig. 7 . AnimateDiff, limited to basic movements, does not support user-defined camera directions.Although the CameraCtrl method can accept user input, it generates videos with subpar aesthetics and objects that exhibit flash movements. In contrast, our MotionBooth model outperforms both the Zeroscope and Lavie models. The proposed latent method generates videos that adhere to user-defined camera movements while maintaining time consistency and high video quality.Training technologies. We analyze the technologies proposed during the subject learning stage.The ablation results are shown in Table 3 . Clearly, without the proposed modules, the quantitative metrics drop accordingly. These results demonstrate that the proposed subject region loss, STCA loss, and video preservation loss are beneficial for subject learning and generating motion-aware customized videos. Specifically, the R-DINO metric decreases significantly by 0.256 without the subject region loss, highlighting its core contribution in filtering out image backgrounds during training. Additionally, the "w/ class video" experiment, which uses class-specific videos instead of randomly sampled common videos, yields worse results. This approach restricts the scenes and backgrounds in class-specific videos, hindering the models' ability to generalize effectively. Figure 8 : Human preference study. Our Motion-Booth achieves the best human preference scores in all the evaluation aspects.To evaluate our approach to understanding user preferences, we conducted a user study experiment. We collected 30 groups of videos generated by MotionBooth and baseline methods.We then asked 7 colleagues to select the best videos based on the following criteria: subject motion alignment, camera movement alignment, subject appearance alignment, and temporal consistency. For each group of videos, the annotators selected only the best one. For the subject appearance alignment, the annotators were provided with corresponding subject images. As shown in Fig. 8 , MotionBooth was the most preferred method across all models and evaluation aspects, particularly in subject appearance alignment. These results further demonstrate the effectiveness of our method.In Fig. 9 , we illustrate several failure cases of MotionBooth. One significant limitation of Motion-Booth is its struggle with generating videos involving multiple objects. As shown in Fig. 9 (a), the subject's appearance can sometimes merge with other objects, resulting in visually confusing outputs. This issue might be resolved by incorporating advanced training technologies for multiple subjects. Another limitation is the model's capability to depict certain motions indicated by the text prompt. As depicted in Fig. 9 (b), MotionBooth may fail to accurately represent motions that are unlikely to be performed by the subject. For example, it is hard to imagine a scene where a wolf plushie is riding a bike. These failure cases highlight the need for further improvement in the model's subject separation and motion understanding capabilities to enhance the realism and accuracy of the generated videos. Utilizing more powerful T2V models may eliminate these drawbacks. Future work could focus on refining these aspects to address the current limitations and provide more robust performance in complex scenarios. 
conclusion: 
 This paper introduces MotionBooth, a novel framework for motion-aware, customized video generation. MotionBooth fine-tunes a T2V diffusion model to learn specific subjects, utilizing subject region loss to focus on the subject area. The training procedure incorporates video preservation data to prevent background degradation. Additionally, an STCA loss is designed to connect subject tokens with the cross-attention map. During inference, training-free technologies are proposed to control both subject and camera motion. Extensive experiments demonstrate the effectiveness and generalization ability of our method. In conclusion, MotionBooth can generate vivid videos with given subjects and controllable subject and camera motions. temporal cross-attention by scaling the attention maps with a hyper-parameter of less than 1. Directed Diffusion [35] focuses on image generation in a similar manner. Our approach, in contrast, targets only the spatial cross-attention, setting attention values outside the object's bounding box to zero. This targeted manipulation simplifies implementation and significantly enhances the performance in generating motion-aware customized videos.Boximator [52] relies on a training-based technique, requiring box coordinates as inputs to a newly trained self-attention layer. In comparison, our method is training-free, thus providing a more accessible and user-friendly solution for controlling subject and camera motion.Motion-Zero [4] also operates without additional training, but it utilizes a test-time tuning technique to adjust the latent space using cross-attention map loss during the denoising process. This technique, however, increases the generation time and memory requirements considerably-from approximately 15 seconds to several minutes per video. Additionally, our experiments indicate that Motion-Zero often produces poor-quality outputs, with videos collapsing into unrecognizable elements. This outcome is likely due to the detrimental effects of test-time tuning on parameters and latent distributions in customized scenarios. In contrast, our method directly manipulates the cross-attention map, adding only 0.3 seconds to the generation process and consistently yielding more reliable visual outcomes.MotionCtrl [56] uses a training-based approach that requires inputting point trajectories to control camera poses and object motion. Our method does not involve additional training, offering a simpler alternative for subject and camera motion control.Text2Video-Zero [26] builds on a pre-trained text-to-image (T2I) model and extends it to video generation by utilizing consistent noise across frames. However, this approach is unsuitable for text-to-video (T2V) models, which typically use different noise for each frame. Additionally, Text2Video-Zero uses latent shifting for overall scene movement and mirrors latent information to fill in missing regions. In comparison, our approach employs random sampling along the x and y axes, resulting in more concise and coherent video generation for camera motion control.We conducted quantitative experiments to evaluate the performance of these methods, as reported in Table 2 and Table 4b . Some methods are not included due to the unavailability of their code. Nonetheless, our results indicate that our method generally outperforms the other alternatives. In particular, we observed significant limitations in the test-time tuning strategy of Motion-Zero [4] when applied to customized video generation, which further emphasizes the strengths of our approach.Zoom-in/out effect with subject motion control. Our subject motion control technique effectively manages changes in bounding box size, such as gradual enlargement or reduction. Specifically, enlarging the bounding box results in an appropriate scaling of the subject, creating a zoom-in effect in the generated video. Conversely, reducing the bounding box size produces a zoom-out effect. This zoom-in and zoom-out behavior is demonstrated in Fig. 12a , where examples illustrate these effects in response to bounding box adjustments. Fig. 12d shows an example of combining zoom-in and camera motion control, demonstrating the method's flexibility.Large camera motion speeds. We evaluate the performance of our camera motion control technique under varying levels of camera movement intensity. As depicted in Fig. 12b , the method was first tested with a camera movement vector of [c x , c y ] = [-0.5, 0.45], corresponding to a movement of half the video width to the left and nearly half downward. Under these conditions, our method successfully managed the camera movement, producing accurate results. When the camera motion speed was doubled, the model continued to perform well, demonstrating its ability to handle highspeed scenarios. However, at a speed three times the initial setting, i.e., [c x , c y ] = [-1.5, 1.35], the method exhibited limitations, resulting in only a downward tiling effect. These findings indicate that while our method can effectively manage camera movements spanning the entire video width, its performance diminishes under extremely high-speed conditions.Camera motion control with large foreground objects. We evaluate the camera control capability in scenes containing substantial foreground objects. The results, presented in Fig. 12c , show an experiment involving a scene dominated by a large candy house, which nearly occupies the entire frame. Despite the significant presence of the foreground object, our technique effectively managed to pan the camera to the right, suggesting that the method is capable of handling complex scenes with large foreground elements. This robustness is attributed to the method's reliance on latent shifts, which simultaneously move both foreground and background elements, thereby ensuring that the presence of large foreground objects does not significantly impair performance. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We describe the safeguards in Appendix A.5.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use existing assets with properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Training our model by fine-tuning a specific subject takes approximately 10 minutes. For LaVie [54] , the naive text-to-video (T2V) pipeline takes about 15.0 seconds per video. When incorporating subject control, the inference time is 15.3 seconds per video; with camera control, it increases to 20.6 seconds per video; and when applying both camera and subject control, the inference time is 21.5 seconds per video. Comparison with text guidance for camera motion control. we compare our latent shift method with text guidance for controlling camera motion with the results presented in Table 4a and Fig. 12h . Specifically, we utilize simple text prompts, such as "camera pan right" and "camera pan up right," to influence camera movement. The generated videos reflect these instructions to some extent, especially for straightforward motions like panning to the right. However, we observe that such text prompts often lack sufficient specificity, particularly in terms of conveying critical details such as the speed or distance of the camera movement. As a result, this approach yielded sub-optimal outcomes when compared to our proposed method. This indicates that our approach offers a more precise and stable mechanism for controlling camera motion than is possible with text prompts alone.Comparison of subject motion control with more baselines. Table 4b presents quantitative experiments comparing our method with several baseline approaches, including TrailBlazer [34] ,Directed Diffusion [35] , and Motion-Zero [4] . The results demonstrate that our approach generally outperforms these alternatives.Subject motion control hyperparameters. We conduct ablation studies on the hyperparameter for subject motion control, with the results presented in Fig. 13a . We examined the effects of varying the factor α of S and the maximum cross-attention manipulation timestamp τ . The findings indicate that increasing α and extending the controlling steps lead to stronger control. With lower control strengths, the subject does not appear in the desired position, or only part of its body aligns with the intended spot. Conversely, when the control strengths are too high, the generated subjects tend to appear unnaturally square in shape.Latent shift hyperparameters. We experiment with the influence of σ 1 and σ 2 in the latent shift module. The results are shown in Fig. 13b . The results indicate that applying latent shift in the earlier steps of the denoising process results in incomplete camera movement, as evidenced by the trees in the background. Conversely, shifting the latent in the later steps degrades video quality and introduces artifacts, highlighted by the red boxes in the last row. Empirically, setting σ 1 and σ 2 to middle values provides optimal control over camera movement.We conduct an ablation study on the number of preservation videos. As shown in Table 5 , ranging the preservation videos from 100 to 900 does not bring large changes to the quantitative scores. We conclude that the key is to use video data to preserve the video generation ability of the pre-trained T2V models. The number of video data can be flexible. Controlling Single Motion Type. We conduct an additional experiment focusing specifically on the scenario where only subject motion is controlled, and the results are summarized in Table 6a . Notably, we observe that the inclusion of both subject and camera motion controls leads to a slight decrease in some metrics. For instance, using the Zeroscope model, there is a 0.1 drop in the R-CLIP score and a 0.01 decrease in the T-Cons metric when both types of motion are controlled compared to the scenario where only subject motion is controlled. However, there are also instances where metrics such as R-DINO and CLIP-T show a slight improvement under combined control conditions. We consider this trade-off acceptable within the context of motion-aware customized video generation.Qualitative results in Fig. 12g show that our method can successfully handle controlling both subject and camera motion.Masking the training images. We conduct an ablation study to evaluate the effect of masking the background regions directly in pixel space of the training images, compared to masking the diffusion loss. As illustrated in Fig. 12e , directly masking the training images significantly impairs the model's capacity to learn the subject effectively. The quantitative metrics shown in Table 6b indicate a marked decrease in performance when the training images are masked. We hypothesize that this decline arises from the unnatural distribution created by the masked images, which disrupts the learning process. Consequently, we conclude that masking the diffusion loss, rather than masking the training images directly, is a more effective strategy for preserving the integrity of the learned representations.Latent Filling Methods. We test several latent filling approaches and present the qualitative results in Fig. 12f and the quantitative results in Table 6c .Random Init: This method involves filling the hole with random values. Our experiments revealed that this technique leads to severe artifacts due to the disruption of the natural horizontal and vertical distribution of latent pixels, ultimately degrading the overall visual quality.Random Sample: In this approach, values are randomly sampled in the latents. Similar to Random Init, this method produced significant artifacts, leading to poor visual quality in the generated video.Loop: This method reuses the moved-out-of-scene values to fill the missing region, thereby creating a looping background effect. While this technique was found to preserve video quality better than the random methods, it introduced a limitation in terms of flexibility for camera movements, resulting in repetitive looping effects. Therefore, it is not suitable for more diverse camera controls.Reflect: This approach is employed in Text2Video-Zero [26] , where the missing region is filled by reflecting the surrounding content. However, in our Text-to-Video (T2V) scenario, this method collapsed, failing to maintain the desired visual quality.The quantitative results presented in Table 6c corroborate these findings. Random Init and Random Sample lead to significant artifacts, whereas the Loop method provides better visual quality but at the cost of limiting camera movement diversity. The Reflect method, despite its success in other applications, did not yield satisfactory results in our T2V context.In conclusion, our proposed straight-sample method consistently maintained high visual quality without introducing significant artifacts or limiting camera movement flexibility, demonstrating its superiority in maintaining the desired video fidelity.Positive societal impacts. MotionBooth allows for precise control over customized subjects and camera movements in video generation, opening new avenues for artists, filmmakers, and content creators to produce unique and high-quality visual content without extensive resources or professional equipment.Potential negative societal impacts. The ability to generate realistic customized videos could be misused to create deepfakes, leading to potential disinformation campaigns, privacy violations, and reputational damage. This risk is particularly significant in the context of political manipulation and social media. If the underlying models are trained on biased datasets, the generated content might reinforce harmful stereotypes or exclude certain groups. Ensuring diversity and fairness in training data is crucial to mitigate this risk.Mitigation strategies. Developing and adhering to strict ethical guidelines for the use and dissemination of video generation technologies can help mitigate misuse. This includes implementing usage restrictions and promoting transparency about the generated content.We show more qualitative results in Fig. 15 . Justification: We include the main claims in the abstract and introducion.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper, which introduces a unified video generation framework enabling control over motion and camera movements, has received 1 accept, 1 weak accept, 2 borderline accepts, and 1 reject. Reviewers aAio, ZPXz, N7np, and yPkE support the submission, praising the method as intuitive, sensible, and well-formulated. They also note that the experimental results are impressive compared to the state-of-the-art. Concerns raised, such as the zoom in/zoom out effects, have been satisfactorily addressed in the rebuttal. While the camera control is described as relatively coarse, it is deemed acceptable limitation. Reviewer q2XJ expressed a negative view, citing issues with the suppression formula as a basis for rejection, but the authors appear to have adequately addressed this concern in their rebuttal. Considering these factors, the AC recommends acceptance of the paper.

Reviews: review: paper_summary: paper_summary: This paper targets at animating customized subjects with both motion and camera control. To achieve that, they first customize t2v models with subject images finetuning without hurting video generation capability. Then, they propose a training-free approach to manage subject and camera motions. Extensive experiments demonstrate the superiority of their method.
strength_weakness: strength_weakness: •	The subject motion control module is interesting by directly amplifying in the bounding box region and suppressing outside the box region. However, how to define the value of parameter alpha remains unclear. Is it only defined based on experimental performance? And does the authors keep this alpha parameter the same for all customization experiments?

•	The proposed method only control the camera movement in 2D such as up, down. However, camera movement is quite important for 3D, such as moving into the scene, rotation, etc. The proposed camera control module is not able to handle such camera motions and can be a weakness for camera control.
questions: questions: •	For subject motion control module, if the bounding box size is changing, e.g., gradually enlarging bounding box, will the method enlarge the subject accordingly?

•	Subject motion is composed of both position translation and subject movements, such as walking, dancing, jumping, etc. The paper only control the subject position for subject motion. The reviewer wonders is it a bit overclaiming for controlling subject motions, since the subject movements cannot be customized or controlled?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper primarily addresses the problem of motion-aware customized video generation. Traditional customized generation methods either tend to lose the motion information of the video or require additional training of motion modules. To enhance subject fidelity and video dynamics, this paper introduces subject region loss and video preservation loss. Additionally, a training-free subject- and camera-motion control method is proposed, enabling the model to flexibly control the motion information of the video. Extensive experiments demonstrate that the proposed method outperforms existing state-of-the-art methods.
strength_weakness: strength_weakness: 1.	Some details need to be added to more clearly distinguish the differences. For example, in line 169, existing methods use class-specific data, whereas this paper uses common videos. Do the common videos need to include customized subjects? How are these common videos obtained?
2.	The assumptions made by the method, such as sampling operations on the x- and y-axes when controlling camera motion, may limit its effectiveness. For instance, in scenarios with significant changes from left to right or top to bottom, or in close-up videos with large foreground objects, this operation may result in uncoordinated outcomes. However, the paper does not seem to discuss these limitations.
3.	Methods that control subject and camera motion through object trajectories and camera parameters have also been proposed, such as motionCtrl, TrailBlazer, Boximator, and motion-zero. However, this paper does not provide an in-depth discussion or experimental comparison with these methods. Although some of these methods are training-based, it is still worthwhile to explore the existing gaps.
questions: questions: 1.	How many subject samples and/or common videos are used for training each customized model?
2.	The proposed mainly explores the camera motion on panning left or right, is it possible to control other motion types, like zoom-in/out effects? 
3.	what is the difference between the proposed subject region loss, subject token cross attention loss, and those losses in Mix-of-show [1] for customized image generation
ref:
[1] Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
limitations: limitations: 1. This paper needs to discuss more methods addressing the same tasks, like motion control, and specify the differences between the proposed loss functions and those of existing methods.
2. More limitations or failure cases needed to be discussed.
paper_summary: paper_summary: This paper addresses the challenge of how to control the identity and the motion of the subject while generating video, in a text-to-video setup. Specifically, in addition to the standard text prompt, it allows the user to control: the subject's identity by providing a few images of it (e.g. 5 photos of my dog); the subject's position in the video, as provided by a bounding box sequence; and the camera motion, as provided by a sequence of delta-x & delta-y.

Identity preservation is achieved by two changes to the loss function, during fine-tuning on images of the subject. First a "subject region loss" zeros out gradients that lie outside the subject's bounding box during fine-tuning, and second a video preservation loss (similar to the class-specific preservation loss introduced by DreamBooth) to ensures that the ability to generate dynamic motion is not lost due to over-fitting on a few static images. 

Subject and camera motion are controlled at inference time, by modifying the cross-attentional map between the prompt and the latent feature, by shifting the latent feature, respectively.

Finally, a third loss "subject token cross-attention" links the two, enabling the inference motion controls by ensuring that the cross-attention between the subject token (in the text prompt) and the diffusion model's U-Net features is strong where the object should be and weak elsewhere.

These proposed changes are validated quantitatively through several metrics: CLIP and DINO scores between the source images and the desired video locations of the subject, CLIP score between the text prompt and video frames, and a difference in optical flow between the generated video and ideal video (according to the subject and camera motion prompts).

Examples of generated videos are included in the supplemental.
strength_weakness: strength_weakness: These are relatively minor weaknesses in my opinion:

This region metrics measure whether the desired object is translating around the frame as per the bounding box guidance, but because they are image metrics, can't evaluate the actual "animation quality" of the subject. I believe the entire suite of metrics could be optimized by a static image crop of the subject sliding across the video frame according to bounding box motion and flow. Can you confirm whether this is correct?

The set of camera motions that can be models is limited to 2d translations in x and y, missing zooms, rotations, etc.

Although this paper presents its techniques as transferable across many latent diffusion models, they require that the "latent can be considered a "shrunk image," which maintains the same visual geographic distribution as the generated images."  This should generally be true for most latent video diffusion models, but may not work with new latent encoders being developed.
questions: questions: Re "Subject Region Loss": How certain are you that the binary mask, when applied to the latent (rather than the pixels), actually does mask out the background? Given that the latent is produced by a stack of many convolutional or transformer layers, the receptive fields of each latent location almost certainly include information from the nearby background. Have you tried masking the training images directly in pixel space, rather than masking the diffusion loss? How does that compare?

Re "Ablation studies": w/o video and w/ class video give quite-nearly identical numbers. Is this a copy-paste error? If not, it's worth discussing in more detail.

In Figure 3, what is the difference between c, d, e, and f? Are they different samples from the model, or from different models?

Have you tried to achieve the desired camera motions via the text prompt, and considered that as a baseline? I have seen it work well for many basic types of motions in some recent t2v models that you cite.
limitations: limitations: The limitations described in the appendix are important enough that they should be included in the main paper.

Otherwise, yes, they have addressed limitations and impact.
paper_summary: paper_summary: The paper proposes a video generated method - for customizing a subject, along with its motion, provided in the form of bounding boxes, and camera movement provided in the form of camera pose. To do this, the training method has a subject region loss to prevent bias from background and other details in the image, and a video regularization loss function to ensure that video generation properties are not forgotten. At inference, in order to control camera movements, latents are shifted. Cross-attention maps are edited with the provided bounding boxes, to fix the subject motion.
strength_weakness: strength_weakness: My main concerns are with the experiments, which are very non-convincing. While the paper does provide detailed quantitative results for comparisons and ablations on various metrics validating the solution, the qualitative results do not support that.

1. I watched the supplementary video, which has the videos for the results presented in the paper. First of all, the number of qualitative results provided is severely limited, raising questions about cherry picking. Second, the method seems to perform reasonably well on cases that either have camera motion or subject motion - the results look good. In cases involving both camera as well as subject motion, which is the theme of the paper, the results are highly suboptimal - for instance, at 1:48 of the video, the last video showing the dog does not show the subject motion appropriately.

2. I would like to see more detailed results that segregate the various kinds of motion (camera and subject), to see how effective the method is for just camera motion, just subject motion and a combination of both. This will give a clearer picture of where the method is failing.
questions: questions: I am unable to judge the quality of this work with the limited number of qualitative results that have been provided. Moreover, the provided qualitative results are sub-optimal, esp in the case of subject + camera motion, questioning the working of the method. Given that the limit on the size of the supplementary material is considerably large, it would have been nice if the authors had provided more results showcasing their method.

Since the rebuttal does not allow for more qualitative results, I am rejecting the paper at this stage. 

The method, while pretty good, probably needs a little more work to get it working for the subject + camera motion case. I recommend that the authors continue working on it to strengthen the paper.

Post rebuttal: it would have been nice to see more qualitative results and videos, but i understand that is not possible in the 1 page rebuttal pdf. Based on the submitted rebuttal, I am happy to increase the score.
limitations: limitations: Yes
paper_summary: paper_summary: The authors propose MotionBooth, a method to fine-tune a pre-trained text-to-video model on a collection of images of a specific object to enable the model to generate controlled videos of that object. Fine-tuning incorporates three losses: diffusion loss on image data, restricted to the object's region; video preservation loss, i.e diffusion loss on sample video data, to prevent overfitting to static images; subject token cross-attention loss, to improve controllability at generation time. During inference the motion of the object is controlled via editing the cross-attention maps to amplify the attention to the object tokens within the object's bounding boxes. The camera is controlled via shifting the noised latent and filling the new regions with latents sampled from the background of the original noised latent.
strength_weakness: strength_weakness: 1) **Novelty**: 
    - Similar training-free object motion control was proposed in [1] and [2] for image and video models respectively, where cross-attention amplification within object's bounding box was used to control its location in the generated frame. Similar training-free camera motion control was used in [3], where a series of noised latent shifts were used to control the global scene and camera motion. 
Discussion and comparisons with those works are missing in the paper.
    - Some missing references: [4, 5].

2) **Method**: 
    - Based on the formula for subject motion control, the attention is suppressed (set to -infinity) for all query-key pairs other than the queries from the bounding box and the keys from the object tokens. This way it seems that the rest of the prompt is ignored. Moreover, it is stated in the paper that "the attention suppression outside the bounding box regions persists throughout the generation". Details clarifying this are missing in the paper.

3) **Evaluation**: 
    - The paper lacks quantitative evaluation. Comparing CLIP features between the frames doesn't measure temporal consistency. Metrics like FVD, or optical flow warpping error could be better choices for this purpose.
    - Details on how exactly the metrics are calculated are missing. Formal definitions of newly introduced metrics are required for better understanding. E.g. how exactly is the flow error calculated in the case when not all the motion except for the object's motion corresponds to camera motion? Having those would help with understanding the significance of the improvements reported in the tables. 
    - The evaluation dataset consists of limited number of different object classes, mostly dogs, cats and toys. Evaluation on more general collection of object classes would better support the claims made in the paper.
    - More ablation studies would better illustrate the contributions. E.g. what would happen if the missing region in the shifted noised latent was not filled with samples from the same noised latent? Or if camera control like in [3] was used instead?

4) **Results**:
    - Figure 14 shows that bounding boxes often don't fully contain the object of interest. Given that the attention is suppressed outside of bounding boxes, this looks like a flaw in the proposed control and needs to be investigated.

[1] Ma et al., Directed diffusion: Direct control of object placement through attention guidance, AAAI 2024.

[2] Ma et al., TrailBlazer: Trajectory Control for Diffusion-Based Video Generation, arxiv 2024.

[3] Khachatryan et al., Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators, ICCV 2023.

[4] Wang et al., Boximator: Generating rich and controllable motions for video synthesis, arxiv 2024.

[5] Chen et al. Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based
Video Generation, arxiv 2024.
questions: questions: 1) Could the authors discuss the connection between the proposed MotionBooth and the prior work with similar training-free controls? Some ablation studies would be ideal. 

2) Could the authors provide more details about their subject motion control attention suppression technique? Especially regarding the issue with attention to the prompt tokens other than the subject language tokens.

3) The authors claim that their fine-tuning is efficient (line 3 in the abstract). Could the authors provide more details on this? E.g. what is the training time to adapt the model to a new subject?
limitations: limitations: The authors have adequately addressed the limitation and the potential negative societal impact of their work.

Comments: title: Response to Reviewer aAio
Response by Reviewer aAio
Response to Reviewer q2XJ
Response to the authors
Response to Reviewer ZPXz
Please let us know whether we address all the issues
Please let us know whether we address all the issues
Please let us know whether we address all the issues
Response to reviewer q2XJ
Response to the rebuttal
Response to Reviewer N7np
Response to Reviewer  yPkE
Response to rebuttal
comment: Dear reviewer:

Thanks for your comments.

We will update our draft following your comments with the analysis on camera zoom in/out, the limitations of the camera, and the subject motion control ability.

Best regards!

Authors of #9629.
Dear Authors,

Thanks for your detailed responses. I have read through the rebuttal and the rebuttal has solved most of my concerns. Enlarging and shirking subjects performance is good. However, the reviewer thinks the method is a bit limited to coarse camera control and subject position control, and cannot control subject motions such as walking, jumping, etc. Therefore, the reviewer decides to keep the original score. 

Best regards,

Reviewer aAio
Dear Reviewer,

Thank you for your thoughtful engagement with our work and for acknowledging that most of your concerns were carefully addressed in our rebuttal. We sincerely appreciate the time and effort you have invested in reviewing our submission.

We understand that the discussion surrounding the "token suppression" formula may have left some concerns unresolved, and we deeply regret that our initial response did not fully clarify this aspect of our method. With numerous questions from five different reviewers, we recognize that our initial explanation may have been unclear, and we apologize for any confusion this may have caused.

Once you highlighted the misunderstanding in your follow-up comments, we took immediate action to revisit the issue. We carefully re-examined our paper, code, and rebuttal. We are sure that we have made a correct and accurate explanation, which aligns with the method as described in our work. Our intention was to clarify any ambiguities and ensure that our method was clearly understood.

We respect your decision to maintain your original rating, but we hope you will consider that the misunderstanding was promptly addressed during the discussion phase. We believe that our work's overall contributions, including the introduction of a novel framework for motion-aware video generation and innovative techniques for subject and camera motion control, should be taken into account in the final evaluation.

We value your feedback immensely and have learned from this experience. We sincerely hope that our clarification, combined with the broader contributions of our paper, will be considered in your final assessment.

Thank you once again for your time, patience, and thoughtful consideration.

Sincerely,

Authors of #9629
Dear authors,

Thank you for additional clarifications, most of my concerns were carefully addressed in the rebuttal and the comments.

However, unfortunately, the discussion about the suppression formula has left the impression that the authors did not fully understand how and why their method worked, and hence I have to keep my original rating.

Best regards,
Reviewer
Dear Reviewer,

Thank you for your valuable feedback and for taking the time to review our responses. We appreciate your insightful comments, which have helped us further refine our work.

We will update our manuscript to reflect the functional limitations of the camera motion control technique, particularly highlighting that the zoom in/zoom out effect can indeed be achieved by adjusting the size of the bounding box. We acknowledge that while this method is effective, it does not utilize the camera movement control module, which might reduce the perceived contribution of that component. Additionally, we understand your concern that other methods not reliant on bounding boxes might struggle to replicate this effect, and we also recognize the challenges in achieving camera motion control that involves rotating around an object using our current approach.



Regarding your comments on the comparison of losses with the Mix-of-Show code, we have explored this further and would like to provide additional clarification. The code url is at `https://github.com/TencentARC/Mix-of-Show/blob/main/mixofshow/pipelines/trainer_edlora.py`.

1. We have identified that the Mix-of-Show code includes a mask loss and subject token loss, which are used during the training of ED-LoRA. The mask loss, implemented in line 252 of the code, is indeed the same as ours. Both approaches focus the learning process on the subject region, minimizing the influence of background elements in the training data. This similarity is likely due to the shared goal of enhancing subject representation.

2. The subject token loss in Mix-of-Show is implemented between lines 254-259 and further detailed in lines 263-313. This loss is composed of two parts: subject token loss and adjective token loss. Both are MSE losses that penalize attention outside the subject mask while enhancing attention within the subject mask. While similar to our approach, there are notable differences:


- Mix-of-Show calculates both subject and adjective token losses, whereas our method focuses solely on the subject token loss, allowing for a more concentrated emphasis on the subject itself.

- Importantly, in Mix-of-Show, this token loss is disabled by default (as indicated in line 29 of the code). Additionally, the training configuration provided in their `README.md` also disables this loss. Given that this loss is not explicitly mentioned in their paper, it appears that it may have been an ablation experiment rather than a core component of their main experiments.

- Mix-of-Show includes a `reg_full_identity` argument in their training configuration, which is also disabled by default. When disabled, the loss formula is $\mathcal{L}_1 = - \mathbf{M} \log(\mathbf{A})$, while when enabled, it becomes $\mathcal{L}_2 = - \left[\mathbf{M} \log(\mathbf{A}) + (1 - \mathbf{M}) \log(1 - \mathbf{A}) \right]$. Only the latter formula aligns with the STCA loss in MotionBooth.

- Furthermore, the comment in line 30 of the Mix-of-Show code suggests that the choice of loss formula may vary depending on the subject (e.g., "Thanos" vs. a real person), while our approach consistently applies the loss across all our experiments.


We hope this detailed explanation addresses your concerns. We will incorporate these comparisons and clarifications into our paper to provide a comprehensive discussion of the similarities and differences between our approach and that of Mix-of-Show.

Thank you once again for your thoughtful review.

Best regards,
Authors of #9629
I thank the authors for their thorough rebuttal to my and other reviewers' questions. Your answers helped me better understand your method and clarified several of my concerns. However, some of my main concerns remain.

I was pleasantly surprised to see that the zoom in/zoom out effect could be achieved by enlarging or reducing the bounding box. However, this effect is not implemented by the camera movement control module, which diminishes the contribution of the camera movement control. Other methods that do not rely on bounding boxes might not be able to achieve this effect. Additionally, I feel that this approach may also struggle to achieve camera motion control that involves rotating around an object.

Additionally, even though the Mix-of-Show paper does not mention it, its code uses an attention regularization loss that is quite similar to the subject token cross-attention loss in this paper. It would be helpful if the paper could also explore the effects and differences between these two implementations. Of course, since the Mix-of-Show paper does not explicitly state this, it will not be a factor in my scoring.

In summary, I appreciate the completeness of the paper, but the innovation seems somewhat limited to me. Nonetheless, I still hold a positive view toward the paper's acceptance, so I will maintain my score.
Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and a PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider the raising score after we address all the issues.

If you still have more questions and concerns, please comment here. We will reply it as soon as possible. 

Thank you
Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and also more results are shown in PDF file.
Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you
Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you
Dear reviewer:

Thanks for your comments. Here are our reply for your questions.

# Details about the expriments in `Table R.1 (f)`

Thank you for your question. In Table R.1 (f), all the baseline experiments are conducted using the Zeroscope base model for consistency across the experiments. Additionally, the subject learning process of MotionBooth is applied uniformly to all the baseline models to ensure a fair and direct comparison.


# The reason why the reflect method fails

Thank you for your question. The discrepancy between our results and those reported in the Text2Video-Zero paper can be attributed to the difference in how the initial latent $z_T$ is handled in the two approaches.

In the Text2Video-Zero paper, the authors utilize the same initial random latent $z_T$ for all frames of the video, which is then denoised with a few steps to $z_t$. The reflect latent shift is applied to $z_t$, and noise is added back to generate $\hat{z}_T,$ which serves as the starting latent for denoising the video frames. This method works well within their framework because they extend a pre-trained text-to-image (T2I) model for video generation, allowing them to maintain consistency across frames by using the same $z_T$ throughout the process.

However, our model is designed for text-to-video (T2V) generation, where the initial latent $z_T$ is randomly initialized for each frame. This approach is crucial for capturing the frame-by-frame variations necessary in video generation. When we applied the reflect method under these conditions—where $z_T$ is distinct for each frame—it resulted in collapsed outputs. This suggests that the reflect latent shift method struggles when applied to distinct latents for each frame, which is a key requirement in our T2V model.

Furthermore, we also tested using the same latent initialization ($z_T$) across all frames in our model, similar to the method used in Text2Video-Zero. Unfortunately, this approach also led to collapsed results, even when there were no subject or camera motion controls in place. This is because the pre-trained T2V model requires distinct initialized latent for each frame. This reinforces our conclusion that the reflect method is not suitable for models that require distinct latents for each frame, as is the case in our T2V framework.

We hope this explanation clarifies why the reflect method did not work in our scenario despite its effectiveness in the Text2Video-Zero paper.


# Re-clarification about the suppression formula

Thank you for your insightful question and for giving us the opportunity to clarify our approach. We apologize for any confusion caused by our previous explanation. After thoroughly reviewing our code and methodology, we confirm that Equation 7 in the paper is correct.

To clarify, our motion control module performs suppression across all denoising steps, but this suppression is only applied outside the subject's bounding box region. Within the subject region, amplification is applied during the earlier steps, while in the later steps, there is neither amplification nor suppression within this region.

Regarding the suppression of tokens, the language tokens, except for the subject tokens (e.g., verbs, background nouns), are indeed effectively ignored across the denoising process because they are assigned $-\infty$ values. This stark difference in token values is intentional, as it allows us to strongly amplify the motion control of the subject.

This raises the question of how the necessary information from other language tokens is incorporated into the generated results. We believe that this information is retrieved through the <start> and <end> tokens. As mentioned in lines 216-217 of the paper, we do not perform any attention score editing for these two tokens. This means they can share the softmax outputs from the subject tokens since they both have finite values. Transformer-based language encoders, such as CLIP or BERT, tend to encode the overall information of a sentence into these special tokens, given their training with classification tasks. As a result, even though other tokens are ignored in the softmax function, the model can still retrieve the necessary information about verbs, backgrounds, and other elements for the generated results.

To support this claim, we conducted a toy experiment where we printed out the softmax outputs in a vanilla text-to-video pipeline using the Zeroscope model. We observed that the softmax value of the <start> token is the highest, nearing 1, while the <end> token has the second-largest value. The remaining tokens, including nouns, adjectives, verbs, and conjunctions, share the remaining softmax values.

We hope this explanation addresses your concerns, and we are happy to provide further clarification if needed.


Best regards!

Authors of #9629.
Dear authors,

Thank you for your detailed rebuttal.

However, I still have some concerns and would keep the same rating for now.

1) Could you clarify how the comparisons in table R.1 (f) are done? E.g. are the backbones the same? Is the subject learning performed for all the variants? 

2) Do you have any explanation why the reflect method collapsed in your case, while it seems to work reasonably well in the Text2Video-Zero paper?

3) Regarding the suppression formula. So is the equation 7 in the paper incorrect? In the rebuttal you are saying that all the tokens are suppressed to $-\infty$ for the later denoising steps, while in the equation 7 the attention edit for the subject tokens is 0. Note that if all the tokens are suppressed to $-\infty$ then the attention is indeed equivalent to taking the average of all the values, while in the case when there are any finite pre-softmax values in the attention matrix, only those will be weighted despite the normalization.

Best regards, Reviewer
Dear reviewer:

Thanks for your comments.

We will update our draft following your comments with the analysis on camera zoom in/out, text-based camera control, and the exploration of masking in pixel space.

Best regards!

Authors of #9629.
Thank you for including the many clarifications, and the additional analyses. In particular, I appreciate the examples demonstrating camera zoom in/out, text-based camera control, and the exploration of masking in pixel space.

I have read through the other review and authors responses, and believe all of my questions and concerns have been addressed.
Dear reviewer:

Thanks for your comments and increasing the score for our work.

We will update our draft following your comments with more qualitative results and comparison.

Moreover, we believe our method can also be applied with stronger text-to-video base models. We will add these later once the base models are open-sourced.

Best regards!

Authors of #9629.
Dear authors,

Thanks for your very detailed rebuttal.

I appreciate the experiments segregating the various types of motion and the additional qualitative results. They look good! 

It would have been nice to see videos and more qualitative results, which are highly limited, but I understand that this is not possible at the post submission stage in a 1 page rebuttal.

I am happy to increase my score based on the rebuttal.

Thanks,
Reviewer