Zero-Shot Subject-Driven Video Customization with Precise Motion Control

Authors: Yujie Wei, Shiwei Zhang, Hangjie Yuan, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng, Feng Liu, Zhizhong Huang, Jiaxin Ye, Yingya Zhang, Hongming Shan
Primary area: generative models
Keywords: Video Generation, Customized Generation

Abstract:
Recent advances in customized video generation have enabled users to create videos tailored to both specific subjects and motion trajectories. However, existing methods often require complicated test-time fine-tuning and struggle with balancing subject learning and motion control, limiting their real-world applications. In this paper, we present $\textbf{DreamCustomizer}$, a zero-shot video customization framework capable of generating videos with a specific subject and motion trajectory, guided by a single image and a bounding box sequence, respectively, and without the need for test-time fine-tuning. Specifically, we introduce reference attention, which leverages the model’s inherent capabilities for subject learning, and devise a mask-guided motion module to achieve precise motion control by fully utilizing the robust motion signal of box masks derived from bounding boxes. While these two components achieve their intended functions, we empirically observe that motion control tends to dominate over subject learning. To address this, we propose two key designs: $\textbf{1)}$ the masked reference attention, which integrates a blended latent mask modeling scheme into reference attention to enhance subject representations at the desired positions, and $\textbf{2)}$ a reweighted diffusion loss, which differentiates the contributions of regions inside and outside the bounding boxes to ensure a balance between subject and motion control. Extensive experimental results on a newly curated dataset demonstrate that DreamCustomizer outperforms state-of-the-art methods in both subject customization and motion control. The dataset, code, and models will be made publicly available.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729472345062

thread_type:
official_review

<thread_object>
creator: reviewer_xvgx

<content>
summary:
The paper introduces a video customization framework, DreamCustomizer, which can generate videos based on a specific subject and motion trajectory.
The framework incorporates two primary components: reference attention for subject learning and a mask-guided motion module for precise motion control.
Experiments demonstrate that DreamCustomizer outperforms state-of-the-art methods in both subject customization and motion control.

strengths:
- The paper is well-written and the figures are properly plotted. Both of them make readers easy to understand the proposed method.

- The experiments are extensive, covering the quantitative and qualitative comparisons with the state-of-the-arts, human evaluation, testing under different conditioning scenarios, and ablation study.

- The authors provide a lot of generated videos, which can help the reader better understand the quality and the potential problems of the proposed method.

- The model is an optimization-free method, which does not require fine-tuning during inference stage.

- The motion control of the proposed model is precise and impressive.

weaknesses:
- Low visual quality:
    - The generated videos have significant artifacts. Did the authors try a better and more recent base model, such as VideoCrafter2?
    - With this video quality, it is hard to judge the fidelity of subject customization.

- Weakness of reconstruction-based method:
    - The model adopts a reconstruction-based training strategy, which collects the input conditions (reference image and bounding boxes) from the video and learn to reconstruct the video with these inputs.
    - However, such training strategy is well-known by worse performance when the user inputs prompt and image from different sources.
    - For example, in the "a corgi is swimming" sample, the model seems to directly copy and paste the input standing corgi and make it floating on the water without introducing reasonable pose or appearance change.
    - Have the authors tried to solve this problem? Applying some image augmentations could alleviate the problem.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1729897497342

thread_type:
official_review

<thread_object>
creator: reviewer_szvt

<content>
summary:
The paper introduces DreamCustomizer to generate video with precise motion control and specific subject without the need for fine-tuning during inference. DreamCustomizer leverages "reference attention" and a "mask-guided motion module" to achieve accurate video customizations, controlled by a single subject image and bounding box sequences. The methodology uses masked reference attention and a reweighted diffusion loss to balance subject learning and motion control. The paper claims superior performance over state-of-the-art methods by a new dataset and extensive quantitative and qualitative evaluations.

strengths:
1. The paper distinguishes and mitigates the challenge of motion control dominance by introducing masked reference attention and reweighted diffusion loss, successfully balancing subject fidelity with motion accuracy.

2. The newly curated, diverse dataset provides comprehensive annotations, facilitating training and evaluation for subject and motion control and supporting future research in video customization.

3. With extensive quantitative and qualitative evaluations, the paper demonstrates the effectiveness of DreamCustomizer against state-of-the-art methods.

weaknesses:
1. While DreamCustomizer introduces elements like reference attention and reweighted diffusion loss, these techniques lack substantial novelty. Reference attention, for instance, has been well-studied in prior works such as "StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation," where it has been used to effectively maintain subject consistency. Similarly, the reweighted diffusion loss does not introduce a novel approach to balancing subject fidelity with motion control, as similar weighting techniques have been explored in generative models. 

2. DreamCustomizer requires bounding boxes as inputs for motion guidance, similar to control techniques seen in previous works like MotionBooth. This limitation reduces its capability for more complex and high quality video generation tasks and the proposed method is incremental to resolve this problem.

3. Moreover, the presented video quality is not convincing to demonstrate that these proposed components are instrumental in advancing controllable video generation.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730219880271

thread_type:
official_review

<thread_object>
creator: reviewer_b9q7

<content>
summary:
The paper presents a zero-shot approach to subject-driven video generation with controlled motion, showing potential for reducing test-time fine-tuning overhead. The reference attention mechanism and mask-guided motion control provide innovations over previous methods by enhancing the fidelity of subject appearance within controlled bounding boxes.

strengths:
- The DreamCustomizer framework proposed in this paper does not require fine-tuning during inference, and can directly customize the target subject and motion trajectory in a zero-shot situation. 

- This generation framework that does not require fine-tuning improves the efficiency of video generation and is conducive to a wide range of practical applications.

- Users only need to provide the subject image and a set of bounding box sequences to generate a custom video, without the need for complex inference stage debugging.

weaknesses:
- While DreamCustomizer claims tuning-free inference, the paper acknowledges the challenge of decoupling camera movement from object motion, leading to camera drift in certain contexts.

-  DreamCustomizer is designed for single-subject customization and does not handle multi-subject videos, which is highlighted as a limitation but without proposed extensions.

- The effectiveness of motion control relies heavily on the precision of bounding box annotations, making the system vulnerable to errors if box tracking is inconsistent.

- Missing some important previous works:

[1] MotionFollower: Editing video motion via lightweight score-guided diffusion. (2024).

[2] FaceChain-ImagineID: Freely crafting high-fidelity diverse talking faces from disentangled audio. CVPR 2024

[3] MotionEditor: Editing video motion via content-aware diffusion. CVPR 2024

[4] Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony.  (2024).

questions:
- The paper mentions data filtering steps, but additional details on the refinement of bounding boxes and control signals could provide readers with greater insight into handling imperfect data in training.

- Given the limitations in complex motions and single-subject restriction, it's recommended that the paper discuss specific applications that could benefit from these features as they currently exist.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730440256200

thread_type:
official_review

<thread_object>
creator: reviewer_gedd

<content>
summary:
The paper introduces DreamCustomizer, a video customization model capable of generating videos with a specific subject and motion trajectory. Technically, DreamCustomizer incorporates reference attention into the T2V model to learn the appearance information of the subject, and employs a mask-guided motion module to achieve motion control based on a sequence of bounding boxes. Additionally, the paper proposes masked reference attention and reweighted diffusion loss to emphasize the model's focus on learning from the subject input. Extensive experiments conducted on a newly curated dataset demonstrate the effectiveness of DreamCustomizer for subject customization and motion control.

strengths:
S1: The paper introduces DreamCustomizer, a video generation method capable of controlling both the appearance of the generated subject and the motion trajectory. Additionally, it proposes two strategies, masked reference attention and reweighted diffusion loss, to enhance the model's focus on learning subject appearance. 

S2: DreamCustomizer constructs a dataset from WebVid-10M for training, enabling subject customization and motion control during inference without the need for fine-tuning.

S3: The paper presents good qualitative and quantitative results.

weaknesses:
W1: The description of the dataset used for evaluation in Section 5.1 could benefit from further clarification. I would appreciate it if the authors could specify the total number of subject-BBox pairs. Additionally, the statement "we design 60 textual prompts for validation" raises a question for me: does this imply that there are only 60 pairs used for evaluation? It seems that each input pair would typically have a corresponding text prompt.

W2：I’m unsure if I missed this detail, but I would like to know whether Tables 2, 3, and 4 are evaluated on the same validation set. If so, I’m curious why the values for DreamCustomizer differ for the same metrics, such as CLIP-T and Temporal consistency, across the three tables.

W3：While the qualitative comparison indicates that DreamCustomizer achieves better mIoU and CD, Figure 10 shows that the model often generates camera movement rather than subject movement. This type of camera movement is not the intended form of motion in this paper and typically results in low CD values. My concern is that the metrics CD and mIoU may not accurately assess the appropriateness of the generated video’s motion (e.g., distinguishing between camera movement and subject movement), potentially reducing the credibility of these metrics.

questions:
Q1: I hope the authors can address the concerns raised in the "Weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732088400013

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Author Rebuttal

comment:
We sincerely thank all the reviewers for their valuable feedback and for recognizing the strengths of our work. 
Specifically, we appreciate their acknowledgment of the effectiveness of the components in the proposed DreamCustomizer (Reviewer gEDD), the impressive motion control capacity (Reviewer XVgX), the practicality of the tuning-free approach (Reviewers b9Q7 and XVgX), the construction of a large-scale, diverse dataset (Reviewers gEDD and szvT), and the comprehensive experiments and ablation studies (Reviewers gEDD, szvT and XVgX).

We would like to clarify our key contributions:
1. We propose a tuning-free video customization framework to control both subject appearance and motion trajectory using the devised reference attention and the mask-guided motion module.
2. We identify the problem of motion control dominance and address it by proposing masked reference attention and reweighted diffusion loss.
3. We curate a large, comprehensive, and diverse video dataset to facilitate future research in video customization. 

Moreover, we comprehensively address the concerns of reviewers in detail within our subsequent responses, and have revised the manuscript accordingly with changes reflected in blue.

These changes include more details on the evaluation setting and data annotation, added experimental results on subject motion evaluation, extended discussions on applications and future work, and added related works.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732802882349

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Better Results based on VideoCrafter2

comment:
We sincerely appreciate the reviewers’ constructive feedback, which has been invaluable in improving our work. We also express our gratitude for their patience and time in reviewing our paper.

To generate higher quality videos, we have reproduced the training process of VideoCrafter2, a recent advanced model, and trained a preliminary version based on VideoCrafter2 with a larger resolution of 512x320, compared to the 448x256 resolution of our trained ModelScopeT2V. 

The generated results are included in the revised paper in Appendix A.4 and A.5, and the corresponding videos have been added to the supplementary materials. We kindly invite the reviewers to review the videos located in `./rebuttal_videos`, which should address the concerns regarding the video quality.

The visual results in Fig. 12 of the revised paper indicate that replacing the backbone with VideoCrafter2 further improves video quality, encompassing both aesthetics and clarity. We also provide additional visual results based on VideoCrafter2 in Fig. 13 to showcase more generated high-quality videos.

These experimental results validate the transferability and generalization of our method across different models. In fact, our DreamCustomizer represents a novel zero-shot video customization paradigm, and we anticipate that it will function independently of specific foundational models. We also believe that our method could yield even better results when applied to more powerful models.  

Once again, we thank all reviewers for their invaluable suggestions, which have greatly strengthened our approach.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work presents DreamCustomizer, a video generation method capable of controlling both subject appearance and motion trajectory in a tuning-free manner. Despite the authors' efforts to address the reviewers' concerns through clarifications and additional experiments, the issues related to limited novelty, low visual quality generation and limited motion control remain unresolved. Therefore, this paper, in its current form, does not meet the standards for acceptance on ICLR. The authors are encouraged to address the mentioned limitations and further strengthen this work.