MotionClone: Training-Free Motion Cloning for Controllable Video Generation

Authors: Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, Yi Jin
Primary area: generative models
Keywords: controllable video image, viusal representation, text-to-video generation

Abstract:
Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, we propose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. 
Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730388232892

thread_type:
official_review

<thread_object>
creator: reviewer_wtap

<content>
summary:
The paper introduces MotionClone, a training-free framework designed for motion cloning in controllable video generation. Unlike traditional methods that require tailored training or fine-tuning, MotionClone leverages the temporal attention mechanism within video generation models to capture and clone motion from reference videos. This approach allows for detailed motion synthesis while minimizing dependencies on the structural components of the reference video, enabling flexible motion cloning across various scenarios. The framework is versatile, supporting tasks like text-to-video (T2V) and image-to-video (I2V), and it has been shown to outperform existing methods in terms of motion accuracy and text alignment.

strengths:
1. MotionClone eliminates the need for complex training processes, making it more accessible and efficient.
2. The framework supports multiple video generation tasks, including T2V and I2V, demonstrating its broad applicability.
3. Enhanced Motion Accuracy: MotionClone outperforms existing methods in capturing and cloning localized object motions with higher fidelity.

weaknesses:
The core idea of this paper, i.e., achieving motion cloning by constraining the temporal attention map, has already appeared in previous work, specifically in UniEdit [1] and MotionMaster [2]. The authors should acknowledge this prior work as the source rather than presenting it as an original contribution.

The primary technical distinction from [1][2] is the proposal to use a sparse temporal mask 
M rather than constraining all temporal attention maps. However, this choice lacks motivation or insight, which makes its effectiveness seem questionable.

Although the authors call this approach “motion cloning,” it is functionally similar to previous video editing methods [1][3], which also replace appearance while preserving motion. The authors should compare their method with this class of approaches. It is noteworthy that UniEdit [1] and MotionMaster [2] appear capable of achieving the same objectives as this work, namely cloning object and camera motion.

As mentioned in (1), the proposed method closely resembles [1][2]. Therefore, these methods should also be included in the comparisons. Moreover, It is recommended that the authors compare FID scores of generated videos to evaluate video quality.

[1] Bai J, He T, Wang Y, et al. Uniedit: A unified tuning-free framework for video motion and appearance editing[J]. arXiv preprint arXiv:2402.13185, 2024.
[2] Hu T, Zhang J, Yi R, et al. MotionMaster: Training-free Camera Motion Transfer For Video Generation[J]. arXiv preprint arXiv:2404.15789, 2024.
[3] Qi C, Cun X, Zhang Y, et al. Fatezero: Fusing attentions for zero-shot text-based video editing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 15932-15942.

questions:
Please refer to the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730432204508

thread_type:
official_review

<thread_object>
creator: reviewer_ehkx

<content>
summary:
The authors propose a training-free framework that transfers motion from reference videos to video generation. They find temporal-attention maps drives motion synthesis. Based on this observation, they utilize this temporal-attention maps as motion guidance to control the motion of generated videos. In addition, they conduct extensive experiments to validate the effectiveness of the proposed method.

strengths:
+ This paper find temporal-attention maps contain important motion information. The temporal-attention map alignment between reference videos and generated videos can transfer motion from reference videos to video generation. This observation is interesting.
+ Based on this observation, the authors propose a simple but effective method, which can transfer global motion into video generation. For efficiency, they also introduce fixed-time motion representation as the motion guidance. It is a practical solution for deployment.
+ The authors conduct extensive experiments to validate the proposed method. Compared with existing methods, the proposed method achieves superior performance on motion-based controllable video generation.

weaknesses:
- Ablation results. The experiments are exhaustive, but they are only the qualitative results. It is necessary to show the quantitative results for each component. With quantitative results, the proposed method is more convinced.
- Temporal attention block. To my knowledge, down blocks also have temporal attention blocks. Thus, it is an option to apply the proposed motion guidance on down blocks. However, we only show the results with motion guidance applied in up blocks. It is necessary to provide the ablation results of temporal attention in down blocks.
- Guidance weight $\lambda$. The value of $\lambda$ is 2000. To my knowledge, it is too large. What the explanation of this high value is?
- The illustration of Figure 4. The Figure 4 needs to be improved. It is difficult for readers to learn the proposed method clearly only with the contents of Figure 4. Besides, there are some errors in Figure 4. For example, motion representation is not $t_{\alpha}$. According to main text, $t_{\alpha}$ is a time step.

questions:
The authors need to conduct more experiments and improve Figure 4, see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520806148

thread_type:
official_review

<thread_object>
creator: reviewer_gxzm

<content>
summary:
This paper introduces MotionClone, a training-free framework for motion cloning from reference videos, facilitating motion-controlled video generation for tasks like text-to-video and image-to-video. By utilizing sparse temporal attention weights as motion representations, MotionClone enhances motion transfer across different scenarios. It simplifies the process by allowing direct extraction of motion representations in a single denoising step, improving efficiency. Experimental results demonstrate that MotionClone excels in global camera motion and local object motion, achieving high motion fidelity, textual alignment, and temporal consistency.

strengths:
1. The paper is well-written and presents a novel method to perform temporally-consistent video editing in a zero-shot manner.
2. The proposed motion control strategy is simple yet effective through motion guidance from sparse temporal attention map.
3. The paper conducts comphresive experiments to verify the superiority in global camera motion and local object action.

weaknesses:
1. This paper only employs CLIP scores to evaluate the temporal consistentcy of generated videos, which are not convincing. Please provide appropriate metrics like warping error to improve the soundness of MotionClone.
2. MOFT [1] and Space-Time Diffusion Features [2] also performs zero-shot video editing using pre-trained video diffusion models. The proposed method shares similar ideas with them, could the authors give elaborate comparisons with them?

[1] Xiao, Zeqi, Yifan Zhou, Shuai Yang, and Xingang Pan. "Video Diffusion Models are Training-free Motion Interpreter and Controller." arXiv preprint arXiv:2405.14864 (2024).
[2] Yatim, Danah, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. "Space-time diffusion features for zero-shot text-driven motion transfer." In CVPR 2024.

questions:
1. The supplementary materials only show the generated videos of the proposed method. Could the authors provide the comparisons with other methods?
2. Both VMC and the proposed method employs pre-trained video diffusion models to perform zero-shot video editing. The authors are encouraged to fair compare them under the same setting. 
3. Section 4.7 shows the limitation and failure cases of the proposed method. Could the authors discuss their reasons and potential solutions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556607599

thread_type:
official_review

<thread_object>
creator: reviewer_3psb

<content>
summary:
This paper proposes a training-free method that enables motion cloning from reference videos. It explores the use of sparse temporal attention as guidance for motion cloning, which can be extracted through a single denoising step.  Experiments have been conducted to evaluate the effectiveness both qualitatively and quantitatively.

strengths:
1. The exploration of temporal attention for motion representation is interesting and novel to me.
2. The presented results exhibit good motion cloning.
3. The paper is well-written and easy to follow.

weaknesses:
1. The generalization of the proposed method is a concern. The proposed method heavily relies on temporal attention maps, where spatial and temporal attention are separate. Can the proposed method be applied to the latest video methods, such as CogVideoX, which employ the full attention layers?
2. Some existing methods have explored motion guidance in T2V models, such as [1]. The authors should cite and discuss these works.
3. The generated videos exhibit inconsistent details across different frames, such as the robot's eyes and legs in Fig. 1. A better T2V model might help to resolve these inconsistencies.

[1] Xiao, Zeqi, et al. "Video Diffusion Models are Training-free Motion Interpreter and Controller."

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732200417189

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely appreciate all reviewers for their time and efforts in the review.
We are highly encouraged that:
Reviewer 3PSB found our work **interesting, novel, and "exhibit good motion cloning"**;
Reviewer gxZm described our work as **novel** and the proposed motion control strategy is **simple yet effective**; 
Reviewer eHkX found our observation interesting and the method is **"simple but effective"** and "offers a practical solution for deployment"; Reviewer WTAP described our work **"more accessible and efficient"** and ``demonstrating the broad applicability and enhanced Motion Accuracy". 
These positive comments have greatly motivated us, which will certainly guide our future work.

We have carefully revised the manuscript based on the valuable feedback and suggestions. Specifically, we have improved the manuscript from the following aspects:

i) We have validated the potential of the proposed **MotionClone in the latest DiT-based T2V model**, i.e., CogVideoX-2B [1], in which MotionClone demonstrates effectiveness in training-free motion customization by only using motion representation from a single denoising step, offering both generalizability and flexibility.

ii) We have added the **comparison with more methods**, including MOFT [2], Space-Time Diffusion Features[3], UniEdit[4], and MotionMaster[5].

iii) We have introduced **optical-flow based consistency** between generated videos and real reference videos for evaluating motion alignment and added the evaluation regarding the FID metric.

iv) We have further analyzed the **reasons regarding the limitations** and accordingly provided corresponding **potential solutions**. 

v) We have supplemented the **quantitative results of the ablation study**.

vi) We have provided the ablation study on **applying motion guidance in different down blocks**.

vii) We have provided **detailed proof about the choice of motion guidance weight $\lambda$**.

viii) We have made a **clearer description of the motivation** and accordingly presented the difference between the proposed method with previous video editing work.

The detailed point-by-point responses are listed in the columns below.

**Reference**

[1] Yang, Zhuoyi, et al. "Cogvideox: Text-to-video diffusion models with an expert transformer." arXiv preprint arXiv:2408.06072 (2024).

[2] Xiao, Zeqi, et al. "Video Diffusion Models are Training-free Motion Interpreter and Controller." arXiv preprint arXiv:2405.14864 (2024).

[3] Yatim, Danah, et al. "Space-time diffusion features for zero-shot text-driven motion transfer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Bai, Jianhong, et al. "Uniedit: A unified tuning-free framework for video motion and appearance editing." arXiv preprint arXiv:2402.13185 (2024).

[5] Hu, Teng, et al. "MotionMaster: Training-free Camera Motion Transfer For Video Generation." arXiv preprint arXiv:2404.15789 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
All reviewers have a positive view on this submission, acknowledging the novelty and interest in exploring temporal attention for motion representation. They find the generated motion cloning results promising and the presentation well-written and clear. 
As for the concerns raised by the reviewers, the authors have addressed them during the discussion phase. Considering the novelty, experimental results, and the authors' responsiveness to the reviewers' feedback, the paper is recommended to accepted for publication.