MotionClone: Training-Free Motion Cloning for Controllable Video Generation
Authors: Pengyang Ling
Jiazi Bu
Pan Zhang
Xiaoyi Dong
Yuhang Zang
Tong Wu
Huaian Chen
Jiaqi Wang
Yi Jin
Venue: ICLR 2025 Poster (2025)
Keywords: controllable video image
viusal representation
text-to-video generation
Primary area: generative models
TL;DR: Extracting compact motion representation from given reference videos facilitates controlllable video generation

Abstract: Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, we propose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. 
Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.

Published as a conference paper at ICLR 2025
MOTIONCLONE: TRAINING-FREE MOTION CLONING FOR CONTROLLABLE VIDEO GENERATION
Pengyang Ling! Jiazi 2 Pan Zhang"i Xiaoyi Dong
Yuhang Zang Tong Wu Huaian Chen
Jiaqi Wang' Yi Jin't
'University of Science and Technology of China
SShnghaii Jiao Tong University
'The Chinese University of Hong Kong
*Shanghai Al Laboratory
ABSTRACT
Motion-based controllable video generation offers the potential for creating cap- tivating visual content. Existing methods typically necessitate model training encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, pro- pose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text- to-video and image-to-video. Based on the observation that the dominant com- ponents in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal atten- tion weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypass- ing the cumbersome inversion processes and thus promoting both efficiency and flexibility. Extensive experiments demonstrate that MotionClone exhibits profi- ciency in both global camera motion and local object motion, with notable supe- riority in terms of motion fidelity, textual alignment, and temporal consistency.
INTRODUCTION
Video generations that align with human intentions and produce high-quality outputs has recently attracted significant attention, particularly with the rise of mainstream text-to-video (Guo et al. 2023b Blattmann et al. [2023b CCen et al. (2024) and image-to-video (Guo et al., 2023a Blattmann
al. 2023a Dai et al. 2023) diffusion models. Despite the substantial progress witnessed in condi- tional image generation, the domain of video generation presents unique challenges, primarily due to the complexities introduced by motion synthesis. Incorporating additional motion control not only mitigates the ambiguity inherent in video synthesis for superior motion modeling but also enhances the manipulability of the synthesized content for customized creations.
In the realm of video generation that is steered by motion cues, pioneering methodologies can be generally classified into two principal strategies: one that leverages the dense depth or sketch of reference videos (Wang et al. [2024 Jeong & Ye, [023] Guo et al. 2023a), and another that relies on motion trajectories (Wang et al.] 2023b Yin et al. 2023 Niu et al. 2024). The former methodology typically involves the integration of a tre-trained model to extract motion cues at the pixel level. Despite achieving highly aligned motion, these dense motion cues can be intricately entangled with the structural elements of the reference videos, impeding their transferability in novel scenarios. The latter trajectory-bart based methodology, by contrast, provides a more ssir-friendly approach for capturing broader object movements but struggles to delineate finer, localized motions such as head turns or hand raises. Additionally, both methodologies typically entail model training to encode particular motion cues, implying suboptimal generation when applied outside the trained domain. Such limitation is also observed in approaches relying on fine-tuning (Jeong et al. 2023 Zhao et al. 2023). which aim to fit the motion patterns of certain videos.
*Equal contribution. & Corresponding author.

---

Dog, sitting the grass Published as a conference paper at ICLR 2025
Object motion cloning
Camera motion cloning
Duck, swims the river
Robot, the street
Pig, drinks water on beach Building, around with flowers Figure 1: Motion cloning in varying scenarios. Given a reference video, MotionClone can clone the contained motion into novel scenarios with excellent prompt-following ability, without motion- specific fine-tuning. The red arrows indicate the motion direction.
Lion, tuns its head on grass
In this work, introduce MotionClone, a novel training-free framework designed clone motions from reference videos for controllable video generation. Diverging from traditional approaches involving tailored training or fine-tuning, MotionClone employs the commonly used temporal- attention mechanism within video generation models to capture the motion in reference videos. This strategy effectively renders detailed motion while concurrently preserving minimal interdepen- dencies with the structural components of the reference video, offering flexible motion cloning in varying scenarios, as shown in Fig. [ To be specific, is observed that the dominant components in temporal-attentinn weights significantly drive motion synthesis, while the rest mainly refer to noisy applied uniformly across the model,
or very subtle motions. When the whole temporal-attentie the majority of temporal-attention weights overshadow the motion guidance, consequently sulting in the suppression of the primary motion. Therefore, we propose to leverage the principal components of the lemporal-attention weights motion representation, which serves motion guidance that overlooks noisy or less significant motions and concentrates on the primary motion, thus substantially enhancing the fidelity of motion cloning. Moreover, it has been demonstrated that the motion representation extracted from a certain denoising step holds effective guidance across all time steps, offering high efficiency without the burden of cumbersome video inversion. Further- more, MotionClone is compatible with range of video generation tasks, including text-to-video (T2V) and image-to-vides (12V), highlighting its versatility and broad applicability.
In summary, (1) we propose MotionClone. novel motion motion-guided video generation framework that enables training-free motion cloning from given reference videos; (2) we design primary motion control strategy to perform substantial motion guidance over sparse temporal attention map, allowing for efficient motion transfer across scenarios; (3) we validate the effectiveness and versatility of MotionClone in various video generation tasks, in which extensive experiments demonstrate its proficiency in both global camera motion and local object action, with notable superiority in terms of motion fidelity, text alignment, and temporal consistency.
2 RELATED WORK
2.1 TEXT-TO-VIDED DIFFUSION MODELS
Equipped with sophisticated text encoders (Radford et al. 2021 Zhang et al. 2024). great break-
through has been achieved in the realm of text-to- text-to-image image (T21) generation (Gu et al.
2022 Nichol
al. 2021 Rombach al.) 2022 Podell et al. 2023), which sparks the enthusiasm for ad advanced
text-to- texteoo-video (T2V) models (Blattmann et al., 2023b]] Wang et al. 2023a Chen et al.
2023a 2024
Guo et al. 2023b). Notably, VMdeoL.DM (Blattmann et al. 2023b introduces motion module that
2

---

Published as a conference paper at ICLR 2025
utilizes 3D convolutions and temporal attention capture approach, AnimateDiff (Guo et al.) 2023b) enhances a pre-trained T2I diffusson model with motion modeling capabilities. This is achieved by fine-tuning a series of specialized temporal attention lay- ers on extensive video datasets, allowing for a harmonious fusion with the original '21 generation process. To address the challenge of data seareity, VideoCraft2 (Chen al. (024) suggests inno- vative strategy of learning motion from low-quality videos (Bain et al. (202]) while simultancously learning appearance from high-quality images (Sun al. 2024). Despite these advancements, there remains significant disparity in the quality of generated content between the available T2V mod- els and their sophisticated T2I counterparts, primarily due the intricate nature of diverse motions and the limited availability of high-quality video data. In this work, a motion guidance strategy is developed, which ingeniously incorporates motion cues from given videos to ease the challenges of motion modeling, yielding more realistic and coherent video sequences, without model fine-tuning.
correlations. In novel
2.2 CONTROLLABLE VIDEO GENERATION
Building on the success of controllable image generation through the integration of additional con-
ditions (Zhang et al. 2023 Kim al. 2023 Li et al. 2023 Qin et al. [0023] Huang et al.) (023),
multitude of studies (Chen et al. [203aa] Yin et al. 2023 Dai et al. [2023] Ma 2024 B Blattmann
al. 2023a) have endeavored to introduce diverse control signals for versatile video generation.
These include control over the first video frame (Chen et al. (023a). motion trajectory (Yin et al.
2023). motion region (Dai et al. 2023), , and motion object (Ma et al., 2024). Furthermore, in
pursuit of high-quality video customization, several studies delve into reference-based video gener- ation, leveraging the motion from an existing real video to direct the creation of new video content.
A straightforward solution developed in Wang et al. (2024); [Esser et al. (2023); Xing et al. (2024), involves the direct integration of frame-wiss wise depth maps or canny maps to regularize motion. How- ever, this approach inadvertently introduces motion- inriion-iddeendentt features, such as structures in static areas, which can disrupt the alignment of the resulting video appearance with new text. To address this issue, motion-specific fine-tuning frameworks, explored in (Zhao al. 2023 Jeong al., )023), have been developed to extract a distinet motion pattern from a single video or a collection of videos with identical motion. While holding promise, these methods are subject to complex train- ing processes and potential model degradation. To address this, we present a novel motion cloning scheme, which extracts temporal correlations from existing videos as explicit motion clues to guide the generation of new video content, providing a plug-and and-play motion customization solution.
2.3 ATTENTION FEATURE CONTROL
Attention mechanisms have been confirmed as vital for high-quality content generation. Prompt2Prompt (Hertz et al., 2022) illustrates that cross-attention maps are instrumental in dic- tating the spatial layout of synthesized images. This observation subsequently motivates serious work in semantic preservation (Chefer al. (003). multi-object generation (Ma al. 2023 Xiao jet al. 2023), and video editing (Liu et al. 2023). V2V (Ku et al.] (024) reveals dense injec- tion of both CNN and attention features facilitates improved alignment with source videos in video editing. FreeControl (Mo et al./ (2023) highlights that the feature space within self-attention layers encodes structural image information, facilitating reference-based image generation. While previous methods mainly concentrate on spatial attention layers, our work uncovers the untapped potential of temporal attention layers for effective motion guidance, enabling flexible motion cloning.
MOTIONCLONE
In this section, we first introduce video diffusion models and temporal attention mechanisms. Then we explore the potential of primary control over sparse temporal attention maps for substamtial motion guidance. Subsequently, we elaborate on the proposed MotionClone framework, which performs motion cloning by deliberately manipulating temporal attention weights.
3.1 PRELIMINARIES
Diffusion sampling. Following pioneering work (Rombach et al. 2022), video diffusion models encode input video x into latent representation 20 {(x) by using pre-trained encoder 8(-). To
3

---

Published as a conference paper at ICLR 2025
Reference video
Prompt: A plays in the forest
Reference video
Promp:: A lank runs in the desert Generated videos from same initial noise
G Generated videos from same initial noise
w/o control Plain control Primary control Comparision of plain control and primary control over temporal attention map. Leveraging temporal attention maps derived from reference videos to guide video generation. Plain control refers a rudimentary approach whereby all weights are uniformly applied. Primary control only applies constraint to the sparse temporal attention map.
w/o control Plain control Primary control
Figure 2:
enable video distribution learning, diffusion model eø is encouraged estimate noise component from noised latent 2 that follows time-dependent scheduler (Ho et al. 2020), i.c.,
[())
[[] co co c, ())1
where L is the time step, and is the condition signal such as text or image. In the inference phase, the sampling process commences with a standard Gaussian noise. The sampling trajectory, however, can be adjusted by incorporating guidance for extra controllability. This is typically achieved by customized energy function g( y, t) with label indicating guidance direction, i.c.
60 = co(2,, c, + s(co(2t, c, - <0(2,, 0,)) + Avi C 22, t)
(2)
where cg(2,, t) the classifier-Fi free guidance (Ho Salimans 2022), & denotes the unconditional class identifier (such as null text for textual condition), 8 and À are guidance weights, and the term vI- 0e used to convert the gradient of energy function g(-) into noise prediction, in which vā, is the hyperparameter of noise schedule, i.e., 2t 20 + vI Y₁e. During sampling, the gradient generated by energy function g(.) indicates the direction toward generation target.
Temporal attention. In video motion synthesis, temporal attention mechanism is broadly applied to establish correlation across frames. Given a f-frame video feature fin E xxx xxxfxxxxxx where denotes batch size, e denotes channel number, h and w are spatial resolution, temporal attention first reshapes it into 3D tensor fin ((xxxxw)x fxc by merging the spatial dimensions into the batch size. Subsequently, it executes self-attention along the frame axis, which can be expressed as:
fout (ttenttionQ(((
v(f..))
where Q(-), K(-), and V--) are projection layers. Correspondingly, the attention map is labeled as A E R( which captures the temporal relation for each pixel feature.
3.2 OBSERVATION
Since temporal attention mechanism governs the motion in the generated video, videos with similar temporal attention maps are expected to share similar motion characteristics. To investigate this hypothesis, we manipulate the sampling trajectory by aligning the temporal attention maps of the generated video with those from a reference video. As depicted in Fig. [2] simply enforcing alignment on the entire temporal attention map (plain control) can only partly restore coarse motion patterns of reference videos, such as the gait of a cat and the directional movement of a tank, demonstrating limited motion alignment. We postulate that this is because not all temporal attention weights are essential for motion synthesis, with some reflecting scene-specific noise or extremely small motions. Indiscriminate alignment with the entire temporal attention maps dilutes critical motion guidance, resulting in suboptimal motion cloning in novel scenarios. As evidence, primary control over the sparse temporal attention map significantly boosts motion alignment, which can be attributed to the emphasis on motion-relate related cues and the disregard of motion-irrelevan factors.
4

---

Published as a conference paper at ICLR 2025
Irame-1 frame-6 frame-11 frame-16
tu 800
Eu = 600 t = 400 = 200
Figure 3: Visualization of motion representation. The mean intensity of 4 in frame axis from "up_ble blocks. 1st (resized to the represented resolution) indicates the area and magnitude of motion. This performance encounters decline in complex "head turning" scenario when L, 800.
3.3 MOTION REPRESENTATION
Given reference video, the corresponding temporal attention map in t denoising step is denoted as Aree/ E [R(Ixxx))ffx which satisfies [j = The value of [Areeln.s, reflects the relation between frame and j frame in position p, and a larger value [A,e implies stronger correlation. The motion guidance over temporal attention maps, depicted by energy function g(-), is modeled as:
|M [ A Agen))1
(4)
where M is the temporal mask for sparse constraint, and Alen is the temporal attention weights of generated videos in time step I. Essentially, Eq. promotes motion cloning by forcing Aleen close to Aleef while M determines the sparsity of constraint, time-dependence ( constitute the motion guidance. Particularly, .M' = rrffers to the "plain control" that exhibits limited motion
transfer capability as illustrated in Fig. 2 Since the value of Afe, is indicative of the strength of
inter-frame correlation, we propose obtain the sparse temporal mask according to the rank of A value in the temporal axis, i.e.,
Ma..o = 1, if [Ateflass E Dt p.i
(5)
O, otherwise,
where 22 = [TI, T2, The is the subset of index that comprising the top k values in attention map Are, along the temporal axis j, and i is hyper-paramett Particularly, in the case where k = 1, motion guidance focuses solely on the highest activation for each spatial location. Supervised by Eq. 5 motion guidance Eq. encourages the sparse alignment with the primary component in A ref while ensures spatially even constraint, facilitating stable and reliable motion transfer.
Despite enabling effective motion cloning, the above scheme has obvious flaws: i) for real reference videos, laborious and time-consuming inversion operation is required for preparing Afe,s and ii)
the considerable size of the time-dependent { ( poses significant challenges for large- scale preparation and efficient deployment. Fortunately, is noted that the representation from certain denoising step can provide substantial and consistent motion guidance in generation process. Mathematically, motion guidance in Eq. can be converted into
- MM (A A Aben>>1? ||
-=
M
At gen
where t, denotes certain time step, and L = M A'e For given reference videos, the corre- sponding motion representation is denoted as H: (L',,''>) comprising two elements that are both highly temporally sparse. For real reference videos, their H can be easily derived by directly adding noise to shift them into the noised latent of f., time step, followed by single denoising step. This straightforward strategy, impressively, proves to be remarkably effective, As shown in Fig. 3] larger range of time steps (t., from 200 600), the mean intensity of H effectively high- lights the region and magnitude of motion. However, it is also observed that H in early denoising stage (t,, 800) shows some discrepancies with the "head-turning" motion. This can be attributed
5

---

Published as a conference paper at ICLR 2025
Reference video
Prompt: mull-text
Motion representation
Noise adding
H
ta
Sparse Control
Time step decrease
Motion guidance stage
Motion guidance
Prompt: Tiger, raises its head
M
Random noise
Zt-1
Iterative denoising
A gen
Z
Figure 4: The pipeline of MotionClone, in which the motion representation 7 extracted from reference videos serves as motion guidance in novel video synthesis.
the fact that motion synthesis has not yet been fully determined this early stage. Therefore, we suggest employing the motion-aligned H from the latter denoising stage to guide motion synthesis the entire sampling process, facilitating substantial and consistent motion guidance for superior motion alignment.
3.4
MOTION GUIDANCE
The pipeline of MotionClone is depicted in Fig. Given a real reference video, the correspond- ing motion representation 7t is obtained by performing a single noise-adding and denoising step. During the video generation process, an initial latent is initialized from a standard Gaussian distribu- tion and subsequently undergoes an iterative denoising procedure via a pre-trained video diffusion model, advised by both classifier-free guidance and the proposed motion guidance. Given that im- age structure is determined the early steps of the denoising process (Hertz et al., (022), whereas motion fidelity primarily depends the structure of each frame, motion guidance only involves the early denoising steps, allowing for sufficient flexibility for semantic adjustment and thus empower- ing premium video generation with compelling motion fidelity and precise textual alignment.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
In this work, we employ AnimateDiff al. 2023b) as the base text-to-video generation model and leverage SparseCtrl (Guo et al.) 2023a) for image-to-vided and sketch-to-video generator. For given real videos, we apply single denoising f. 400 for motion representation extraction. k = 1 is adopted for mask in Eq. 5 to facilitate sparse constraint. "null-text" is uniformly used as textual prompt for preparing motion representations, promoting a more convenient video customiza- tion. The motion guidance is conducted ablations of above setting are represented in 4.6 Guidance weight s and À in Eq. 2 are empirically set 7.5, and 2000, respectively. For camera motion cloning, the denoising step is configured to 100, in which the motion guidance steps set as 50. For object motion cloning, the denoising step is raised to 300, while applying motion guidance in the early 180 steps.
temporal attention layers in tup.block.11 1"" The detailed

---

Published as a conference paper at ICLR 2025
Prompt: Teddy bear, the grass.
Prompt: Island, on the ocean.
Reference
VMC
VideaCommooer
Gen-1
Tune-A-Video
Control-A-Vide
MotionClone
Figure 5: Visual comparison in camera motion cloning, in which MotionClone achieves superior textual alignment by better suppressing the original structure.
4.2 EXPERIMENTAL SETUP
Dataset. For experimental evaluation, 40 real videos sourced from DAVIS (Pont-Tuset al. 2017) and website are utilized for thorough analysis, comprising 15 videos with camera motion and videos for object motion. These videos encompass rich tapestry of motion types and scenarios, ranging from the dynamic motions of animals and humans to the global camera motion.
Evaluation metrics For objective evaluation, two commonly used metrics adopted: i) Textual alignment, which quantifies the congruence with the provided textual prompt. Following previous work (Wang al. 2024), it measured by the average CLIP (Radford et al. [202]) cosine simi- larity between all video frames and text (Jeong et al., 2023); ii) Temporal consistency, the indicator of video smoothness, quantified by calculating the average CLIP similarity among consecutive video frames. Beyond the scope of objective metrics, a user study is employed for more nuanced assessment of human preferences in video quality, incorporating two additional criteria: i) motion preservation which evaluates the motion's 's adherence to the reference video, and ii) appearance di- versity which assesses the visual range and diversity in contrast to the reference video. The user study scores are derived from the average ratings provided by 20 volunteers, ranging from I to 5.
Baselines. For a thorough comparative analysis, various alternative methods have been examined in the comparison, including VideoComposer (Wang et al. 2024), Tune-A-video (Wu et al. 2023). Control-A- Video (Chen et al., 2023b), VMC (Jeong et al. (023). and Gen-l (Esser et al. (023) A detailed description of each method is depicted in the Appendix.
4.3 QUALITATIVE COMPARISON
Camera motion cloning, As shown in Fig. 5 the "clockwise rotation" and "view switching" motion present significant challenge. VMC and Tune-A-Viee --iidoo generate scenes with acceptable textual
7

---

Published as a conference paper at ICLR 2025
Cat, moves its head. in the bedroom.
Iron man, walks, in the New York City.
Reference
VMC
VideoComposer
Gen-1
Tune-A- Vidico Control-A-Videe MotionClone
Figure 6: Visual comparison in object motion cloning, in which MotionClone performs prefer- able motion fidelity with improved prompt-following ability.
Table 1: Quantitative comparison by using automotive metrics and user study.
Method
VMC
VideoComposer Gen-l
Tune-A-Video Control-A-Video MotionClone
Textual Alignment 0.3134
0.2854
0.2462
0,3002
0.2859
0.3187
Temporal Consistency
0.9614
0.9577
0.9563
0,9351
0.9513
0,9621
Motion Preservation
2.59
3.28
3.50
2.44
3.33
3.69
Appearance Diversity
3.51
3.23
3.25
3.09
3,27
4.31
Textual Alignment
3.79
2.71
2.80
3.04
2.82
4.15
Temporal Consistency
2.85
2.79
3.34
2.28
2.81
4.28
alignment but exhibit deficiencies in motion transfer. The outputs from VideoComposer, Gen-1, and Control-A- -otrol-A-Video are notably unrealistic, which can be attributed to the dense integration of the structural elements from the original videos. Conversely, MotionClone demonstrates superior text alignment and motion consistency, thereby suggesting its superior video motion transfer capabilities within global camera motion scenarios.
Object motion cloning. Beyond the scope of camera motion, the proficiency in handling local object motions has been rigorously validated. As evidenced by Fig. 6 VMC falls short in match- ing motion with the source videos. Videocomposer appears to generate grayish colors with lim- ited froopt-following ability. Gen-l is inhibited by the original videas' structure. Tume-A-Vine Video struggles with capturing detailed body motions, while Control-A- Video cannot maintain faithful appearance. In contrast, MotionClone stands out in scenarios with localized object motions, enhanc- ing motion accuracy and improved text alignment.
8

---

Published as a conference paper at ICLR 2025
Prompt:
Promp:: Blue in the road.
Girl, similing house.
Prompt: Airplane airport, sumny day.
Prompt: Lion, walks in the forest.
Figure 7: Motion Clone also supports I2V and sketch-to-video, facilitating versatile applications. The red arrows indicate the motion direction.
4.4 QUANTITATIVE COMPARISON
The quantitative comparison on real videos with various motion patterns is outlined in Tab. I It is observed that MotionClone gains competitive scores in both textual alignment and temporal consistency. Moreover, MotionClone has outperformed its rivals in motion preservation, appearance diversity, temporal consistency, and textual alignment in human preference tests, underscoring its ability to produce visually compelling outcomes.
4.5 VERSATILE APPLICATION
Beyond T2V, MotionClone is also compatible with 12V and sketch-to-video. As shown Fig. 7 by incorporating the first frame sketch image as additional condition, MotionClone achieves impressive motion transfer while aligning with the specified condition, underscoring its significant potential for wide range of applications.
4.6 ABLATION AND ANALYSIS
Choice of k. k determines the mask in Eq. 3 and thus impacts the sparsity of motion constraint. As illustrated Fig. g a lower value helps better primary motion alignment, attributed by the enhanced elimination of scene-specific noise and subtle motion.
Choice of la. The value of ta determines diffusion feature distribution used for preparing motion representations. As shown in Fig. 8 an excessively large f, 800 causes substantial loss of motion information due to excessive noise injection, while la E {200, 400, 600} can all achieve a certain degree of motion alignment, implying the robustness of t.. In this work, we chose t. 400 as the default value as it typically yields appealing motion cloning in our experiments.
Choice of temporal attention block. Fig, iillustrates the results with motion guidance applied in different blocks. is observed that "up block. 1" stands out for its superior motion manipulation capabilities while safeguarding visual quality, underscoring its dominant role in motion synthesis.
Does precise prompt help ? During motion representation preparation procedure, few differences arise when using tailored prompts regarding video content, represented in Fig. g We speculate that metion-related information is effectively preserved in diffusion features at la 400, thereby diminishing the significance of the precise prompt.
Does video inversion help ? Video inversion can provide time-dependence { M for Eq.
and certain time step ( for Eq. but entails considerable time costs. As shown in Fig. (Inversion_] I vs. Inversion 2), to MM outperforms { Mc) due to the consistent motion

---

Published as a conference paper at ICLR 2025
Pront: W amamn walks in the mall.
Promt: Shark, swims on the ocean.
Reference
=
k 12 k
Reference La
800
GOO f0 400 La 200
Figure 8: Influence of different ls value and different time step t
Prompe: Dior, street
Prompt:
Refrennc
MedoaClose Up 2ock2
Up. Necc.
Pionpee
Inveroers
Retiracc
MatlanCleee Up 3xct2
Up. Mocks
Phorrgt
Ievenion
Inverase
Influence of different attention blocks, precise prompt, and DDIM inversion. "Prompt" denotes motion representation involves precise prompt ("Leopard, walks the forest" for the left case and "Man, turns his head." for the right case); "Inversion_ represents represeen time-dependence
Figure 9:
[Aief, from DDIM inversion; "Inversion indicates (Ma from DDIM inversion.
guidance from the same representation. Meanwhile, there not obvious quality difference regarding whether DDIM inversion is applied (MotionClone Inversion_2). We leave how to perform better diffusion inversion for enhanced motion cloning to further work.
4.7 LIMITATION
Given that MotioClone is conducted latent space, the spatial resolution of dif- fusion features in temporary attention is significantly lower than that of in- put videos, thus MotionClone struggles in local subtle motion, such wink- ing, as shown in Fig. TO Additionally, when multiple moving objects overlap, MotionClone risks quality dropping, at- tributing that coupled motion raises the difficulty of motion cloning.
Prompt: Pigs, play in the mud. Figure 10: MotionClone struggles to handle local subtle motion and overlapping motion.
Prompt: A dog is winking.
5 CONCLUSION
In this work, we observe that the temporal attention layers embedded within video generation mod- els harbor substantial representational capacities pertinent to video motion transfer. Motivated by these findings, we introduce a training-free ffree method dubbed MotionClone for motion cloning. Lever- aging sparse temporal attention weights as motion representations, MotionClone facilitates motion guidance by promoting primary motion alignment, enabling diverse motion transfer across different scenarios. Employing real reference video, MotionClone demonstrates its capability to preserve motion fidelity robustly while concurrently assimilating novel textual semantics. Furthermore, Mo- tionClone demonstrates efficiency by avoiding cumbersome inversion processes and offers versatil- ity across various video generation tasks, establishing itself as highly adaptable and efficient tool for motion customization.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project funded by Anhui Provincial Natural Science Foundation 2408085J024; National Nat- ural Science Foundation of China 62401532; and Anhui Provincial Key Research and Development Plan 202304a05020072. This project is funded in part by Shanghai Artificial Intelligence Labora- tory, Shanghai Innovation Institute, the National Key RD Program of China (20222D0010021).
REFERENCES
Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, and Jiang Bian. Uniedit: A unified tuning-frre free framework for video motion and appearance editing. arXiv preprint 24X11:2402.13185, 2024.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15177 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023b.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention- based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing. Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXiv:2310.19511 2023a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint 1XXx:2441.009447 2024.
Weifeng Chen, Jie Wu, Pan Xie. Hefeng Wu, Jiashi Li. Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305. 13840, 2023b.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang- Animateanything: Fine-grained open domain image animation with motion guidance. arXiv e- prints, pp. arXiv-2311, 2023.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7346-7356, 2023.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706 2022.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16933. 2023a.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023b.
11

---

Published as a conference paper at ICLR 2025
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen- 2022.
image editing with cross attention control. arXiv preprint arXiv:2208.01626,
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma. Motionmaster: Training-free -ree camera motion transfer for video generation. arXiv preprint arXiv:2404. 15789, 2024.
Lianghua Huang. Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.0977, 2023.
Hyeonho Jeong and Jong Chul Ye. Ground-a-video: Zero-shot grounded video editing using text- to-image diffusion models. arXiv preprint arX1v:2310.01107, 2023.
Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. Vme: Video motion customization using temporal attention adaption for text-to-video diffusion models. arXiv preprint arXiv:23/2000845, 2023.
Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Wor Woo Ha, and Jun- Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7701-7711, 2023.
Max Ku, Cong Wei, Weiming Ren, Huan Yang. and Wenhu Chen. Anyv2v: A plug-and aangg--nd-payy frame- work for any video-to-video editing tasks. arXiv preprint arXiv:2403. 14468, 2024.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2251 1-2521,, 2023.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint (rXii:2303.04761 2023.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. arXiv preprint arXiv:2307. //400, 2023.
Yue Ma, Yingqing He, Hongfa Wang, Andong Wang. Chenyang Qi. Chengfei Cai, Xiu Li, Zhifeng Li, Heung- Yeung Shum, Wei Liu, et al. Follow-your-click Open-domain regional image anima- tion via short prompts. arXiv preprint arXiv:2403..m 08268, 2024.
Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-fre spatial control of any text-to-image diffusion model with any condi- tion. arXiv preprint arXiv:2312.07536, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:2 12.1441, 2021.
Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to- video diffusion model. arXiv preprint arXiv:2405.20222 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01192, 2023.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 1004:/704.0000 2017.
12

---

Published as a conference paper at ICLR 2025
Can Qin, Shu Zhang. Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Car- los Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint 21051:2305.1114,, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Kegiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. Advances in Neural Information Processing Systems, 36, 2024.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint
2023a.
Zhouxia Wang, Ziyang Yuan, Xintao Wang. Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint 0rX1v:2312.03641, 2023b.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7623-7633, 2023.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free mubti-subject image generation with localized attention. arXiv preprint 1rX1v:2305.10431 2023.
Zeqi Xiao, Yifan Zhou, Shuai Yang, and Xingang Pan. Video diffusion models are training-free motion interpreter and controller. arXiv preprint arXiv:2405.1488 2024.
Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Y He, H Liu, H Chen, X Cun, X Wang, Y Shan, et al. Make-your-video: Customized video generation using textual and structural guid- ance. IEEE Transactions on Visualization and Computer Graphics, 2024.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint ar 2iv:2408.00677 2024.
Danah Yatim, Rafail Fridman, Omer Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Shengming Yi,, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv 2023.
preprint ar
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403. 15378, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 3836-3847, 2023.
13

---

Published as a conference paper at ICLR 2025
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint arXhv:2110.0845,, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 BASELINE DESCRIPTION
Among the compared methods, VideoComposer (Wang et al. 2024) creates videos by extracting specific features such frame-wise depth or canny maps from existing videos, achieving a com- positional approach to controllable video generation. Gen-1 (Esser et al. 2023) leverages the orig- inal structure of reference videos to generate new video content, akin to video-to-video translation. Tune-A-Vided expands the spatial self-attention of pre-trained text-to-image models into spatio- temporal attention, and then fine-tuning for motion-specific generation. Control-A-Vides (Chen) al. 2023b) incorporates the first video frame as additional motion cue for customized video generation. VMC (Jeong et al., 2023) aims to distill motion patterns by fine-tuning the temporal attention layers in a pre-trained text-to-video diffusion model.
A.2 ABLATION STUDY OF TEMPORAL ATTENTION IN DOWN BLOCK
We have further evaluated the performance of applying motion guidance in down blocks, the quali- tative results are represented in Fig. [T can be observed that "up_ bup_block.1" showcases superiority in motion customization, which performs better motion alignment while enabling effective seman- tic preservation. We conjecture this is because the video motion is determined by the structure of each frame, which is mainly modeled in the "up.block.1", as validated by previous image structure customization work Mo et al. (2023).
A.3 COMPARISON WITH MORE METHODS
We have also compared the proposed method with more methods, including MOFT Xiao et al. (2024), Space-Time Time Diffusion Features Yatim al. (2024), UniEddit Bai al. (2024). and Mo- tionMaster Hu al. (2024). The generated videos of different methods regarding camera motion cloning and object motion cloning are presented in Fig. 12 and Fig. 13 respectively. It is observed that the proposed MotionClone achieves better motion alignment while performing effective appear- ance preservation, thus offering superior motion customization. Moreover, it is worth mentioning that all of the compared methods require time-consuming DDIM inversion, which implies high com- putation costs in practical application. In comparison, the proposed MotionClone can extract motion representation from single denoising step with a unified null-text prompt, offering both efficiency and flexibility.
A.4 APPLICATION IN DIT-BASED ARCHITECTURE
We have validated the potential of the proposed MotionClone in the latest DiT-based T2V model, i... CogVideoX: 2B Yang et al. (2024), in which MotionClone demonstrates inspiring effectiveness in training-free motion customization by only using motion representation from single denoising step, offering both generalizability and flexibility. The results are represented in Fig. 14 and the detailed implementations are listed below. The video synthesis stage in Diffusion DiT involves the collaboration between textual token Steat E (Ixxn xe) and image token Simg I (xxgxe)) in which n and n/2 are token number, and e channel number. During video synthesis, Cog VideoX leverages full attention module for complete relation modeling, in which the resolution of the vanilla attention map thus reaches 1 x (n1 + n22) x (n1 + n2)). Since we focus on the dependence between image tokens for motion transfer across scenarios. we introduce auxiliary attention map to better describe inter-frame dependencies. Specifically, within the self-attention module, the image token reshaped into Simg E attention, h and are spatial resolution, and f is frame number, the corresponding attention map Aouz E RI Xnx fxwxxxxx can obtained by using self-attention in the last frame axis. Following the operation in Diffusion U-Net. MotionClone applies sparse constraints Aqux according to the rank of their values. For given real reference videos, we use null-text as the default prompt for preparing motion representation to facilitate urer-friendly deployment.
in which n, is the head number of multi-head
15

---

Published as a conference paper at ICLR 2025
A.5 MORE GENERATED RESULTS
A broader array of generated content is displayed to validate the versatile generation capability. As shown in Figs. 15 18 MotionClone is able adeptly extract motion cues from diverse range of existing videos and thus enables the creation of content that is both prompt-aligned and motion-preservven showcasing its robust motion cloning capabilities. For better demonstration of MotionClone, we highly recommend viewing the video file in the supplementary material.
A.6 BROADER IMPACT
The development MotionClone, novel training-fres framework for motion-based controllable video generation, carry distinct societal implications, both beneficial and challenging.
On the positive side, MotionClone's capability to efficiently clone motions from reference videos while ensuring high fidelity and textual alignment opens avenues in numerous fields. In the realm of digital content creation, film and media professionals can utilize this technology to stream- line the production process, enhance narrative expressions, and create more engaging visual experi- ences without extensíve resource commitments. Furthermore, in the educational sector, instructors and content creators can leverage this innovation to produce customized instructional videos that incorporate precise motions aligned with textual descriptions, potentially increasing engagement and comprehension among students. This could be particularly transformative for subjects where demonstration of physical actions processes plays a crucial role, such in sports training or scientific experiments.
On the negative side, the power of MotionClone to generate realistic videos based on text and ex- isting motion cues raises concerns about potential misuse, including the creation of deepfakes or misleading media content. Such applications can undermine trust in media, affect public opin- ion through the dissemination of false information, and infringe on personal rights and privacy. Moreover, the case of generating convincing vidcos might enable the proliferation of propaganda or harmful content that can have widespread negative implications on society.
In conclusion, while MotionClone presents significant advancements in the field of AI-driven video generation, is imperative that these technologies are developed and utilized with a conscious com- mitment to ethical standards and regulatory oversight. Promoting transparency in Al-generated con- tent, establishing clear usage guidelines, and fostering an open dialogue about the capabilities and ethics of such technologies crucial steps in ensuring that the benefits of MotionClone realized while its risks are effectively mitigated. This involves collaborative efforts among technologists, pol- icymakers, industry stakeholders, and the broader public steer the responsible development and application of Al-driven media tools.
16

---

Published as a conference paper at ICLR 2025
Reference
Down block.1
Down block.2
Down block.3
Ours
blue car, running, in the forest
A group of sharks, swimming, in the ocean,
Figure 11: Ablation study on applying motion guidance different "down block".loock..
17

---

Published as a conference paper at ICLR 2025
Reference
Space- -paee-Time
UniEdit
MOFT
Maston
Ours
Teddy bear, on the grass,
Island, the ocean,
Figure 12: Comparison with more methods on camera motion cloning.
18

---

Published as a conference paper at ICLR 2025
Reference
Space- Time
UniEdit
MOFT
Ours
Cat, moves its head, in the bedroom
Iron man, walks, in the New York City,
Figure 13: Comparison with more methods on object motion cloning.
Camera Motion Cloning
Object Motion Cloning
Reference
MotionClone
Reference
MotionClone
Reference
MotionClone
Reference
MotionClooee
Figure 14: Results of MotionClone on Diffusion DiT architecture (Cog VideoX).
19

---

Published as a conference paper at ICLR 2025
Reference video (camera zoom in)
Prompt: A road, in the mountain.
Prompt: Relices, on the seabed.
Prompt: Caves, path for exploration.
Prompt: Rail way for train.
Harper
Reference video (camera pan up)
Prompt: Camel, in the desert.
Prompt: Forest, in winter.
Prompt: Lake surface, in the crater.
Prompt: Ship on the ocean.
Figure 15: More results of MotionClone in camera motion cloning.
20

---

Published as a conference paper at ICLR 2025
Reference video (camera pan down)
Prompt: Car, in the street.
Prompt: Cliffs by the sea.
Prompt: Eugle, standing a tree.
Prompt: Forest, with various flowers.
Reference video (camera zoom out)
Prompt: Man, standing his garden.
Prompt: Penguin, on the beach.
Prompt: Red car, on the track.
Prompt: Tree, in the mountain.
Figure 16: More results of MotionClone in camera motion cloning.
21

---

Published as a conference paper at ICLR 2025
Prompt: Batman, turns his head.
Prompt: Red car, runs the beach.
Prompt: Petals falling in the wind.
Prompt: Cat, runs in house.
Figure 17: More results of MotionClone in object motion cloning.
22

---

Published as a conference paper at ICLR 2025
Prompt: Blue car, runs on the beach.
Prompt: Greek sculpture, walks in the forest.
Prompt: Cat, turns its head in house.
Prompt: Pig. walks in the forest.
Figure 18: More results of NotionClone in object motion cloning.
23


Reviews:
[{"review_id": "Nx85OfLO8T", "reviewer": "Reviewer_3PSB", "summary": "This paper proposes a training-free method that enables motion cloning from reference videos. It explores the use of sparse temporal attention as guidance for motion cloning, which can be extracted through a single denoising step.  Experiments have been conducted to evaluate the effectiveness both qualitatively and quantitatively.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The exploration of temporal attention for motion representation is interesting and novel to me.\n2. The presented results exhibit good motion cloning.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The generalization of the proposed method is a concern. The proposed method heavily relies on temporal attention maps, where spatial and temporal attention are separate. Can the proposed method be applied to the latest video methods, such as CogVideoX, which employ the full attention layers?\n2. Some existing methods have explored motion guidance in T2V models, such as [1]. The authors should cite and discuss these works.\n3. The generated videos exhibit inconsistent details across different frames, such as the robot's eyes and legs in Fig. 1. A better T2V model might help to resolve these inconsistencies.\n\n[1] Xiao, Zeqi, et al. \"Video Diffusion Models are Training-free Motion Interpreter and Controller.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5lPelFV4wx", "reviewer": "Reviewer_gxZm", "summary": "This paper introduces MotionClone, a training-free framework for motion cloning from reference videos, facilitating motion-controlled video generation for tasks like text-to-video and image-to-video. By utilizing sparse temporal attention weights as motion representations, MotionClone enhances motion transfer across different scenarios. It simplifies the process by allowing direct extraction of motion representations in a single denoising step, improving efficiency. Experimental results demonstrate that MotionClone excels in global camera motion and local object motion, achieving high motion fidelity, textual alignment, and temporal consistency.", "questions": "1. The supplementary materials only show the generated videos of the proposed method. Could the authors provide the comparisons with other methods?\n2. Both VMC and the proposed method employs pre-trained video diffusion models to perform zero-shot video editing. The authors are encouraged to fair compare them under the same setting. \n3. Section 4.7 shows the limitation and failure cases of the proposed method. Could the authors discuss their reasons and potential solutions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and presents a novel method to perform temporally-consistent video editing in a zero-shot manner.\n2. The proposed motion control strategy is simple yet effective through motion guidance from sparse temporal attention map.\n3. The paper conducts comphresive experiments to verify the superiority in global camera motion and local object action.", "weaknesses": "1. This paper only employs CLIP scores to evaluate the temporal consistentcy of generated videos, which are not convincing. Please provide appropriate metrics like warping error to improve the soundness of MotionClone.\n2. MOFT [1] and Space-Time Diffusion Features [2] also performs zero-shot video editing using pre-trained video diffusion models. The proposed method shares similar ideas with them, could the authors give elaborate comparisons with them?\n\n[1] Xiao, Zeqi, Yifan Zhou, Shuai Yang, and Xingang Pan. \"Video Diffusion Models are Training-free Motion Interpreter and Controller.\" arXiv preprint arXiv:2405.14864 (2024).\n[2] Yatim, Danah, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. \"Space-time diffusion features for zero-shot text-driven motion transfer.\" In CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "54MhCmsBOc", "reviewer": "Reviewer_eHkX", "summary": "The authors propose a training-free framework that transfers motion from reference videos to video generation. They find temporal-attention maps drives motion synthesis. Based on this observation, they utilize this temporal-attention maps as motion guidance to control the motion of generated videos. In addition, they conduct extensive experiments to validate the effectiveness of the proposed method.", "questions": "The authors need to conduct more experiments and improve Figure 4, see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper find temporal-attention maps contain important motion information. The temporal-attention map alignment between reference videos and generated videos can transfer motion from reference videos to video generation. This observation is interesting.\n+ Based on this observation, the authors propose a simple but effective method, which can transfer global motion into video generation. For efficiency, they also introduce fixed-time motion representation as the motion guidance. It is a practical solution for deployment.\n+ The authors conduct extensive experiments to validate the proposed method. Compared with existing methods, the proposed method achieves superior performance on motion-based controllable video generation.", "weaknesses": "- Ablation results. The experiments are exhaustive, but they are only the qualitative results. It is necessary to show the quantitative results for each component. With quantitative results, the proposed method is more convinced.\n- Temporal attention block. To my knowledge, down blocks also have temporal attention blocks. Thus, it is an option to apply the proposed motion guidance on down blocks. However, we only show the results with motion guidance applied in up blocks. It is necessary to provide the ablation results of temporal attention in down blocks.\n- Guidance weight $\\lambda$. The value of $\\lambda$ is 2000. To my knowledge, it is too large. What the explanation of this high value is?\n- The illustration of Figure 4. The Figure 4 needs to be improved. It is difficult for readers to learn the proposed method clearly only with the contents of Figure 4. Besides, there are some errors in Figure 4. For example, motion representation is not $t_{\\alpha}$. According to main text, $t_{\\alpha}$ is a time step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yzXR9F0wsv", "reviewer": "Reviewer_WTAP", "summary": "The paper introduces MotionClone, a training-free framework designed for motion cloning in controllable video generation. Unlike traditional methods that require tailored training or fine-tuning, MotionClone leverages the temporal attention mechanism within video generation models to capture and clone motion from reference videos. This approach allows for detailed motion synthesis while minimizing dependencies on the structural components of the reference video, enabling flexible motion cloning across various scenarios. The framework is versatile, supporting tasks like text-to-video (T2V) and image-to-video (I2V), and it has been shown to outperform existing methods in terms of motion accuracy and text alignment.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. MotionClone eliminates the need for complex training processes, making it more accessible and efficient.\n2. The framework supports multiple video generation tasks, including T2V and I2V, demonstrating its broad applicability.\n3. Enhanced Motion Accuracy: MotionClone outperforms existing methods in capturing and cloning localized object motions with higher fidelity.", "weaknesses": "The core idea of this paper, i.e., achieving motion cloning by constraining the temporal attention map, has already appeared in previous work, specifically in UniEdit [1] and MotionMaster [2]. The authors should acknowledge this prior work as the source rather than presenting it as an original contribution.\n\nThe primary technical distinction from [1][2] is the proposal to use a sparse temporal mask \nM rather than constraining all temporal attention maps. However, this choice lacks motivation or insight, which makes its effectiveness seem questionable.\n\nAlthough the authors call this approach “motion cloning,” it is functionally similar to previous video editing methods [1][3], which also replace appearance while preserving motion. The authors should compare their method with this class of approaches. It is noteworthy that UniEdit [1] and MotionMaster [2] appear capable of achieving the same objectives as this work, namely cloning object and camera motion.\n\nAs mentioned in (1), the proposed method closely resembles [1][2]. Therefore, these methods should also be included in the comparisons. Moreover, It is recommended that the authors compare FID scores of generated videos to evaluate video quality.\n\n[1] Bai J, He T, Wang Y, et al. Uniedit: A unified tuning-free framework for video motion and appearance editing[J]. arXiv preprint arXiv:2402.13185, 2024.\n[2] Hu T, Zhang J, Yi R, et al. MotionMaster: Training-free Camera Motion Transfer For Video Generation[J]. arXiv preprint arXiv:2404.15789, 2024.\n[3] Qi C, Cun X, Zhang Y, et al. Fatezero: Fusing attentions for zero-shot text-based video editing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 15932-15942.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)