Video Diffusion Models are Training-free Motion Interpreter and Controller
Authors: Zeqi Xiao
Yifan Zhou
Shuai Yang
Xingang Pan
Venue: NeurIPS 2024 poster (2024)
Keywords: video generation
motion control
Primary area: generative_models

Abstract: Video generation primarily aims to model authentic and customized motion across frames, making understanding and controlling the motion a crucial topic. Most diffusion-based studies on video motion focus on motion customization with training-based paradigms, which, however, demands substantial training resources and necessitates retraining for diverse models. Crucially, these approaches do not explore how video diffusion models encode cross-frame motion information in their features, lacking interpretability and transparency in their effectiveness. To answer this question, this paper introduces a novel perspective to understand, localize, and manipulate motion-aware features in video diffusion models. Through analysis using Principal Component Analysis (PCA), our work discloses that robust motion-aware feature already exists in video diffusion models. We present a new MOtion FeaTure (MOFT) by eliminating content correlation information and filtering motion channels. MOFT provides a distinct set of benefits, including the ability to encode comprehensive motion information with clear interpretability, extraction without the need for training, and generalizability across diverse architectures. Leveraging MOFT, we propose a novel training-free video motion control framework. Our method demonstrates competitive performance in generating natural and faithful motion, providing architecture-agnostic insights and applicability in a variety of downstream tasks.

Video Diffusion Models are Training-free Motion Interpreter and Controller
Zeqi Xiao', Yifan Zhou', Shuai Yang", Xingang Pan'
SS-Lab, Nanyang Technological University, 2 Wangxuan Institute of Computer Technology, Peking University
Ozeqi001, yifan006) .tt. edu. williamyang@pku. edu. en, xingang panentu.edu.co
Abstract
Video generation primarily aims to model authentic and customized motion across frames, making understanding and controlling the motion a crucial topic. Most diffusion-basee studies on video motion focus on motion customization with training-based paradigms, which, however, demands substantial training resources and necessitates retraining for diverse models. Crucially, these approaches do not explore how video diffusion models encode cross- fross-frame motion information in their features, lacking interpretability and transparency in their effectiveness. To answer this question, this paper introduces a novel perspective to understand, localize, and manipulate motion-awand features in video diffusion models. Through analysis using Principal Component Analysis (PCA), our work discloses that robust motion-aware feature already exists in video diffusion models. We present new MOtion FeaTure (MOFT) by eliminating content correlation information and filter- ing motion channels. MOFT provides distinct of benefits, including the ability to encode comprehensive motion information with clear interpretability. extraction without the need for training, and generalizability across diverse architectures. Leveraging MOFT, we propose a novel training-free vidco motion control frame- work. Our method demonstrates competitive performance in generating natural and faithful motion, providing prchitecture-agnosee insights and applicability in a variety of downstream tasks.
Introduction
Video generation has experienced notable advancements in recent years, particularly in the realm of video diffusion models, such as text-to-video (T2V) generation [15]
5
45 43 and image-to-
video (I2V) generation 14 Apart from producing high-quality content in individual frames, capturing authentic and customized motion across frames is a crucial feature of video generation. Thus, understanding and controlling the motion play pivotal roles in video generation.
that study video motion focus on motion customization, i.e. allowing users to specify a moving direction [46 or a point-drag command [50]. These methods
Most methods [46
typically adopt maining-based paradigms [46] 50 42 15] that introduce motion conditions and
train additional modules ensure that the output videos adhere these conditions. Despite their progress, these approaches require significant training resources and need retraining for different models, and their effectiveness often remains black-box. More critically, they do not address fundamental question: How do video diffusion models encode cross-frame motion information within their features?
Project page at this URL
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Reference
(b) Similarity heatmap of MOFT on various videos.
(c) MOFT as guidance for motion control Figure 1: Characteristics of MOtion FeaTure (MOFT), (a-b) Rich Motion Information: We extract MOFT at the red point in the reference video in (a) and draw similarity heatmaps in (b) across various videos (yellow indicates higher similarity). The heatmap aligns well with the motion flow in the bottom left. (c) MOFT serves as guidance for controlling motion direction in the light-masked region, with the motion direction signal illustrated by red arrows the first image.
Understanding the encoding of motion information is crucial for two reasons: a) it offers architecture- agnostic insights, meaning that such knowledge can be applied across different models and their checkpoints, an important consideration given the rapid evolution of video diffusion models; and b) it supports various downstream applications. For instance, the DIffusion FeaTure [4]] demon- strates how diffusion features can encapsulate rich semantic information, enabling applications like
correspondence extraction 117 23 and image/video editing ( 30 g)
To this end, this paper introduces a novel perspective to understand, localize, and manipulate motion-aware features in video diffusion models. We first establish that removing content correlation information helps to pronounce motion information in video diffusion features. By applying Principal Component Analysis (PCA) [47 on these diffusion features, we observe strong correlation between the principal components and video motions. Further explorations reveal that certain channels of the features play a more significant role in determining motion direction than others. Based on these observations, we present a straightforward strategy to extract motion information embedded in the features, termed MOtion FeaTure (MOFT). Through content correlation removal and motion channel filter, MOFT establishes impressive correspondence on videos with the same motion direction, as illustrated in Fig. [ (a-b). Importantly, this strategy proves to generalizable across various text-to- video or image- -to -oreeeoooovvoooooooooooooooooooe generation models [15 14 43 2 (Fig. 4h such as AnimatedDiff [15). ModelScope [43] and Stable Vidco Diffusion (2).
Building upon the motion-aware MOFT, we propose pipeline for video motion control in training- free manner, without the modification of model parameters. The approach leverages compositional loss functions for content manipulation (9: [1 (1 Specifically, we design loss functions to optimize noisy latents in the denoising process with reference MOFT, which can be synthesized via direction signal or extracted from reference videos. Furthermore, point-drag manipulation. With MOFT guidance to generate coarse motion in the early denoising stages, fine-grained point-drag manipulation with DIFT [1 guidance becomes feasible for videos. Various experiments showcase the effectiveness of MOFT in controlling the motions of diverse scenarios across different video diffusion models without the need for any training. Remarkably, training-fire free method even outperforms some data-driven methods in achicving natural and faithful motion. Our main contributions are summarized follows:
pipeline can be extended for
We perform a deep analysis of motion information embedded in video generation models. Our work discloses that robust motion-aware feature already exists in video diffusion models.
Through our analysis, we present MOtion FeaTure (MOFT) that effectively captures motion information. MOFT has several advantages: a) it encodes rich motion information with high

---

interpretability; b) it can be extracted in training-free free way; and c) it is generalizable to various architectures.
We propose a novel training-free video motion control framework based on MOFT. Our method demonstrates competitive performance with natural and faithful motion. Unlike pre- vious training- based methods that need independent training for each different architecture and checkpoint, our method is readily applicable different architectures and checkpoints.
2 Related Works
Video Diffusion Models. The field of video generation has witnessed substantial progress in recent years, particularly in the domain of video diffusion models. Noteworthy contributions include which aim generate high-
advancements in text-to-video (T2V) generation 200 5 45 43 32 fidelity videos that align with textual descriptions. Besides, image-to-video (12V) 12 14: 6 takes
image conditions as inputs and generates videos aligned with the image. Beyond the production of high-quality content within individual frames, the capability to capture authentic and customized motion across frames stands out as a significant feature the realm of video generation.
Diffusion Feature Understanding. The analysis and comprehension of diffusion features 41
29 10
27 37
have garnered increasing attention. A comprehensive understanding of diffusion features not only yields architecture-agnoste insights applicable across diverse models and checkpoints but also enhances various downstream applications. For instance, DIffusion FeaTure (DIFT) [4] demonstrates that diffusion features embed impressive semantic correspondence and can be extracted with simple strategy. This strategy proves effective across various architectures, spanning image diffusion models [34] to video diffusion models [5 43 Its versatility facilitates a range of applications, including correspondence extraction [7 23 and image/video editing 30 g. Recently, Freecontrol [29] applied PCA on diffusion features and extracted semantic basics for training- free spatial control. Its method can be generalized to any conditional input and any model. Similarly, video diffusion models encode rich motion information within the features. However, less effort has been made to analyze it.
Video Motion Control. Considerable efforts have been dedicated tailoring video motion according 550 42: 16: 77 13 48). For example, MotionCtrl [46] facilitates precise control over camera poses and object motion, allowing for fine-grained motion manipulation. VideoComposer < introduces motion control through the incorporation of additional motion vectors, while DragNUWA (50) proposes a method for video generation that relies on an initial image, provided point trajectories, and text prompts. These methodologies typically rely on training- based paradigms, incorporating motion conditions during training and requiring additional modules to ensure that the resulting videos adhere to these specified conditions. Despite their advancements, these approaches demand substantial training resources, necessitating retraining for different models, and often exhibit black-hox nature in terms of their effectiveness. In contrast, this paper introduces a novel pipeline for controlling video motion using an interpretable motion-aware feature. Notably, this approach is training- free and can be generalized across various architectural frameworks, offering a more versatile and resource-efficient solution.
to user preferences [466] 44
14
3 MOtion FeaTures (MOFT)
In this section, we first analyze how video diffusion models encode cross-frame motion information, then provide the strategy to extract motion features from pre-trained video diffusion models.
our analysis focuses on diffusion features extracted from the intermediate blocks of diffusion models. We denote them as X' E R. Hxx Fx where H, W, F and D are dimensions of height, width, frames, and channels, respectively. As proved by prior works [5] 46). cross- frame- features play crucial role in video motion control. For example, Anfm [[5] trains temporal self-attention LoRAs (2 that operate on the temporal dimension to control the global motion direction. Consequently, we argue that the temporal dimension encapsulates rich motion information. However, extracting motion information from diffusion features is non-trivial, as they also contain other information such as semantic and structural correlation.
Similar to 41 49 29
3

---

Down
Down 2
Principal Component 2
100-
Principal Component 2
200
Still
100
Up
Right
Right
-00
Left 1
Left
-00
-150
-200
-150
-100
-50
100
150
-500
-200
-100
U00
200
300
Principal Component
Principal Component
(a) PCA vanilla feature
(b) PCA of feature after correlation removal
Figure 2: Visualization of PCA video diffusion features. The left side indicates the frame- wise panning direction, with each color representing a specific direction pattern. We apply PCA to diffusion features extracted from videos with different motion directions and plot their projections the leading two principle components. (a) The result does not exhibit a distinguishable correlation with motion direction. (b) Features are clearly separated by their motion direction.
0-15: right
right, lest Fun right
right,
right, right
8.74
Chand
Clare Ua
nin
12
22
Piadpi Compeacu
Framz
weight
(b) motion
motoon
non-motion channel
Figure 3: Cross-frame Channel Value. (a) We plot the histogram of the weight of P1- It reveals that only few channels significantly contribute to determining the principal components. (b-c) The motion channels exhibit a pronounced correlation with motion direction trends. (d) In contrast, the non-motion channels show little correspondence with motion direction.
3.1 Content Correlation Removal
Inspired by VidcoFusion (28] which uses shared noise to model content correlation across frames and residual noise model dynamic difference, hypothesize that filter out the content correlation by eliminating similar information across frames:
X morrm X
Σ X₁
F
1=1
where X; indicates the 7th frame of feature X. The shared latents, to which we refer as content correlation information, encompass shared aspects such as semantic content and appearance. In contrast, the residual latents primarily capture motion information, which also can be interpreted as deformation in structure.
To validate the hypothesis, we apply Principal Component Analysis (PCA) [47 on Y and noeen Specifically, we create a series of videos with the entire scene moving horizontally resulting in a set of features {X' X22 .??...,,")) extracted from videos in the process of DDIM [39 inversion. this subsection, we omit the choice of video model architecture and feature selection for simplicity. We analyze and project the D- D-dimensional features of the first frame on the leading two principal components (P, and P2). shown in Fig. (a), the result of vanilla feature does not exhibit a distinguishable correlation with motion direction. In contrast, shown in Fig. (b), normalized features are successfully separated by their motion direction. reveals that the normalization operation removes content correlation information and emphasizes motion information.
vertically,

---

(a) Reference video
(b) Vanilla Feature
(c) CR
(d) CR&MCF (MOFT) (e) Down block-1 (2x) (f) Mid block (1x)
(g) Up block-1 (4x) (h) Up block-3 (8x)
(i) A nimaleDDif
(i) ModelSSoppe
(k) ZeroScope
()) SVD
Figure 4: Similarity heatmap between feature of the source point and target features. Given the red source point in (a), we plot the similarity heatmap on target videos. Yellow indicates regions with higher similarity. We normalize all similarity to 0-1 for better illustration. (b-d) Similarity heatmap of features with different designs. "CR" indicates "content removal". "MCF" indicates motion channel filter. (e-h) Similarity heatmap of MOFT in different layers in the U-Net. (2x) means relative spatial resolution scale (i-l) Similarity heatmap of MOFT in different video generation models.
3.2 Motion Channel Filter
Principal components can not only reduce dimension but also reflect the importance of each dimension by the projection weights. We visualize the projection weights of Pi E RDx in Fig (a). It reveals that only few channels significantly contribute determining the principal components, indicating these channels encode richer motion information. We term them Motion Channels.
To further explore the relationship between these channels and the motion in videos, create videos panning in different directions at various frames and visualize the channel with the highest two projection weights in P1. As depicted in Fig. 3 (b-c), the value trend closely associated with the panning direction of the video. Specifically, in Fig. (b), the motion channel value decreases during rightward pan and increases during leftward pan. In contrast, channel with low projection weight does not exhibit much correspondence (Fig. (d)). These observations indicate that we can extract motion-awarr aware features by filtering these motion channels.
3.3
MOFT Extraction
With the above explorations, we introduce straightforward strategy for extracting motion information from video diffusion models, which Motion Feature (MOFT). Our method includes two designs: content correlation removal and motion channel filter. The process can be represented as follows
F M.)) jec, F
M (X,)
where M is the extracted MOFT, operates the frame dimension, and j operates channel dimension. is the channel index set motion channels.
We illustrate how content correlation removal and motion channel filter improve the motion corre- spondence in Fig. 4 (a-d). Vanilla video features demonstrate weak alignment with the reference motion. The proposed content correlation removal significantly improves the alignment. Further application of the motion channel filter enhances focus on the motion area (e.g., the rabbit head), yielding higher correspondence.
We conduct an additional ablation study and visualize impact of selecting different video diffusion features from various blocks within the U-Net of AnimateDiff [5]] Fig. (e-h) intuitively reveals that features with relative medium resolutions achieve better motion correspondence, To this end, we select the features after upper block 1.
5

---

Optimize
I
Content Correlation
Region Mask
Removal
Motion Channel
Latents
U-Net
Features
Filter
MOFT
Musked MOFT
Reference MOFT HxwxFxd
M*N<FxC
HxWxFxD
HxWx
hxwxFxd
Figure 5: Motion Control Pipeline. We use reference MOFT as guidance and optimize latents to alter the sampling process. one denoising step, we get the intermediate features and extract MOFT from it with content correlation removal and motion channel filter. We optimize the latents alter the sampling process with the loss of masked MOFT and reference MOFT.
While the above analysis based on AnimateDiff [15,, the property of MOFT holds in different base video models [ 43 4 (Fig, (i-l)), demonstrating that MOFT is versatile across different video generation frameworks, consistently achieving reliable motion alignment.
While MOFT is reminiscent of optical flow, which also describes motion, a key limitation of optical flow is that cannot be directly extracted from video diffusion models during the denoising process and hence cannot serve as the guidance for motion control. In contrast, MOFT is available even early denoising steps and is naturally suitable for motion control, as will discuss in the next section.
4 MOFT Guidance
With the motion-awan MOFT, propose pipeline for video motion control in training- manner (Sec. 4.1 Furthermore, our pipeline can be extended for point-drag manipulation (Sec. 4.2).
4.1
Motion Control
We design pipeline to control motion in the generated videos in training- free way, as depicted in
Fig- 5
Following
33 49 we optimize latents to alter the sampling process. The loss function L
C
Σ
1|M -M
R
(L.))ER
where M is the MOFT we extract during the denoising phase, M the reference MOFT feature, and R is the position set of the region that want to control motion. We provide two possible ways to construct the reference MOFT MM': 1) Extract MOFT from reference videos. We perform DDIM inversion [39] on reference videos and extract MOFT in the inversion stage. 2) Synthesize MOFT based on the statistic regularity. As shown Fig. (b-c), frame-wise motion channel values exhibit high correspondence with frame-wise motion. We can fit it into piecewise linear function, where each piece function ranges from statistic minimum to statistic maximum. In this way, we can flexibly modulate frame- wise reference motion guidance. The detailed process is shown in Alg. [
Algorithm I: Optimization Process
Input: Noisy latents = timestep I, region mask R, reference MOFT M", the network NN, Motion Channel Mask C, learning rate v)
Output: Optimized latents E
begin
Get intermediate feature X from the network. V;;
Given X, C, extract MOFT M by Eq. 2
Given M", M,, and R, compute the loss by Eq. Optimize é by updating +- VL:
return 2;
end
4.2 Point-Drag Manipulation
Point-drag manipulation is designed to precisely relocate points within image and video frames to reach specific target points. In the image domain, this manipulation method often relies motion

---

DIFT
MOFT
(a) Reference (b) Step 800 (c) Step-400
(c) Step
(e) Reference (1) Step 800 (g) Step 400
(h) Step 0
Figure 6: Effects of DIFT and MOFT on different denoising time steps. Given the source point in (a) (for DIFT) and (e) (for MOFT), we plot the similarity heat map of DIFT (b-d) and MOFT (f-h) of different denoising steps. Yellow indicates higher similarity. The red point in (b-d) indicates the position with highest similarity. It suggests that MOFT can provide more valid information than DIFT at the early denoising stages.
supervision and point-tracking 33 36 ensuring the precise tracking of point trajectories to achieve the desired target points. In the video domain, however, we can directly optimize whole point trajectories by setting targets in each frame. The loss function for optimizing point trajectories T P1. PP is:
- = Σ D(p.) sg. (g((Dr)))l
(4)
where the diffusion feature (DIFT) and sg the "stop gradient" operation.
However, direct application of this method results in poor video motion control because DIFT struggles with semantic correspondence at early denoising steps, as shown in Fig. 6 Row 1. Since spatial and temporal structures are already determined at early steps. DIFT's effectiveness is limited. Conversely, MOFT provides relatively distinguished motion information early denoising stage performance (Fig. Row 2), suggesting a strategy of using MOFT for initial coarse motion control and DIFT for precise point-drag manipulation. Please refer Supplementary Material for details.
5
Experiments
5.1 Implementation details
not specified, the default video generation models of the following experiments are implemented in AnimateDiff [13] (T2V) and SparseCtrl [14] (I2V). For T2V generation, first generate normal video as the editing source, then apply motion direction and region mask to the video for motion control. To preserve consistency with the source video, we apply (1) region gradient clip, and (2) shared key and value. Details of these techniques and video results can be found in the Supplementary Material. Our results are resolution of 512x512 and 16 frames unless otherwise specified. We use DDIM with 25 denoising steps for each sample. It takes approximately 3 minutes to generate one sample on an RTX 3090 GPU.
5.2 Qualitative Results
We showcase qualitative outcomes in Fig. 7 The figure illustrates the successful animation of videos by our method, guided by diverse control signals while preserving a natural and authentic sense of motion. Additionally, we exhibit the results of applying
motion control technique to alternative video generation models, such as ModelScope [43 and ZeroScope 44, employing the same control strategy (see Fig. 8). These results highlight the generalizability of MOFT across various video
generation models. We also showcase the application of our method on point-drag manipulation
(Sec. 4.2 in Fig. where successfully move the starting points to the targets.
5.3
Motion Feature Design
This subsection experiments on the effectiveness of motion feature designs with two metrics: a) Motion Fidelity. Following [49,, we use Motion Fidelity to assess the fidelity of our results in the alignment of synthesis guidance reference guidance. We use off-the-shelf tracking method [24] to estimate the tracklets T (TI,...; TM) in the generated videos. For guidance, we manually construct synthesized tracklets for synthesis guidance and use estimated tracklets for reference guidance,
7

---

(a) Reference Control Cameral Motion
(b) Reference Control Object Motion
(c) Synthesized Control Camera Motion
Synthesized Control Object Motion
Figure 7: Qualitative results. We illustrate several animation clips with different reference or synthesized motion control signals. The red boxes in (a-b) stand for reference videos. We highly recommend readers refer to the supplementary material for better visual experience.
denote them both as 7 (o....,
F for simplicity. The motion fidelity score is defined follows: 1
max corr 7)
Σ max corr(r, 7).
(5)
m
TET
n
TET FET
FET
The correlation between tracklets corr(7, 7) computed
1 1 F Σ (og)+++
corr(T,
(6)
are the 4tth frame displacement of tracklets т, F. respectively. b) Image that uses image quality predictor trained the SPAQ dataset [12] to evaluate frame-wise quality regarding distortion like noise, blur, or over-exposure. We collect a total of 270 prompt-motion direction pairs for experiments.
where (Vg, v
Quality. We follow 12 25
Table summarizes our results. The vanilla feature shows poor motion fidelity and image quality due to extraneous information disrupting motion control. Removing content correlation significantly

---

Sodee Close-up view of a cat's face
ZenSoope: A lion lying on the grasslan
Figure 8: Qualitative results Modelscope [43] and ZeroScope [4].
Figure 9: Qualitative results of point-drag manipulation. Red points indicate starting points. Blue points indicate target points of the corresponding frames. We display three frames per video clip.
Gev-2
Owrs
Figure 10: Motion quality comparisons. Gen-2 35
and Ours accept editing region and motion direction as the control signal. DragNUWA 50 accepts point trajectories the control signal.
improves both metrics, yielding results comparable to the Space-Motion Map (SMM) feature (49), likely because SMM also removes content correlation through frame-wise differences. MOFT guidance achieves the highest motion fidelity, with only minor loss in image quality compared to the original unguided generation.
5.4 Point-drag Manipulation
We conducted additional experiments assess the efficacy of incorporating motion control in point-based manipulation. In this comparison, we introduce DragNUWA [50), potent data-driven method, for reference. We follow B33 36 use the Mean Distance between edited points and target points to evaluate the drag precision. Specifically, we still use (2 to estimate the tracklets
T T .... T.=M} }or of given small region. We average these tracklets into and calculate the mean distance with target tracklet r². We normalize the final distance into [0, with 0 indicating no mean distance error. We collect total of 40 image-motion direction pairs for experiments. As indicated in Table 2 applying only DIFT guidance results in poor drag precision. By comparison, incorporating our MOFT yields substantial improvements, effectively narrowing the performance gap with the training- based NUAA. The finding is coherent with analysis in Sec. 4.2

---

Table 1: Experiments on Motion Feature Design
Guide type
Methods
Motion Fidelity (1)
Imaging Quality (f)
None
Origin
0.697
SMM feature 49
70.2
0.681
Reference Guidance
Vanilla Feature
31.1
0,512
CR CR MCF (Ours)
67.1
0,671
82.5
0.693
Synthesis
Ours
84.0
0.693
Guidance
Table 2: Drag Precision
Table 3: User Preference
Name
Mean Distance (4)
Methods
Faithfulness (1)
Naturalness (1)
DragNUWA [50
0.075
DragNUWA (50
2.50
2.08
DIFT 4
0.437
Gen-2 MB 35
3.37
2.90
+ MOFT (Ours)
0.175
Ours
3.21
3,49
5.5
User study
We conducted survey investigate users' preferences regarding videos generated with motion control. Employing blind rating protocol, participants were randomly exposed to videos generated by Gen-2 Motion Brush [3). DragNUWA [50], and our proposed method. Participants were instructed to rate from to 5 (worst to best) on two metries: 1) Motion Faithfulness to measure how well the motion aligns with the control signal. 2) Motion Naturalness evaluate the naturalness and realism of the motion. We collect human feedback from 26 people on 56 video clips. As depicted in Table 3 and Fig. 10 is evident that Gen-2 MB excels in achieving highly faithful motion control the cost of motion naturalness. Gen-2 MB and DragNUWA tend to generate stiff and unrealistic motions. In contrast, our proposed methods demonstrate competitive motion faithfulness while simultaneously preserving the natural and authentic quality of motion.
Limitations add Future Works
While our approach has yielded appealing results, some limitations require future studies:
1) Presently, our approach lacks support for motion control in real videos. Primarily, this limitation stems from the lack of research on video inversion techniques over video diffusion models. We have observed significant alterations in content when employing initial noise from DDIM inversion [39 on real videos. Future research focused on video inversion holds promise for resolving this issue.
2) Our current approach does not allow for precise motion scale guidance motion control. While there strategies roughly control motion scales, such adding up control weights for larger motion scales or implementing gradient clips for smaller ones, achieving high precision in motion scale manipulation requires further investigation.
7 Conclusion
In summary, our analysis reveals robust motion-aware feature in video diffusion models, leading to the development of training-free MOtion FeaTure (MOFT). MOFT encodes rich, interpretable motion information, is extracted without training, and applicable across diverse architectures. We introduce novel training-fie -free video motion control framework based on MOFT, demonstrating competitive performance with natural motion. Importantly, our approach is versatile, easily adaptable to various architectures and checkpoints without the need for independent training.
Acknowledgements. This research supported by MOE ACRF Tier (RG97/23) and is also supported under the RIE2020 Industry A Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-dind contribution from the industry partner(s).
10

---

References
[1] Bansal, A., Chu, H.M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., Goldstein, T:: Universal guidance for diffusion models. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 843-852 (2023)
[2] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z.., Voleti, V., Letts, A., al.: Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1513 (2023)
[3] Cao, M., Wang, X., Qi Z., Shan, Y., Qie, X., Zheng. Y.: Masactrl: Tuning-free mutual self- attention control for consistent image synthesis and editing. In: Proceedings of the IEER/CVF International Conference on Computer Vision. pp. 22560-22570 (2023)
[4] Cerspense:
Zeroscope.
https://huggingface httggg/iii....oooooorrrass/////a
(2023)
[5] Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q., Wang. X., et al.: Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512 (2023)
[6] Chen, H., Zhang. Y., Cun, X., Xia, M., Wang, X., Weng, C., Shan, Y: Videocrafter2: Overcom- ing data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047 (2024)
[7] Chen, X., Liu, Z., Chen, M., Feng. Y., Liu, Y., Shen, Y., Zhao, H.: Livephoto: Real image animation with text-guided motion control. arXiv preprint arXiv:2312.029288 (2023)
[8] Deng, Y.. Wang, R., Zhang. Y., Tai, Y.W., Tang, C.K.: Dragvideo: Interactive drag-style video editing. arXiv preprint arXiv:2312.02216 (2023)
[9] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, 8780-8794 (2021)
[10] Du, X., Kolkin, N., Shakhnarovich, G., Bhattad, A.: Generative models: What do they know? do they know things? let's find out! arXiv preprint arXiv:2311.17331 (2023)
[11] Epstein, D., Jabri, A., Poole, B., Efros, A., Holynski, A.: Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems 36 (2024)
[12] Fang, Y., Zhu, H.,, Zeng, Y., Ma, K., Wang, Z.: Perceptual quality assessment of smartphone
photography. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3677-3686 (2020)
[13] Geyer, M., Bar-Tal, O., Bagon, S., Dekel, T.: Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307. (2023)
[14] Guo, Y., Yang. C., Rao, A., Agrawala, M., Lin, D., Dai, B.: Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:231 1.. (2023)
[15] Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specifie tuning. arXiv preprint arXiv:2307.04722 (2023)
[16] He, H., Xu, Y., Guo, Y., Wetzstein, G., Dai, B., Li, H., Yang, C.: Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02101 (2024)
[17] Hedlin, E., Sharma, G., Mahajan, S., Isack, H., Kar, A., Tagliasacchi, A., Yi, K.M.: Unsu- pervised semantic correspondence using stable diffusion. Advances in Neural Information Processing Systems 36 (2024)
[18] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840-6851 (2020)
[19] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598 (2022)

---

[20] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M.. Fleet, DJJ.: Video diffusion models. Advances in Neural Information Processing Systems 35, 8633-8646 8646 (2022)
[21] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L.,, Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 (2021)
[22] Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q. Chanpaisit, N., al.: Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982 (2023)
[23] Ju, Y., Hu, K., Zhang, G., Zhang, G., Jiang, M.. Xu, H.: Robo-abe: Affordance generaliza- tion beyond categories via semantic correspondence for robot manipulation. arxiv preprint arXiv:2401.0PF 1.7487 (2024)
[24] Karaev, N., Rocco, I., Graham, B., Neverova, N.. Vedaldi, A., Rupprecht, C.: Cotracker: It is better to track together. arXiv preprint arXiv:2307.0765 (2023)
[25] Ke, J., Wang, Q., Wang, Y., Milanfar, P., Yang. F:: Musiq: Multi-scale image quality transformer.
In: Proceedings of the IEEE/CVF Internatonn Conference on Computer Vision. pp. 5148-514 (2021)
[26] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.611 (2013)
[27] Luo, G., Darelll T., Wang, O., Goldman, D.B., Holynski, A.: Readout guidance: Learning control from diffusion features. arXiv preprint 1r (2023)
[28] Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang. L., Shen, Y., Zhao, D., Zhou, J., Tan, T.: Videofusion: Decomposed diffusion models for high-quality video generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10209-10218 (2023)
[29] Mo, S., Mu, F. Lin, K.H., Liu, Y., Guan, B., Li, Y., Zhou, B.: Freecontrol: Training- free spatial control of any text-to-image diffusion model with any condition. arXiv preprint arXiv:2312.07555 (2023)
[30] Mou, C., Wang. X., Song, J., Shan, Y., Zhang, J.: Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02422 022221 (2023)
[31] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Miskkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint 1tXXi::1122.004: (2021)
[32] OpenAl: Video generation models as world simulators. https://openai. http:///pppaai.coomeeeeeaaaaa www.researc.c///ooooeo.eoooeccae as- www.wwwwwwwwdlaaaeeeeeeeee mumuttorsssuoo (2024)
[33] Pan, X., Tewari, A., Leimkühler, T., Liu, L.., Meka, A., Theobalt, C.: Drag your gan: Interactive
point-based manipulation on the generative image manifold. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1-11 (2023)
[34] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffuusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684-10695 (2022)
[35] runway: Gen-2. https::/ runwayml. com/ (emm/gn22 (2023)
[36] Shi, Y., Xue, C., Pan, J., Zhang, W., Tan, V.Y., Bai, S.: Dragdiffusion: Harnessing diffusion models for interactive based image editing, arXiv preprint arXiv:2306. 14435 (2023)
[37] Si, C., Huang, Z., Jiang, Y., Liu, Z.: Freeu: Free lunch diffusion u-net. arXiv preprint arXiv:2309.1 1497 (2023)
[38] Sohl-Dickstein, J., Weiss, F., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: International conference on machine learning. pp- 2256-2263 PMLR (2015)
12

---

[39] Song, J., Meng. C.. Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
[40] Song, Y. Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Scsre-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1.1345 (2020)
[41] Tang, L.,, Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems 36 (2024)
[42] Wang, J., Zhang, Y., Zou, J., Zeng. Y., Wei, G., Yuan, L., Li, H.: Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566 (2024)
[43] Wang. J., Yuan, H., Chen, D., Zhang, Y., Wang, X., Zhang, S.: Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06577 (2023)
[44] Wang, X., Yuan, H., Zhang, S., Chen, D., Wang, J., Zhang, Y., Shen, Y., Zhao, D., Zhou, J.: Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems 36 (2024)
[45] Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z.,, Wang. Y., Yang, C., He, Y. Yu, J., Yang, P., al.: Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15103 (2023)
[46] Wang, Z., Yuan, Z., Wang, X., Chen, T., Xia, M., Luo, P., Shan, Y.: Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.0364 (2023)
[47] Wold, S., Eshensen, K., Geladi, P.: Principal component analysis. Chemometrics and intelligent laboratory systems 2(1-3), 37-52 (1987)
[48] Yang, S., Zhou, Y., Liu, Z., Loy, C.C.: Rerender video: Zero-shot text-guided video-to-video translation. In: SIGGRAPH Asia 2023 Conference Papers. pp. 1-11 (2023)
[49] Yatim, D., Fridman, R., Tal, O.B., Kasten, Y., Dekel, T.: Space-time diffusion features for zero-shot text-driven motion transfer. arXiv preprint arXiv:2311.17009 (2023)
[50] Yin, S., Wu, C., Liang, J., Shi, Li, H., Ming, G., Duan, N.: Dragnuwa: Fine-grained control
in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089 (2023)
[51] Zhao, R., Gu, Y., Wu, J.Z., Zhang, DJJ., Liu, J., Wu, W., Keppo, J., Shou, M.Z.: Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08465 (2023)
[52] Zheng, Z., Peng. X., Yang. T., Shen, C., Li, S., Liu, H., Zhou, Y., Li. T. You, Y: Open- Democratizing efficient video production for all (March 2024), https: :ottps//gge wiigghhub.... com/ hpccattech/ppen-Soraa
13

---

Supplementory Material
In this section, provide analysis and results. it highly recommended refer the attached webpage for better visual illustrations.
8.1 Preliminary for Latent Optimization
Diffusion models learn to recover clean images from random noise 2T (N(0, I) with a sequential proposed the latent diffusion model (LDM), which maps data into lower-dimensional space via a variational auto-encoder (VAE) [26] and models the distribution of the latent embeddings instead. At the diffusion step random noise 6, is added to æ, giving noisy image 2 at ogEt, with Ote and σ; the time-dependent parameters. The estimation of the denoised image is equivalent to predicting the noise Et.
denoising process 18
Our latent optimization strategy is motivated by [36] 49 that uses intermediate features to supervise the latent optimization process, which can be formulated
ac
24 17 3z
where 1) is the learning rate and L is the loss function.
8.2 More Analysis and Details
Video Consistency Preservation. Since one of our applications is to control the motion in source- generated videos, is important to preserve the consistency between the source video and the target video. To this end, we introduce two techniques: Shared K&V V and Masked Gradient Clip. We
visualize their qualitative comparison Fig. I2
Shared K& V. As proved by many previous works [36 30 3
inserting Key (K) & Value (V) of spatial attention from reference branch target branch can help to preserve content information of reference generation. As shown in Fig. I1 we adopt this method to our motion control pipeline.
N
Latents MrN- F&C Reference Branch
U-Nol
MrNrFxC
MrNxFxC
Shered-VV
1 Lateets
Naise
New Latents
M+N Fa Guidence Branch
U-Net
MxN*FxC
Figure 11: Motion Control Pipeline with Shared K&V. We apply the origin designing process in the reference branch while applying motion guidance in the motion guidance branch. During denoising, we insert the K&V of the reference branch to the motion guidance branch for content preservation.
As shown in Fig. 12 (a-c), shared K&V contributes to the consistency of the whole video. The generated video with vanilla motion guidance (Fig. 12 (b)) adds additional contents (i.e. hat on the
man's head) while adding shared K&V (Fig. T2 (c)) stays consistent with the original generation.
Masked Gradient Clip. Since we do not want to change much content out of the masked region during motion guidance optimization, we simply clip the guidance gradient g out of the masked region, which is
[g. (i.j.) E R&ck E F gllip
else,
14

---

(a) Origin generation
(b) Vanilla Motion Guide
(c) + Shared &&V
(d) + Gradient Clip for the first frames
(e) + Gradient Clip for all frames
Figure 12: Qualitative comparison on video consistency preservation. We compare the generated results w./wo. our introduced techniques. The control signal is shown in the first image of (a), with the red arrow indicating the motion control direction and the light region indicating the control region. We highlight the noticeable region with red boxes. It reveals that Shared K&V contributes to the consistency of the whole video. Gradient Clip adds consistency out of masked regions but meanwhile reduces motion scale.
where i,j, k indices of height, width, and frame, respectively. R is the spatial mask region index set. F is the frame set. As shown in Fig. 12 (d-e), gradient clipping adds consistency to the background content. While applying gradient clipping more frames increases consistency, also results in a smaller motion scale. Thus, applying gradient clipping involves a trade-off. In practice, we apply gradient clipping to the first 8 frames.
Timestep Choice for Motion Extracted from Video. As shown Fig. 13
the trends and ranges of motion channels with the same motion direction are similar among different denoising timesteps, while the curve nearing the end denoising steps smoother and refined. To this end, we use the MOFT at the beginning of the inversion stage the guidance for all control timesteps.
Motion Channel Filter Number. We ablate the effects to filter different numbers of motion channels in Tab. The key finding is that preserving only a few channels most sensitive motion can enhance both motion faithfulness and naturalness, as the effects of irrelevant information in other channels removed. However, when we further reduce the channel number to the top 1%, both motion
15

---

TS-800
20
TS-400
Channel Value
10
TS-0
0-
-00
-20-
30
0
10
12
14
Frame Index
Figure 13: Motion channel value for different denoising timestep. TS-800 indicates denoising step 800.
Table 4: Motion Channel Ablation.
Channel number
Faithfulness (f)
Naturalness (t)
top 100%
67.5
0.671
top 50%
70.5
0.681
top 10%
82.7
0.692
top 5%
83.2
0.694
top 4%
84.0
0.693
top 3%
83.6
0.693
top 1%
76.3
0.677
faithfulness and naturalness significantly decrease due to the loss some motion-sensitive channels. In practice, choose the top 4% of motion channels.
Point-Drag Manipulation Ablation. Following the main paper Sec. 4.2, we use MOFT for initial coarse motion control and DIFT for precise point-drag manipulation. The compositional loss function
u > 0, w = 0, if t> t
wf 0, wf 0, if h 1 >>>= l/2
C. w + wff
wi 0, wp > 0,
if t >= fg
we = O, w = 0, t
where w/ and wp time-dependent weights under the threshold t1, t2 and f3. In practice, the total denoising step is 25, We set & = 19, 12 = 18, f3 = We further ablate the effectivvnesss of the design in Fig. [4] Applying only DIFT results in limited motion. Using only MOFT produces motion but lacks precise point control. By combining DIFT and MOFT, we achieve precise point-drag control.
8.3 More Visualization
PCA video examples. Fig. 15
we provide some of the video examples that we use to conduct PCA. We manually move the whole picture following specified motion directions to synthesis videos.
Qualitative Results. We provide some qualitative results in Fig. 16 More animated results can be found on the attached webpage.
8.4 More Results on Open-Sora
In Fig. 17 a)),, demonstrate that PCA clearly separate videos with different motions based on their diffusion features from Open-Sora [52], an open-source video generation model capable of producing long videos. In Fig. b)),, show that our methods can be applied higher resolutions (768x768) and longer videos (205 frames on Open-Sora).
16

---

MOFT+DIFT
MOFT
DIFT
Figure 14: Qualitative results of point-drag manipulation ablation. Red points indicate starting points. Blue points indicate ending points. We only display three frames per animation clip.
Figure 15: Videos for PCA. We manually move the whole picture following specified motion directions to synthesis videos.
17

---

Figure 16: More qualitative results.
city right
faresi, Jeft
mountain_right
Principal Component
room, left
wall right
185 104 205 frames
book left
table
city glowr
village down
-309
20
300
768*768 frames
Prompt: view of mountain
Princioal Component
(a) PCA on Open-Sora
(b) Results on longer videos and higher resolutions
Figure 17: More results on Open-Sora.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims accurately reflect our contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
This paper does not provide theoretical results. The assumptions in the paper are verified by experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the necessary details for reproducibility in Sec. 5.1 and Supp...
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer:
[No]
Justification:
We will release it later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwiiicgguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the necessary details for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Our experiments are not accompanied by error bars
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide it in Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:r:////uis..eeeee wwwpourissccpppuieee http://///////tteccceeeeeeeeeeee
Answer: [Yes]
Justification: Our research is aligned with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit the code and models used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Our experiments include human feedback. We include the instructions and details Sec. 5.5.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
Our experiments do not have risks.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "1U7ZJWABMq", "reviewer": "Reviewer_zi7R", "summary": "This paper presents a training-free method for motion control in video diffusion models and explores the interpretability of features within these models. The authors demonstrate through experiments that principal components of the features, extracted using PCA, contain motion information. They propose a pipeline that eliminates content correlation information from the features, filters motion channels, and optimizes the initial latent input in the diffusion model's UNet.", "questions": "Q1. After eliminating content correlation information, does the feature retain any information other than motion? If so, could this residual information affect video generation?\n\nQ2. Figure 6 illustrates how the ability to track motion varies with generation steps. Is it possible to manipulate features from earlier generation steps to achieve better results?", "limitations": "Not included. \nWhile the research demonstrates the potential of training-free methods to encode motion information, it remains unclear whether the proposed method completely eliminates motion-correlated information or retains any motion-irrelevant information.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper effectively demonstrates the potential of internal features in video diffusion models to capture motion information. \n\n- The use of PCA to eliminate irrelevant information is well-justified.\n\n- The proposed training-free method can be applied to various manipulation scenarios, including reference-based and drag-based control.", "weaknesses": "- In Section 3, the authors discuss the challenge of extracting motion information from diffusion features due to their encapsulation of other data types, such as semantic and structural correlations. The paper lacks a detailed explanation of \"content correlation information\" (section 3). It is unclear whether this term encompasses semantic, structural, appearance, background, lighting, or other information.\n\n- Figure 6 contrasts the video motion control capabilities of DIFT and MOFT. It is recommended that identical sample images be used and their similarity heatmaps be compared, thereby better visualizing the capability gap between DIFT and MOFT.\n\n- The optimization process of the latent in Section 4.1 could be more clearly explained, particularly the settings from references [31; 41]. \n\n- A more descriptive caption for the Motion Control Pipeline (Figure 5) and a clearer title for the caption of Figure 6 are advised for better comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comment. The detailed responses regarding each concern are listed below.\n\n**W1: Explanation of content correlation information**\n> In Section 3, the authors discuss the challenge of extracting motion information from diffusion features due to their encapsulation of other data types, such as semantic and structural correlations. The paper lacks a detailed explanation of \"content correlation information\" (section 3). It is unclear whether this term encompasses semantic, structural, appearance, background, lighting, or other information.\n\nThis component is inspired by VideoFusion [8], which demonstrates that decomposing video latents into shared latents among all frames and per-frame residual latents enhances video generation. The shared latents, to which we refer as content correlation information, encompass shared aspects such as semantic content and appearance. In contrast, the residual latents primarily capture motion information, which also can be interpreted as deformation in structure.\nBuilding on this insight, we designed a content correlation removal process to remove shared information while preserving motion information. The effectiveness of this approach is validated in Figure 2. We will clarify this term in the revision.\n\n**W2: Identical sample images in Figure 6**\n> Figure 6 contrasts the video motion control capabilities of DIFT and MOFT. It is recommended that identical sample images be used and their similarity heatmaps be compared, thereby better visualizing the capability gap between DIFT and MOFT.\n\nThank you for your valuable input. We have added the corresponding images, and you can find the results from identical images in Figure 2 in the attached PDF, which supports our conclusions.\nIt is important to note that the heatmaps in DIFT and MOFT represent different contents: semantic similarity in DIFT and motion similarity in MOFT. Therefore, even though the same images are used, the heatmaps cannot be directly compared side by side. Instead, we visualize their effects at different time steps to highlight that MOFT is more reliable at the earlier denoising stage.\n\n**W3: Explanation of the optimization process of the latent**\n\n> The optimization process of the latent in Section 4.1 could be more clearly explained, particularly the settings from references [31; 41].\n\nThank you for the suggestion. We will explain this more clearly in the revision.\nPlease refer to the global response for the pseudo-code of the optimization process.\n\n\n**W4: Revision for the captions**\n\n> A more descriptive caption for the Motion Control Pipeline (Figure 5) and a clearer title for the caption of Figure 6 are advised for better comprehension. (MOFT v.s. DIFT)\n\nThank you for your suggestion. We will include the following revisions:  \nRevised caption of Figure 5: In one denoising step, we get the intermediate features and extract MOFT from it with content correlation removal and motion channel filter. We optimize the latents to alter the sampling process with the loss of masked MOFT and reference MOFT. For the detailed motion control process, please refer to the pseudo-code.  \nRevised title of Figure 6: Effects of DIFT and MOFT on different denoising time steps.\n\n**Q1: Clarification on the residual information**\n\n> After eliminating content correlation information, does the feature retain any information other than motion? If so, could this residual information affect video generation?\n\n As illustrated in Fig. 2 of the main paper, after removing content correlation information, videos with entirely different appearances but identical motion directions cluster closely together (e.g., Right 1 and Right 2). This suggests that motion information is the primary component retained in the feature. Additionally, results presented in Figure 7 and the supplementary material further demonstrate that reference videos have minimal influence on appearance, lighting, or other semantic aspects. Thus, the residual information beyond motion has a negligible effect on video generation.  \n\n**Q2: Manipulating features from earlier generation steps**\n\n> Figure 6 illustrates how the ability to track motion varies with generation steps. Is it possible to manipulate features from earlier generation steps to achieve better results?\n\nGood idea! We do manipulate the features from earlier generation steps.\nWe manipulate features from earlier generation steps for two primary reasons:\n1. MOFT provides valid motion information early in the generation process, allowing us to manipulate the motion effectively.\n2. The motion in the generated videos is established during the early generation steps, necessitating manipulation at these stages.\nHowever, at the very beginning of the generation process, MOFT does not offer a very clear motion pattern. Therefore, we typically apply optimization at 850 to 500 denoising time steps for a 1000-timestep denoising process.\n\n**The inclusion of limitations**\n\nWe've described the limitations in the Supplementary due to the limited space of the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sdHfwmkAXp", "reviewer": "Reviewer_zmG9", "summary": "This paper investigates the relationship between the features of video diffusion models and the motion in the generated videos. By extracting motion features and using them as guidance, training-free motion control can be achieved.", "questions": "Please refer to the waknesses.", "limitations": "The limitations are discussed while the potential negative societal impact is not discussed. But I don't think this discussion is necessary.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The technical aspects of this paper are clear and it is easy to read.\n2. The proposed method can achieve training-free motion control for video generation.\n3. The framework can be applied to different forms of control signals.", "weaknesses": "1. From Fig. 6, it is hard to draw the conclusion that \"MOFT can provide more valid information than DIFT at the early design stages\".\n2. The analysis experiments in the method section only focused on very simple motions, such as pan up, down, left, and right, without discussing more complex and realistic motions. \n3. The generated motions presented in the experiment section are also mostly very simple.\n4. The experiments lack comparisons with existing methods. Comparisons with other methods were only made in the point drag mode.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback. I've provided detailed responses to each of your concerns below.  \n\n**W1: Clarification on the conclusion from Fig. 6**\n> From Fig. 6, it is hard to draw the conclusion that \"MOFT can provide more valid information than DIFT at the early design stages\".\n\nFrom Figure 6, we observe that at early denoising steps (e.g., time step 800), DIFT struggles to provide valid information as the semantic correspondence is wrong. In contrast, MOFT delivers a relatively high score in the corresponding motion region (the rabbit's head), which benefits motion control. This observation is further validated by Figure 4 in the supplementary material.\nNote: The similarity heatmaps serve different purposes for each method. In DIFT, the heatmap represents semantic similarity, whereas in MOFT, it represents motion similarity. Therefore, we do not directly compare the heatmaps between DIFT and MOFT at the same time steps. Instead, we visualize their effects at different time steps to highlight that MOFT performs better at the earlier denoising stages.\n\n\n**W2: Reasons for using simple motion in analysis experiments**\n> The analysis experiments in the method section only focused on very simple motions, such as pan up, down, left, and right, without discussing more complex and realistic motions.\n\nWe select simple motions in the analysis experiments for  two reasons: \n(1) Simplifying the analysis allows for clearer insights.\n(2) It is sufficient to filter out motion channels. Motion features obtained in this way generalize well to more complex motions because complex motions are composed of simple ones, i.e., different simple motions occurring at different times and spaces combine to form complex motions.\n\n**W3: Presented motions are not simple**\n> The generated motions presented in the experiment section are also mostly very simple.\n\nIn Figure 8 and the supplementary webpage, we showcase complex motions, including camera motion control, motion control of multiple objects, different motion frequencies and directions, etc. These examples cover common aspects of motion control, such as those in Motionctrl [12], SMM [2], and Cameractrl [13]. We welcome suggestions for additional test cases from the reviewers.\n\n\n**W4: Comparisons with existing methods**\n> The experiments lack comparisons with existing methods. Comparisons with other methods were only made in the point drag mode.\n\nIn addition to comparisons using the point drag mode, we also evaluate our approach against SMM [2], a method that extracts intermediate features for motion control. \nTo make the comparison more comprehensive, we include a qualitative comparison on motion transfer tasks with methods [2,14] (please refer to Fig.1 and Table 1 of the attached PDF), demonstrating our ability to achieve a superior balance between motion fidelity and text alignment. This improvement is attributed to our innovative motion decomposition designs.\nBeyond these comparisons, we emphasize that our approach offers deeper insights into motion interpredictions in video diffusion features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bBnWUrH3qI", "reviewer": "Reviewer_P4uW", "summary": "The paper introduces a training-free framework for understanding and controlling motion in video diffusion models. The key innovation is the MOtion FeaTure (MOFT), which is derived by removing content correlation and filtering motion channels from pre-trained diffusion model features. MOFT provides a training-free way to encode and manipulate motion information, offering high interpretability and generalizability across various architectures. The framework demonstrates competitive performance in generating natural and faithful motion, with applications in video motion control and point-drag manipulation.", "questions": "Providing a detailed description or pseudocode for MOFT extraction would aid in understanding its practical implementation and reproducibility. \nHave you tested MOFT's scalability for generating longer videos or higher-resolution outputs? What challenges, if any, did you encounter, and how did you address them? \nAdditionally, how does this method handle more complex object control, such as multi-object scenarios with different categories and sizes?", "limitations": "The author have discussed limitations in Supp.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Training-free Approach: The framework does not require additional training, leveraging pre-trained diffusion models to control motion, significantly reducing resource requirements.\n2. Interpretability: MOFT offers a clear and interpretable way to understand and manipulate motion information in video diffusion models.\n3. Generalizability: The method is applicable across various video generation models, demonstrating versatility and robustness.", "weaknesses": "Scalability to Longer Videos: The proposed method's scalability to longer videos or higher resolutions is not adequately explored.\n\nComplexity of MOFT Extraction: Removing content correlation and filtering motion channels may be complex and require fine-tuning for optimal results.\n\nExperiments: The text prompts used for quantitative experiments and user studies are unclear, and 56 case studies are insufficient to validate effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. You will find detailed responses to each of the points you raised below.  \n\n**W1: Scalability to longer videos and higher resolutions**\n> Scalability to Longer Videos: The proposed method's scalability to longer videos or higher resolutions is not adequately explored\n\nThanks for your suggestions. We've added experiments showing that our methods can be directly generalized to higher resolutions and longer videos. In Figure 3(a) of the attached PDF, we demonstrate that PCA can clearly separate videos with different motions based on their diffusion features from Open-Sora [5], an open-source video generation model capable of producing long videos. In Figure 3(b), we show that our methods can be applied to higher resolutions (768×768) and longer videos (205 frames on Open-Sora).\n\n**W2: Clarification on MOFT extraction and optimization process**\n> Complexity of MOFT Extraction: Removing content correlation and filtering motion channels may be complex and require fine-tuning for optimal results.\n\nThe extraction of MOFT is both straightforward and efficient, requiring only a subtraction operation and mask indexing, thereby adding minimal time to the overall process. Our method does not require fine-tuning of model parameters. Instead, it only involves efficient optimization during the inference stage, which is a common technique in image and video editing to guide the generation process [2,6,9].\nFor further clarification on the optimization process, please refer to the pseudo-code in the global response.\n\n**W3: Clarification on user studies**\n> Experiments: The text prompts used for quantitative experiments and user studies are unclear, and 56 case studies are insufficient to validate effectiveness.\n\nWe have added the prompts for the test in the appendix, some of them are shown below.\n- b&w photo of 42 y.o man in black clothes, bald, face, half body, body, high detailed\n    skin, skin pores, coastline, overcast weather, wind, waves, 8k uhd, dslr, soft  \n- a rabbit, forest, haze, halation, bloom, dramatic atmosphere, centred, rule of\n    thirds, 200mm 1.4f macro shot  \n- a white deer in the snow, cartoon, centred  \n- a man surfing in the sea, RAW photo, subject, (high detailed skin:1.2), 8k uhd,\n    dslr, soft lighting, high quality, film grain, Fujifilm XT3  \n- a car turns in the winter, 8k uhd, dslr, soft  \n- photo of coastline, rocks, storm weather, wind, waves, lightning, 8k uhd, dslr,  \n    soft lighting, high quality, film grain, Fujifilm XT3  \n- night, old house, post apocalypse, forest, wind, rocks, 8k uhd, dslr, soft lighting,\n    high quality, film grain  \n- ...\n\nUser study usually does not include samples with the scale of quantitative evaluation due to the labor consumption. 56 case studies are common numbers in user studies (~18 samples for each method).  For example, Dragondiffusion [6] (ICLR 2024 , 16 samples for each method), Rerender-A-Video [10] (Siggraph Asia 2023, 8 samples for each method), FateZero [11] (ICCV 2023, 9 samples for each method)\n\n**Q1: Challenges and solutions in this work**\n> What challenges, if any, did you encounter, and how did you address them?\n\nThe main challenge lies in the extraction of MOFT. Unlike DIFT, which directly uses intermediate features as the Diffusion Feature (DIFT) because it encodes rich semantic information, video diffusion features entangle various types of information, including semantic and motion information. Decomposing motion information from these features is not straightforward. Fortunately, our proposed method effectively localizes and extracts motion features, inspired by recent works on understanding video latents [8]. In addition, we want to highlight that the contribution of this paper extends beyond the technical aspects; it also discloses the finding that motion-aware features naturally exist in video diffusion models and provides a clear interpretation of how video diffusion features encode motion information.\n\n**Q2: Solutions for complex cases**\n> Additionally, how does this method handle more complex object control, such as multi-object scenarios with different categories and sizes?\n\nOur method can be naturally applied to multi-object scenarios involving different categories and sizes, thanks to the excellent generalization ability of video generation models. In multi-object scenarios, we employ multiple masks and use different optimization targets for these masked regions, enabling effective multi-object control. We will include more complex cases in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rx9LitT5gZ", "reviewer": "Reviewer_vDWU", "summary": "This paper introduces a new Motion Feature (MOFT) that can effectively capture motion information in video diffusion models. The authors reveal that robust motion-aware features already exist in video diffusion models, allowing to encode comprehensive motion information with clear interpretability. They present MOFT, which can be extracted without the need for training and is generalizable across diverse architectures.", "questions": "Clarification needed:\n\n- Figure 1:\n\n  a. It is unclear whether the motion feature in 1(a) is extracted semantically or spatially. Clarification is needed on how the similarity with other videos in 1(b) is calculated. Additionally, an explanation of what the higher score represents and why motion features from different videos could influence each other would be helpful.\n\n  b. In 1(c), the motion direction seems to be manually defined. If so, why does the paper state that MOFT serves as guidance for controlling motion direction? If MOFT controls the motion, what is the source video for that motion?\n\n- Figure 6: Why the comparison is presented in the form of a point for DIFT and a segment for MOFT.", "limitations": "Limitations are addressed adequately.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Training-free strategy effectively extracts motion information encoded in the features of the video diffusion model, demonstrating its ability to capture and leverage the inherent motion representations learned by the model.\n\n- The method presents a clean and straightforward solution for extracting motion encoding from video diffusion models, making it a ready and practical technique for various applications involving motion analysis or synthesis.", "weaknesses": "- The paper lacks clarity on the training process. While it claims to be training-free, it defines loss functions for other tasks (Equations 3 and 4). It would be helpful to clarify which stages are trained and which are not.\n\n- The PCA analysis is based on a small number of videos (only 2 videos in Figure 2), which limits the generalizability of the results.\n\n- While the motion in the qualitative videos looks good, the differences compared to other alterations appear subtle and hard to recognize. Other methods show too poor results, were they tuned correctly?\n\n- The paper should report the runtime and resolution for better understanding of the method's computational requirements and output quality.\n\n- The idea is heavily inspired by DIFT and utilized for video applications, then novelty seems limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your valuable input. Please see the detailed responses to each of your concerns listed below.  \n\n**W1: Clarification on the training process**\n> The paper lacks clarity on the training process. It would be helpful to clarify which stages are trained and which are not.\n\nOptimization does not necessarily mean training. \"Training-free\" means that we do not need to optimize **model parameters** during the training stage. Instead, the loss function is used to optimize the **latent** at each denoising step during the inference stage, which is a common technique to guide the diffusion process [1, 2].  We will add this clarification in the new version. Please refer to the pseudo-code in the global response for the details of the optimization process.\n\n**W2: Clarification for the number of videos in PCA**\n> The PCA analysis is based on a small number of videos (only 2 videos in Figure 2), which limits the generalizability of the results.\n\nIn Figure 2, PCA analysis is based on six videos, with indices 1 and 2 showing the same motion in different videos. We do not consistently use the same pair of videos to demonstrate that our observations are not tied to specific videos. Although the sample size is small, our results show that PCA-filtered motion channels are robust to the number of videos.\n\nTable 2 in the attached PDF compares channel similarity between PCA results from 6 and 20 videos, showing high similarity in filtered motion channels, indicating PCA's robustness. Table 3 further confirms that motion fidelity and image quality metrics are also robust to the number of videos used.\n\n**W3: Clarification on the qualitative comparison**\n> While the motion in the qualitative videos looks good, the differences compared to other alterations appear subtle and hard to recognize. Other methods show too poor results, were they tuned correctly?\n\nIn the qualitative comparison with other methods, we focus on challenging cases for existing methods. In these cases, other methods often struggle: for example, DragNUWA [3] typically only moves part of the object, and Gen-2 [4] frequently generates unnatural movements. Our method, however, performs well and generates natural motions even in these challenging scenarios. We do not need to fine-tune these methods; instead, we directly test their results using publicly available code or APIs.  \n\n**W4: Report on the runtime and resolution**\n> The paper should report the runtime and resolution for better understanding of the method's computational requirements and output quality.\n\nOur results are at a resolution of 512x512 and 16 frames unless otherwise specified. We use DDIM with 25 denoising steps for each sample. It takes approximately 3 minutes to generate one sample on an RTX 3090 GPU. We will include this information in the revision.\n\n**W5: Clarification on the novelty**\n> The idea is heavily inspired by DIFT and utilized for video applications, then novelty seems limited.\n\nThe inspiration from DIFT lies in the high-level research style, as both works analyze diffusion features. However, our motivations, contributions, and techniques are distinct:\n- Motivation: We aim to decompose motion features from video diffusion features for better motion understanding and control, unlike DIFT which targets semantic correspondence in image diffusion models.\n- Contribution: MOtion FeaTure (MOFT) is the first to reveal rich motion information in video diffusion features, using a straightforward and innovative approach. DIFT shows diffusion features can capture semantic correspondence.\n- Technique: MOFT uses a novel PCA-based strategy to extract motion-aware features, while DIFT directly uses intermediate features without further processing.\n\n\n**Q1-1: Clarification on the motion feature**\n\nThe motion feature in 1(a) is extracted following Eq. 2. The extracted feature contains spatial dimension, but the feature of a point location captures temporal motion of that point instead of semantics. The heatmap in 1(b) is calculated by the cosine similarity between the MOFT at the *red dot* in 1(a) and the MOFT of all points in 1(b), expressed as: $M=cosine(M_a,M_b)$, where $M\\in \\mathbb{R}^{H×W}$ is the output heatmap, $M_a \\in \\mathbb{R}^{C}$ is the MOFT of one point in the source image, and $M_b\\in \\mathbb{R}^{H\\times W\\times C}$ is the MOFT of all points in the target image. $H$, $W$ and $C$ are height, width and channel number, respectively. We use only one frame to calculate the similarity and for visualization.\n\nHigher scores indicate greater motion similarity. For example, higher scores in the man’s region (last case in 1(b)) show that the man’s motion direction matches the reference (red dot) in 1(a), both moving left. This method also allows us to manipulate MOFT to alter motion direction.\n\n**Q1-2: Clarification on the motion signal**\n\nWe need to transfer manually defined motion directions to MOFT for the motion optimization target, otherwise the optimization is not feasible.\nFor obtaining the target MOFT, we offer two methods:\n- Synthesized from manually defined motion directions (as shown in Fig. 7(c) and 7(d)), which is the case discussed in this question. Details of this process are described in Line 175-179 of the paper.\n- Extracted from the reference source video (Fig. 7(a) and 7(b)).\n\n\n**Q2: Clarification on Figure 6**\n\nThe similarity heatmaps serve different purposes in each method.\n\nIn DIFT, the similarity heatmap shows semantic similarity between points in the target and source images, focusing on a **one-to-one** correspondence to find the most accurate match.\n\nIn MOFT, the heatmap represents motion similarity between points in the target image and a reference point in the source image. Here, the heatmap visualizes **regions with similar motion** rather than finding a single best match.\n\nWe visualize their effects at different time steps to highlight the MOFT works better at the earlier denoising stage, instead of direct comparisons on effects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for your thorough, insightful, and constructive feedback. We are pleased that the **interpretability and clarity of MOFT** have been recognized (Reviewers P4uW, vDWU) and that the **technical soundness** of our paper has been acknowledged (Reviewers zi7R, zmG9, P4uW, vDWU). We also appreciate the recognition of **MOFT's versatility** and its potential for diverse applications (Reviewers zi7R, zmG9, P4uW, vDWU), as well as the effectiveness of our **training-free** approach (Reviewers zi7R, zmG9, P4uW, vDWU).
We have addressed each of your comments with care and have provided detailed clarifications in our responses and the attached PDF. Please refer to the responses below for further details.

Thank you for your commitment. We eagerly anticipate your continued feedback and hope that you find the responses to be satisfactory.

---
Following is the pseudo-code of the optimization process since some reviewers raise the common question.

Algorithm: Optimization process  

Input: noisy latents at timestep t $z_t$, region mask $S$, reference MOFT $MO^g$, the network $\epsilon_{\theta}$, Motion Channel Mask $C$, learning rate $\eta$.  

Output: optimized latents $z_t^{new}$.

Begin  

  $X=\epsilon_{\theta}(z_t)$ $\triangleright$ Get intermediate feature $X$ from the network  
Given $C$, extract MOFT $MO$ by Eq. 2 in the main paper  
Given $S$, $MO^g$, $MO$, get the loss $l$ by Eq. 3 in the main paper  
Optimize $z_t$ by $z^{new}_t = z_t - \eta \frac{\partial l}{\partial z_t}$  

Return $z_t^{new}$  
End

---

Reference: due to the rebuttals' character limit, we've placed the references for all rebuttals below.

[1] Shi Y, et al. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing, CVPR 2024.  
[2] Yatim D, et al. Space-time diffusion features for zero-shot text-driven motion transfer, CVPR 2024.  
[3] Yin S, et al. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv  2023.  
[4] Gen-2, https://runwayml.com/research/gen-2.  
[5] Zheng Z, et al. Open-Sora: Democratizing Efficient Video Production for All.  
[6] Mou C, et al. Dragondiffusion: Enabling drag-style manipulation on diffusion models. ICLR 2024.  
[7] Ouyang W, et al. I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models. arXiv preprint 2024.  
[8] Luo Z, et al. Videofusion: Decomposed diffusion models for high-quality video generation, CVPR 2023.  
[9] Shi Y, et al. Dragdiffusion: Harnessing diffusion models for interactive point-based image editing, CVPR 2024.  
[10] Yang S, et al. Rerender a video: Zero-shot text-guided video-to-video translation. SIGGRAPH Asia 2023 Conference Papers.  
[11] Qi C, et al. Fatezero: Fusing attentions for zero-shot text-based video editing, ICCV 2023.  
[12] Wang Z, et al. Motionctrl: A unified and flexible motion controller for video generation, ACM SIGGRAPH 2024 Conference.  
[13] He H, et al. Cameractrl: Enabling camera control for text-to-video generation, arXiv preprint  2024.  
[14] Gen-1, https://runwayml.com/research/gen-1.

Meta-review: According to the reviews, the reviewers acknowledge the paper's potential in utilizing internal features of video diffusion models to capture motion information and its applications for various manipulations, such as reference-based and drag-based control. Following the rebuttal, all reviewers have provided positive ratings for the paper. However, they have also requested that the authors incorporate the materials discussed during the rebuttal into the revised paper as a condition for acceptance. It is advised that the authors carefully follow the reviewers' suggestions and comments to refine the manuscript accordingly.  For example, please incorporate the reviewers' feedback by providing a detailed explanation of the content correlation information in Section 3 and elaborating on the optimization process of the latent variables in Section 4.1. Additionally, update the motion control pipeline in Figure 5, ensure the use of identical sample images in Figure 6 for the similarity heatmaps of DIFT and MOFT, and include the results of more complex motions in the experimental section.
Decision: Accept (poster)