Motion Inversion for Video Customization

## Abstract

Abstract In this work, we present a novel approach for motion customization in video generation, addressing the widespread gap in the exploration of motion representation within video generative models. Recognizing the unique challenges posed by the spatiotemporal nature of video, our method introduces Motion Embeddings , a set of explicit, temporally coherent embeddings derived from a given video. These embeddings are designed to integrate seamlessly with the temporal transformer modules of video diffusion models, modulating self-attention computations across frames without compromising spatial integrity. Our approach provides a compact and efficient solution to motion representation, utilizing two types of embeddings: a Motion Query-Key Embedding to modulate the temporal attention map and a Motion Value Embedding to modulate the attention values. Additionally, we introduce an inference strategy that excludes spatial dimensions from the Motion Query-Key Embedding and applies a differential operation to the Motion Value Embedding, both designed to debias appearance and ensure the embeddings focus solely on motion. Our contributions include the introduction of a tailored motion embedding for customization tasks and a demonstration of the practical advantages and effectiveness of our method through extensive experiments.

## 1 Introduction

In recent years, generative models have rapidly evolved, achieving remarkable results across various domains such as image (Rombach et al., ; Nichol et al., ; Ramesh et al., ; Betker et al., ; Saharia et al., ) and video (He et al., ; Chen et al., ; Guo et al., ; Wang et al., ). Within the realm of imagery, customization is a popular topic, empowering models to learn specific visual concepts from user-provided images at both the object and style levels.
These concepts are combined with the model’s extensive prior knowledge to produce diverse and customized outcomes.
The success of image customization has led to high expectations for extending such capabilities to video generation models, which are developing rapidly and drawing significant attention.

However, extending these techniques to Text-to-Video (T2V) generation introduces new challenges due to the spatiotemporal nature of video. Unlike images, videos contain motion in addition to appearance, making it essential to account for both.
Current customization methods (Hu et al., ; Mou et al., ; Sohn et al., ; Ye et al., ; Zhang & Agrawala, ; Gal et al., ; Ruiz et al., ) primarily focus on appearance customization, neglecting motion, which is critical in video.
Motion customization deals with adapting specific movements or animations to different objects or characters, a task complicated by the diverse shapes and dynamic changes over time (Siarohin et al., ; ; Yatim et al., ; Jeong et al., ).
These methods, however, fail to capture the dynamics of motion. For instance, textual inversion (Gal et al., ) learns embeddings from images but lacks the ability to capture temporal correlations, which are essential for video dynamics. Similarly, fine-tuning approaches like DreamBooth (Ruiz et al., ) and LoRA (Hu et al., ) struggle to disentangle motion from appearance.

In this paper, we address the challenge of motion customization, focusing on the critical issue of motion representation.
The current state-of-the-art methods face several limitations:
1) Some approaches lack a clear representation of motion, as seen in Yatim et al. (), where motion is only indirectly injected through loss construction and optimization at test time, leading to additional computational overhead.
2) Some other methods (Jeong et al., ) attempt to parameterize motion as a learnable representation, yet they fail to separate these parameters from the generative model. This coupling compromises the generative model’s diversity after learning.
3) While there are also some methods that attempt to separate motion representation from the generative model using techniques like low-rank adaptation (LoRA) (Hu et al., ), such as in Motion Director (Zhao et al., ), they lack a well-defined temporal design, limiting their effectiveness in capturing motion dynamics, as evidenced by our experiments.

To address the aforementioned issues, we propose a novel framework for motion customization. Our method learns explicit, temporally coherent embeddings, termed Motion Embeddings, from a reference video.
These embeddings are integrated into the temporal transformer modules of the video diffusion model, modulating the self-attention across frames.

We introduce two types of motion embeddings: Motion Query-Key Embedding, which captures global relationships between frames by influencing the temporal attention map (QK), and Motion Value Embedding, which captures spatially varying movements across frames by modulating the attention value (V).
The Motion Query-Key Embedding excludes spatial dimensions ($H$ and $W$) to avoid capturing appearance information, as the temporal attention map inherently carries spatial details of objects, which could entangle appearance information of the reference video and thus hinder motion transfer. While the Spatial-2D Motion Value Embedding may still risk capturing static appearance information, we address this by introducing a debiasing strategy that models frame-to-frame changes, ensuring that the embeddings primarily represent motion dynamics. This approach is conceptually similar to techniques like optical flow, where motion is isolated by tracking changes between frames, helping to prevent overfitting to specific appearance details and improving generalization to novel content.

In summary, our contributions are as follows:

- •
We propose a novel motion representation for video generation, addressing the key challenges in motion customization.
- •
We design two approaches to debias appearance for this motion representation: a 1D Motion Query-Key Embedding that captures global temporal relationships, and a 2D Motion Value Embedding with a differential operation that captures spatially varying movements across frames.
- •
Our method is validated through extensive experiments, demonstrating its effectiveness and flexibility for integration with existing Text-to-Video frameworks.

## 2 Related Work

Text-to-Video Generation.  Text-to-Video (T2V) generation task aims to synthesize high-quality video from user-provided text prompts, which are composed of the expected appearances and motions. Previously, Generative Adversarial Networks (GANs) (Vondrick et al., ; Saito et al., ; Pan et al., ; Li et al., ; Tian et al., ) and Autoregressive Transformers (Yan et al., ; Le Moing et al., ; Wu et al., ; Hong et al., ) have been widely explored in this area. On the other hand, diffusion models have demonstrated powerful generation capabilities in the field of Text-to-Image (T2I) generation (Rombach et al., ; Nichol et al., ; Ramesh et al., ; Betker et al., ; Saharia et al., ) and have begun to extend to video generation (He et al., ; Chen et al., ; Wang et al., ; He et al., ). Recently, several works have tried to transfer the pretrained T2I diffusion models to T2V generation models by inserting temporal layers into the image generation networks such as AnimateDiff (Guo et al., ), and Make-a-Video (Singer et al., ).
These Text-to-Video (T2V) models approach frame generation as a series of image creations, integrating a temporal transformer to bolster inter-frame relationships—a notable deviation from Text-to-Image (T2I) models (He et al., ; Chen et al., ; Wang et al., ; Singer et al., ; Zhang et al., ; ; Chen et al., ; cerspense, ). Additionally, certain approaches incorporate an extra 3D convolutional layer to enhance temporal consistency (cerspense, ; Wang et al., ).
These T2V models are designed for video generation through text inputs and may encounter difficulties when needed to produce videos with customized motions.

Video Editing.  Video editing generates video that adheres to the target prompt as well as preserves the spatial layout and motion of the input video.
As the basis of video editing, image editing has achieved significant progress by manipulating the internal feature representation of prominent T2I diffusion models (Cao et al., ; Chefer et al., ; Hertz et al., ; Ma et al., ; Tumanyan et al., ; Patashnik et al., ; Bar-Tal et al., ; Qi et al., ; Liu et al., ).
MagicEdit (Liew et al., ) takes use of SDEdit(Meng et al., ) for each video frame to conduct high-fidelity editing.
Tune-A-Video (Wu et al., ) finetunes a T2I model on the source video and stylizes the video or replaces object categories via the fine-tuned model.
Control-A-Video (Chen et al., ) presents Video-ControlNet, a T2V diffusion model that generates high-quality, consistent videos with fine-grained control by incorporating spatial-temporal attention and novel noise initialization for motion coherence.
TokenFlow(Geyer et al., ) performs frame-consistent editing by the feature replacement between the nearest neighbor of target frames and keyframes.
However, these methods fall short as they just duplicate the original motion almost at pixel-level, resulting in failures when being require significant structural deviation from the original video.

Video Motion Customization.  Motion customization involves generating a video that maintains the motion traits from a source video, such as direction, speed, and pose, while transforming the dynamic object to match a text prompt’s specified visual characteristics.
This process is distinct from video editing (Bar-Tal et al., ; Chen et al., ; Wu et al., ; Geyer et al., ; Liew et al., ; Qi et al., ), which typically transfers motion between similar videos within the same object category.
In contrast, motion customization requires transferring motion across diverse object categories, often involving significant shape and deformation changes over time, necessitating a deep understanding of object appearance, dynamics, and scene interaction (Yatim et al., ; Jeong et al., ; Zhao et al., ; Ling et al., ; Jeong et al., ).
Diffusion Motion Transfer (DMT) (Yatim et al., ) injects the motion of the reference video through the guidance of a handcrafted loss during inference, bringing additional computation costs that could not be ignored.
Video Motion Customization (VMC) (Jeong et al., ) encodes the motion into the parameters of a T2V model. However, finetuning the original T2V model could seriously limit the diversity of the generation model after learning the motion.
Motion Director(Zhao et al., ) adopts LoRA(Hu et al., ) to embed the motion outside the T2V model.
Nevertheless, the structure of LoRA limits the scalability and interpretability, as we could not easily integrate several reference motions by these methods. Other works that represent motion using parameterization (Wang et al., ; He et al., ) or trajectories (Ma et al., ; Yin et al., ), but these approaches fall outside the scope of our discussion on reference video-based methods.

## 3 Methodology

### 3.1 Text-to-Video Diffusion Model

In video diffusion models, the evolution from Text-to-Image (T2I) to Text-to-Video (T2V) models is marked by the introduction of the temporal transformer module to the basic block.
While T2V models utilize spatial convolutional layers and spatial transformers in basic block for integrating image features and textual information (Rombach et al., ; Nichol et al., ; Ramesh et al., ; Betker et al., ; Saharia et al., ) , T2V models build on this by adding the temporal transformer (He et al., ; Chen et al., ; Guo et al., ; Wang et al., ).
This module is key for video generation, enabling the treatment of videos as sequences of batched images.
It specifically handles the inter-frame correlations through a frame-level self-attention mechanism, ensuring the temporal continuity vital for dynamic video content.

In this module, an input spatiotemporal feature tensor is provided, initially shaped as $\mathbf{X}\in\mathbb{R}^{1\times C\times N\times H\times W}$, where $C,N,H,W$ represents channels, number of frames, height, and width respectively. Batch size equals to one, and we omit the batch size dimension in our later notation for simplicity.
This tensor is subsequently transformed into a feature tensor $\mathbf{F}$, with dimensions $\mathbf{F}\in\mathbb{R}^{(H\times W)\times N\times C}$.
The temporal attention mechanism (TA) within this module specifically targets the $N$ dimension, corresponding to frames.

To facilitate this operation, $F$ is projected through three distinct linear layers to generate the Query ($\mathbf{Q}=\mathbf{W_{q}}\mathbf{F}$), Key ($\mathbf{K}=\mathbf{W_{k}}\mathbf{F}$), and Value ($\mathbf{V}=\mathbf{W_{v}}\mathbf{F}$) matrices, respectively.
This setup enables the execution of self-attention across the frame sequence, encapsulated by the formula:

$$ $\text{TA}(\mathbf{F})=\text{softmax}\left(\frac{\mathbf{QK^{T}}}{\sqrt{d_{k}}} \right)\mathbf{V},$ (1) $$

where $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are the query, key, and value matrices obtained from $\mathbf{F}$, and $d_{k}$ represents the dimensionality of the key vectors, serving as a scaling factor to maintain numerical stability within the softmax function.
This temporal attention mechanism allows each frame’s updated feature to gather information from other frames, enhancing the inter-frame relationships and capturing the temporal continuity essential for video generation.

### 3.2 Our Proposed Method

Figure: Figure 2: Motion Inversion within T2V diffusion models. The top depicts the training phase, where motion embeddings $\mathcal{M}$ are learned by backpropagating the loss through the temporal transformer, influencing the spatiotemporal feature tensor $\mathbf{F}$. These embeddings are then used to modify the self-attention computations within the temporal transformer modules, ensuring enhanced inter-frame dynamics. The bottom shows the inference phase, where an input text prompt guides the generation of a coherent video sequence with the learned motion embeddings applied across the frames, producing a customized video output with desired motion attributes.
Refer to caption: x2.png

At the heart of our method for enhancing inter-frame dynamics in video models is the innovative
motion embedding concept:

$$ $\displaystyle\mathcal{M}$ $\displaystyle=\{\mathcal{M^{QK}},\mathcal{M^{V}}\},$ (2) $\displaystyle\mathcal{M^{QK}}$ $\displaystyle=\{\mathbf{m}_{1}^{QK},\mathbf{m}_{2}^{QK},\ldots,\mathbf{m}_{L}^ {QK}\},\quad\text{where each }\mathbf{m}_{i}^{QK}\in\mathbb{R}^{1\times N \times C},$ $\displaystyle\mathcal{M^{V}}$ $\displaystyle=\{\mathbf{m}_{1}^{V},\mathbf{m}_{2}^{V},\ldots,\mathbf{m}_{L}^{V }\},\quad\text{where each }\mathbf{m}_{i}^{V}\in\mathbb{R}^{(H*W)\times N \times C}.$ $$

We have designed two distinct types of motion embeddings, each influencing different parts of the temporal attention computation. The Motion Query-Key Embedding $\mathbf{m}_{i}^{QK}$ is a learnable vector with the shape $(1,N,C)$, while Motion Value Embedding $\mathbf{m}_{i}^{V}$ is a learnable matrix with the shape $(H\times W,N,C)$. These embeddings are seamlessly integrated into the spatiotemporal feature tensor $\mathbf{F}$. The variable $L$ denotes the total number of temporal attention modules within the model. Our motion embeddings directly influence the self-attention computation as follows:

$$ $\text{TA}_{i}(\mathbf{F})=\text{softmax}\left(\frac{(\mathbf{W_{q}}(\mathbf{F} +\mathbf{m}_{i}^{QK}))(\mathbf{W_{k}}(\mathbf{F}+\mathbf{m}_{i}^{QK}))^{T}}{ \sqrt{d_{k}}}\right)(\mathbf{W_{v}}(\mathbf{F}+\mathbf{m}_{i}^{V})),$ (3) $$

#### Training

Obtaining this embedding is both efficient and convenient. Given a custom video $x_{0}^{1:N}$, $N$ equals to number of frames of this video, we zero-initialize each motion embedding and train the video diffusion model and backpropagate the gradient to the motion embedding:

$$ $\mathcal{M_{*}}=\arg\min_{\mathcal{M}}\mathbb{E}_{t,\epsilon}\left[\left\| \epsilon_{t}^{1:N}-\epsilon_{\theta}(x_{t}^{1:N},t,\mathcal{M})\right\|^{2}_{2 }\right],$ (4) $$

where $\epsilon_{t}$ represents the noise variable at time step $t$, and $\epsilon_{\theta}$ denotes the noise prediction from the pre-trained video diffusion model parameterized by $\theta$.
The whole process is shown in Figure 3.
Our method also supports the loss formulation of (Jeong et al., ) and  (Zhao et al., ), while the latter we found in the experiment can boost our performance too.

#### Inference

During inference time, we apply a differencing operation to modify the optimized motion value embedding and debias the appearance:

$$ $\tilde{\mathbf{m}}_{i}^{V}[:,j,:]=\begin{cases}\mathbf{m}_{i}^{V}[:,j,:],&j=1 \\ \mathbf{m}_{i}^{V}[:,j,:]-\mathbf{m}_{i}^{V}[:,j-1,:],&j>1\end{cases}$ (5) $$

### 3.3 Analysis

Figure: Figure 3: Debiasing appearance from Motion Embeddings. Left: For the Motion Query-Key Embedding, which influences the attention map, we exclude the spatial dimensions. Including them would cause the attention map between frames to capture the object’s shape (e.g., the shape of the tank in the original video is visible in the attention map). Right: Following the concept of optical flow, we apply a differential operation to the Spatial-2D Motion Value Embedding, removing static appearance and preserving dynamic motion.
Refer to caption: x3.png

The motivation of our approach is to fully capture the motion information from the target video without being influenced by its appearance. In this section, we analyze how $\mathcal{M}^{QK}$ and $\mathcal{M}^{V}$ is designed to achieve this.

#### Motion Query-Key Embedding ( ℳ Q ⁢ K superscript ℳ 𝑄 𝐾 \mathcal{M}^{QK} caligraphic_M start_POSTSUPERSCRIPT italic_Q italic_K end_POSTSUPERSCRIPT )

The Motion Query-Key Embedding $\mathcal{M}^{QK}$ is designed to influence the attention map within the temporal transformer modules by adjusting the query and key components.
By adding $\mathcal{M}^{QK}$ to $\mathbf{F}$ before the projection to queries and keys via Equation 3, we effectively modify the computation of the attention weights. These attention weights determine how frames attend to each other across time, which are critical in modeling the motion of the target video.

Additionally, the shape of $\mathbf{m}_{i}^{QK}\in\mathbb{R}^{1\times N\times C}$ is designed to exclude spatial dimensions ($H$ and $W$), which is crucial for removing appearance information.
The rationale behind this is that the temporal attention map models the relationships between spatial regions across frames, inherently carrying the appearance information of objects.
The temporal attention map has a shape of $(H\times W)\times N\times N$.
By examining any one of the attention maps, which has the shape $H\times W$, the object’s shape becomes apparent, as illustrated in Figure 3. If $\mathbf{m}_{i}^{QK}$ included spatial dimensions, appearance details would be captured in the embedding, limiting the ability to transfer motion to new content.

#### Motion Value Embedding ( ℳ V superscript ℳ 𝑉 \mathcal{M}^{V} caligraphic_M start_POSTSUPERSCRIPT italic_V end_POSTSUPERSCRIPT )

As $\mathcal{M}^{QK}$ excludes spatial dimensions, it is better suited for representing global motion (e.g., camera motion) but is less effective at capturing local motion (e.g., instance motion).
To address this, we incorporate the Motion Value Embedding $\mathcal{M}^{V}$ in our representation.
Specifically, $\mathbf{m}_{i}^{V}\in\mathbb{R}^{(H\times W)\times N\times C}$ includes spatial dimensions, allowing the embedding to represent motion at each spatial location across time frames. This fine-grained representation is essential for modeling local object movements and detailed motion information, enhancing the realism and coherence of the generated videos.

However, $\mathcal{M}^{V}$ may capture static appearance information, leading to overfitting and limiting generalization. To address this, we apply the differencing operation from Equation 5, which isolates dynamic motion by subtracting the motion value embedding of the previous frame from the current one, removing static appearance. This approach, similar to optical flow, ensures $\mathcal{M}^{V}$ focuses on motion dynamics, improving generalization to novel text prompts.

## 4 Experiment

Figure: Figure 4: Sample results of our method. Our framework demonstrates exceptional adaptability in capturing a broad spectrum of movements, accurately representing everything from subtle gestures to intricate dynamic actions across various source videos. It also exhibits remarkable flexibility in responding to diverse textual prompts, enabling users to guide the synthesis process with descriptive language for customized motion outputs. Furthermore, our method seamlessly integrates with a range of T2V models such as (a) zero-scope (cerspense, ) and (b) animate-diff (Guo et al., ), showcasing its effectiveness in enhancing video generation with contextually rich and varied motion patterns.
Refer to caption: x4.png

In this section, we employ three motion customization methods as our baselines: Diffusion Motion Transfer - CVPR24 (DMT) (Yatim et al., ), Video Motion Customization - CVPR24 (VMC) (Jeong et al., ), and Motion Director (Zhao et al., ).
For discussions with video editing method, please refer to the supplementary files.
To ensure a fair comparison, both our approach and the baseline methods are integrated with the same T2V model, ZeroScope (cerspense, ) in all experiments.

To be consistent with prior work (Yatim et al., ; Jeong et al., ), our evaluation utilizes source videos from the DAVIS dataset (Perazzi et al., ), WebVID (Bain et al., ), and various online resources.
These videos represent a wide range of scenes and object categories and include a variety of motion types. Comprehensive details on the validation set, prompts used, and implementation specifics of both our method and the baselines are provided in the Supplementary files.
Figure 4 showcases examples of our method in action, illustrating its proficiency in managing substantial alterations to the form and structure of deforming objects while preserving the integrity of the original camera and object movements.

Figure: Figure 5: Qualitative results. Compared to DMT (Yatim et al., ), VMC (Jeong et al., ), and Motion Director (Zhao et al., ), our method not only preserves the original video’s motion trajectory and object poses but also generates visual features that align with text descriptions.
Refer to caption: x5.png

### 4.1 Qualitative evaluation

As illustrated in Figure 5, our method offers a qualitative enhancement over baseline approaches. It excels in preserving the motion trajectory and the object poses of the original video, as evidenced by the consistent positioning and posture of objects between the initial and final frames. Additionally, our technique demonstrates remarkable precision in generating visual features that are congruent with textual descriptions. For instance, in the scenario “a boy walking in a field”, our model adeptly transforms a “walking duck” into a “walking boy”, while preserving the original movement trajectory. In another instance, “a fox sitting in a snowy mountain”, our approach adeptly embodies the essence of a snow-capped mountain scene with high motion fidelity. In contrast, while Motion Director (Zhao et al., ) is capable of producing similar visual features of the snowy mountain, it does not maintain the motion integrity of the original video as effectively as our method.

### 4.2 Quantitative evaluation

To thoroughly evaluate our method against baselines, we conducted assessments across multiple dimensions:

Text Similarity. Following the precedent set by previous research (Geyer et al., ; Esser et al., ; Jeong et al., ; Yatim et al., ), we utilize CLIP (Radford et al., ) to assess frame-to-text similarity, calculating the average score to determine the accuracy of the edits in reflecting the intended textual modifications.

Motion Fidelity.
To evaluate motion transfer effectiveness, we employ the Motion Fidelity Score introduced by (Yatim et al., ). This metric, which utilizes tracklets computed by an off-the-shelf tracking model (Karaev et al., ), measures the similarity between the motion trajectories in the unaligned videos, thus assessing how faithfully the output retains the input’s motion dynamics. The Motion Fidelity Score is defined as:

$$ $\frac{1}{m}\sum_{\widetilde{\tau}\in\widetilde{\mathcal{T}}}\max_{\tau\in \mathcal{T}}\text{corr}(\tau,\widetilde{\tau})+\frac{1}{n}\sum_{\tau\in \mathcal{T}}\max_{\widetilde{\tau}\in\widetilde{\mathcal{T}}}\text{corr}(\tau, \widetilde{\tau}),$ (6) $$

where $\text{corr}(\tau,\widetilde{\tau})$ indicates the normalized cross-correlation between tracklets $\tau$ from the input and $\widetilde{\tau}$ from the output.

Those metrics above are considered for evaluating motion transfer tasks: conformance to the motion of the source video and the depiction of the appearance described by the text prompts.
In addition to these primary metrics, quality evaluation is also conducted.

Temporal Consistency.
Temporal consistency is widely used in video editing tasks to measure the smoothness and coherence of a video sequence (Jeong et al., ; Zhao et al., ; Kahatapitiya et al., ; Wu et al., ; Chen et al., ).
It is quantified by computing the average cosine similarity between the CLIP image features of all frame pairs within the output video.

Fréchet Inception Distance (FID).
The Fréchet Inception Distance (FID), widely recognized for measuring the quality of images produced by generative models (Heusel et al., ), is applied in our study. In our case, images derived from a selection of 89 videos from the DAVIS dataset (Perazzi et al., ) are used as the reference set.

**Table 1: Quantitatve comparisons with existing methods.**
| Method | Text<br>Similarity<br>$\uparrow$ | Motion<br>Fidelity<br>$\uparrow$ | Temporal<br>Consistency<br>$\uparrow$ | FID $\downarrow$ | User<br>Preference<br>$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| DMT (Yatim et al., ) | 0.2883 | 0.7879 | 0.9357 | 614.21 | 16.19% |
| VMC (Jeong et al., ) | 0.2707 | 0.9372 | 0.9448 | 695.97 | 17.18% |
| MD (Zhao et al., ) | 0.3042 | 0.9391 | 0.9330 | 614.07 | 27.27% |
| Ours | 0.3113 | 0.9552 | 0.9354 | 550.38 | 39.35% |

User Study.
To rigorously evaluate our method’s effectiveness, we designed a user study that involved 121 participants. They were presented with 10 randomly selected source videos paired with corresponding text prompts, creating 10 unique scenarios that test the versatility of our approach under varied conditions. For each scenario, participants were shown a set of 4 videos, each generated by a different method but under the same conditions of motion and text prompts. The survey specifically asked participants to identify the video that best conformed to the combination of the source video’s motion and the textual description provided.

Table 1 presents a summary of the results for each metric. Evaluations were performed on a set of 66 video-edit text pairs, comprising 22 unique videos, for all metrics except user preferences. Both our method and Motion Director (Zhao et al., ) scored highly in text similarity. However, our approach surpassed Motion Director in motion fidelity, reinforcing the findings of the qualitative analysis. With respect to video quality, our method demonstrated a slight lag in temporal consistency when compared to VMC (Jeong et al., ), attributable to a lesser number of parameters. Nonetheless, in terms of individual frame quality, VMC was the least effective, significantly underperforming compared to our method. In the user study, our approach garnered a preference rate of 39.35%, the highest among the four methods evaluated, which further substantiates our method’s proficiency in preserving the original video’s motion and responding to text prompts.

### 4.3 Ablation Study

Figure: Figure 6: Ablation Study.
Refer to caption: x6.png

We conducted an ablation study of our method from two key perspectives: the design of motion embeddings and the inference strategy. For the motion embedding design, we evaluated three configurations: (a) spatial-1D $\mathbf{m}_{i}^{QK}$ with spatial-1D $\mathbf{m}_{i}^{V}$, (b) spatial-2D $\mathbf{m}_{i}^{QK}$ with spatial-1D $\mathbf{m}_{i}^{V}$, (c) ours, and (d) spatial-2D $\mathbf{m}_{i}^{QK}$ with spatial-2D $\mathbf{m}_{i}^{V}$.
For the inference strategy, we compared our results with two approaches: (e) normalize, which reduces the mean value from $\mathbf{m}_{i}^{V}$, and (f) vanilla, which does not use the differential operation defined in Equation 5. The results are shown in Figure 6.
The results demonstrate that our motion embedding design achieves a strong balance between capturing the motion of the original videos and generalizing well to diverse text prompts, reducing overfitting. Furthermore, after adopting our inference strategy, the text-to-video similarity is significantly improved.

### 4.4 Limitations

Our performance relies on the generative priors acquired by the T2V model. Consequently, the interplay between a specific target object and the motion in the input video may occasionally fall outside the T2V model’s training distribution. On the other hand, our method may encounter challenges when the input video contains interfering motions from multiple objects, as this can affect the quality of our motion embedding. This is because we learn the overall motion from the entire video rather than focusing on the motion of a specific instance. Addressing this limitation by isolating instance-level motion is a potential area for future improvement.

Figure: Figure 7: Visual Result of the Ablation Study. Left: Ablation of motion embedding design; Right: Ablation of inference strategy. For better visualization, refer to the videos in the supplementary files.
Refer to caption: x7.png

## 5 Conclusion

In conclusion, we presented a novel approach to motion customization in video generation, addressing the challenge of motion representation in generative models. Our Motion Embeddings efficiently capture both global and spatial motion while preserving temporal coherence. Additionally, our inference strategy ensures motion-focused customization by removing appearance influences. Extensive experiments demonstrate the effectiveness of our method, providing a strong foundation for future advancements in instance-level motion learning.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: Please answer the questions in the weaknesses.

In addition, I have another (open-ended) question: How dependent is the performance of the method on the backbone foundational model? I understand that the results would naturally improve as stronger backbones are used, but I want to understand if something like Tune-A-Video + strong backbone model would be equivalent to this method + strong backbone model? 

Are most of the limitations stated in line 83-92 because of the method or the backbone? 

I am happy to revise my score if all my concerns are resolved!

Post rebuttal comment: I have increased my score based on the authors rebuttal.
soundness: 3
strengths: Method:

The method appears pretty solid, it is logical and intuitive. Disentangling the spatial from the temporal, and learning the temporal dynamics using motion key-query and motion-value embeddings in attention is a good way to approach the problem.

Experiments

There are qualitative results, and quantitative comparisons with various metrics - there could be some changes in this part, elaborated more in the weaknesses.

Paper writing

The paper is well written and easy to understand.
summary: The paper proposes a motion customization method for video diffusion models. Specifically, it introduces two types of motion embeddings to capture the global relationships between frames and the spatial movements between frames. The former excludes spatial dimensions to effectively disentangle the temporal dynamics from the spatial information. Qualitative and quantitative comparisons, along with ablations, reveal the effectiveness of this method.
weaknesses: Experiments

I would really like to see more qualitative results. The results provided in the paper and supplementary are extremely limited, which raises questions on whether the method works well only on a small set of videos.

Additionally, I wonder why there are no comparisons with Tune-A-Video. It'll be great if these comparisons can be provided. 

There are also no comparisons with other motion customization methods such as AnimateDiff, MotionMaster, etc - is that because those methods require a specific motion module to be trained?

Method

One clarification - I assume the method is still zero-shot? Are the motion embeddings pre-trained on any videos, or is it all done using just the source video (from scratch)?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: **Questions**

1.	Could the authors provide an intuitive explanation on why we should use 1D and 2D for QK and V, respectively?
2.	Why not differencing the first 2D embedding? 
3.	Why the frames of the output video are less than the source video. For example, supp\results\teaser\4.mp4 (1 seconds) and 4_source.mp4 (2 seconds)

**Small issues**

Line 191, $F$ should be $\mathbf{F}$

Line 320: `Diffusion Motion Transfer - CVPR24 (DMT) (Yatim et al., 2023), Video Motion Customization - CVPR24 (VMC) (Jeong et al., 2023)`
As DMT and VMC have been accepted to CVPR, the reference should be correspondingly updated to Yatim et al., 2024, Jeong et al., 2024
soundness: 3
strengths: **Originality**: the idea of combining 1D motion Q-K embeddings and 2D motion V embeddings is novel to me. The idea of using 1D to remove the spatial information is not new as in DMT. The idea of using differencing operations is also not new as in DMT. However, combining the two kinds of embeddings for two-level global and local motion control is interesting.

**Quality**: The experimental results generally show that the proposed method is effective and show superior performance over the other state-of-the-art methods in CVPR 24 and ECCV 24. From the videos in the supplementary materials, the performance of the proposed method is satisfying. And the effectiveness of each component has been verified by the ablation study.
summary: This paper proposes motion embeddings for model control. The main idea is to adjust the temporal attentions with the features extracted from the source guidance video. Specifically, this paper proposes a 1D motion Q-K embedding to add to the Q and K features to guide the global motions. Moreover, a 2D motion V embedding with differencing pre-processing is proposed to add to the V features to guide the local motions. The experimental results show that motion embeddings can be trained to effectively guide the motion of the output video without being influenced by the appearance of the source video.
weaknesses: As for **clarify**, though the selection of the 1D and 2D embeddings has been studied in the ablation study, generally speaking, it is still hard to understand why the QK should use 1D embeddings and why the V should use 2D embeddings. There lacks some intuitive explanation on why we should use such settings.

Another **unclear part** is that, 2D embeddings use differencing operations to debias the appearance. However, from Eq. (5), only features with $j>1$ are debiased. When $j=1$, the original features with appearance information are used. Why not debiasing the first feature, for example, differencing $j=1$ with $j=L$?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please address the Weakness section
soundness: 3
strengths: 1. Superiority against sota baselines, where evaluation is done both quantitatively and qualitatively.
2. Novel task
3. Clarity of the presentation and writing
3. Novelty in introducing the motion embedding concept
4. Easy to follow the paper
summary: This paper introduces a novel method for motion customization in video generation, addressing the underexplored challenge of motion representation in video generative models. The proposed approach leverages Motion Embeddings—temporally coherent representations derived from video data—to enhance the motion modeling in video diffusion models without compromising spatial integrity. The paper presents two key components: the Motion Query-Key Embedding, which modulates the temporal attention map, and the Motion Value Embedding, which adjusts the attention values. Additionally, the paper proposes an inference strategy that removes spatial dependencies and focuses the embeddings purely on motion. Through extensive experiments, the method demonstrates its ability to effectively customize motion within videos while maintaining efficiency and spatial consistency.
weaknesses: 1. I understand the main method composed of two part; (1)What they optimize and (2)How they optimize.
In terms of the optimization target (1), there exists technical contribution. But in terms of the loss calculation (2), they either use the standard denoising loss, or the VMC/MotionDirector [1,2] loss.
If my understanding above is right, I think the paper should claim more on optimizing the motion embeddings are so important and superior than other options (temporal attention / temporal lora).

2. Finetuning temporal attn would directly hurt the model's original prior but training temporal lora would avoid this problem. Thus, the paper should show theoretical understanding / empirical insights that their eq(3) is superior to temporal lora optimization.

3. L260 ~ L264 need more explanation. What is is trying to achieve from the modulation? If the goal is to achieve appearance debiasing, why would it work? Is this modulation ablated in the paper?

4. In Figure 4 and in the paper in general, what does "excluding spatial dimensions" mean? Do the authors refer to "optimizing temporal attn related parameters instead of self attn related parameters? In Figure 4 left, what are the authors visualizing in the middle? Attention maps of the same module in the t2v unet? And how did they visualize them (e.g. PCA)? 

5. In Fig 4 right, what do the authors mean by "differential"? If they mean residuals across frame-axis? If so, using different phrase would be more appropriate.

6. In quantitative evaluation, for computing temporal consistency, clip is not the optimal choice. Referencing metrics from V-bench [3] would increase the value of the paper.

[1] VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models, CVPR 2024

[2] MotionDirector: Motion Customization of Text-to-Video Diffusion Models, ECCV 2024

[3] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024

---

confidence: 5
contribution: 1
initial_rating: 5
presentation: 3
questions: 1. The paper claims to utilize 1-D QK embeddings for global motion and 2-D V embeddings for spatially varying movements. Could the authors provide results demonstrating their effectiveness when these embeddings function independently?

2. The proposed method targets temporal attention, while the base models decompose attention into separate spatial and temporal components. It would be more compelling to demonstrate the effectiveness of this technique within full attention architectures, where spatial and temporal attention are integrated into a single attention mechanism (e.g., Open-Sora [1]).

[1] Open-Sora: Democratizing Efficient Video Production for All, Zheng et al., 2024.
soundness: 2
strengths: 1. The proposed techniques for motion representation are clear and reasonable.

2. The experiments are comprehensive, featuring a variety of qualitative and quantitative results.

3. The paper is well-written and easy to understand.
summary: This work introduces Motion Embeddings, a method for enhancing motion customization in video generation. The approach proposes a motion representation with two types of embeddings: Motion Query-Key Embedding for temporal attention modulation and Motion Value Embedding for adjusting attention values. An inference strategy is also proposed to focus exclusively on motion by excluding spatial dimensions and using a differential operation.
weaknesses: 1. The primary concern is that the proposed method's effectiveness has already been demonstrated in previous works, such as MotionDirector [1]. For instance, MotionDirector employs motion LoRA in the temporal layer for motion representation, which essentially involves adding weights to the Q, K, and V projectors (Wq, Wk, and Wv in Eq. 3). This paper introduces learnable embeddings added to the input features in Eq. 3. While these two embedding approaches are not identical, their functionality appears similar, thus limiting the novelty of the contribution.

2. The interpretability of the learned embeddings is insufficiently addressed. Although the paper claims improved interpretability, no convincing evidence is provided. For instance, could the authors visualize the learned QK embeddings to determine if the attention maps exhibit meaningful patterns? Alternatively, are the learned V embeddings interpretable in any discernible way?

[1] MotionDirector: Motion Customization of Text-to-Video Diffusion Models. Zhao et al., 2024.

Meta-review: The paper addresses the problem of motion customization for video models (given a video, generate a video with its approximate motion) and proposes a "motion embedding" which is designed and trained specifically to capture the motion in the source video. Specifically, this motion embedding consists of motion query-key embeddings which influence the temporal attention query-keys and motion value embeddings which influence the temporal attention values. To ensure that this embedding does not entangle appearance and motion, they design the motion query-key to be 1D (to avoid any spatial knowledge) and specifically debias the motion value. The paper demonstrates strong quantitative results in text similarity, motion fideltiy, temporal consistency, FID, and user preference. In my opinion, due to the small number of samples, it is difficult to fully evaluate the qualitative results (particularly wrt Motion Director).

The main strength of the paper is the relatively clear presentation and strong quantitative results. The core weakness as identified by both Reviewers qixD and 5Vdk is the very close similarity of the method with MotionDirector and the lack of clarity as to the primary differences between the two papers. Since it was published in ECCV24 MotionDirector is prior work and should be treated as such.

The paper demonstrates superior quantitative results over MotionDirector though due to the small number of examples, I found it hard to qualitatively evaluate the two in standard setting. It is however worth noting that this improvement is more pronounced when considering the MSE loss (as opposed to the hybrid).

I read through both this paper, Motion Director, and the extensive discussion in the review section between the author and reviewers. The core concern of Reviewers qixD and 5Vdk is the similarity of a LoRA to that of an embedding which acts as an additive edit to the attention weights. While there are some differences in the exact formulation, the main argument from the authors that I found most convincing is that their method could work on full attention models whereas the LoRA is dependent on the separation between temporal and spatial attention. However, as far as I understand, this is just an argument the authors used and not empirically demonstrated. I checked the unet_3d_condition definition in the diffusers library (used by Zeroscope_v2) and it seems to be a TransformerTemporal which does factorized space-time attention. While I believe that the embedding formulation should work with a full attention model, I think this needs to be empirically demonstrated. In addition, the paper needs to be rewritten to draw a much closer comparison, especially technically, with Motion Director. I think one very simple way that this paper could be accepted in future conference is to demonstrate that their method works on full attention models – then there will be no ambiguity between theirs and MotionDirector. I'm generally sympathetic to the author's argument that their improvement performance demonstrates that there is a difference. However, in the current version of the paper, as both Reviewers qixD, 5Vdk, and I struggled with, it is very hard to understand what the difference is. In order to be accepted, this difference needs to be directly explained.

While I think it's a very close call, I agree with Reviewers qixD and 5Vdk and would rate this paper slightly below, though I strongly encourage the authors to add detail to the paper explaining the difference between this and MotionDirector and then demonstrating this difference empirically on for example a full attention model. I would also encourage the authors to include more qualitative samples in the future, particularly for video papers.

Decision: Reject