MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-05 14:21:02

Abstract: Existing pretrained text-to-video (T2V) models have demonstrated impressive abilities in generating realistic videos with basic motion or camera movement. However, these models exhibit significant limitations when generating intricate, human-centric motions. Current efforts primarily focus on fine-tuning models on a small set of videos containing a specific motion. They often fail to effectively decouple motion and the appearance in the limited reference videos, thereby weakening the modeling capability of motion patterns. To this end, we propose MoTrans, a customized motion transfer method enabling video generation of similar motion in new context. Specifically, we introduce a multimodal large language model (MLLM)-based recaptioner to expand the initial prompt to focus more on appearance and an appearance injection module to adapt appearance prior from video frames to the motion modeling process. These complementary multimodal representations from recaptioned prompt and video frames promote the modeling of appearance and facilitate the decoupling of appearance and motion.  In addition, we devise a motion-specific embedding for further enhancing the modeling of the specific motion. Experimental results demonstrate that our method effectively learns specific motion pattern from singular or multiple reference videos, performing favorably against existing methods in customized video generation.

Main paper: 
 introduction: 
 Diffusion-based video generation has achieved significant breakthroughs [3, 11, 17, 33] , facilitating the production of high-quality, imaginative videos. While foundation Text-to-Video (T2V) models can generate diverse videos from provided text, tailoring them to generate specific motion could more closely align with user's preferences. Akin to subjects customization in Text-to-Image (T2I) tasks [9, 26, 36] , human-centric motions in videos can also be customized and transferred to various subjects, which holds significant practical benefits for animation and film production [45, 46] .Existing pretrained T2V models [5, 42] often struggle to generate intricate, human-centric motions like golf swings and skateboarding, which involve multiple continuous sub-motions. One potential reason is that these foundation models are predominantly trained on highly diverse datasets [1] sourced from the internet, which may suffer from imbalanced data distribution. Consequently, the models might encounter certain motions infrequently, leading to inadequate training for those motions. To better generate particular motions, these pretrained T2V models [5, 42] require fine-tuning on a small set of videos containing the desired motion pattern. However, fine-tuning the model directly without any additional constraints is prone to leading to an undesirable coupling between the motion and the appearance in the limited reference videos and weakening the modeling capability of motion patterns.Several works [34, 45, 53, 55] have been proposed to address the issue outlined above. These approaches predominantly leverage a dual-branch architecture, with one branch dedicated to capturing single-frame spatial information and the other to inter-frame temporal dynamics. Additionally, they also introduce decoupling mechanisms, such as embedding appearance priors to guide the focus of temporal layers on motion [45] or adjusting latent codes to minimize the negative impact of appearance [55] . Despite their efforts to separate appearance from motion, these approaches exhibit insufficient learning of motion patterns, resulting in videos with diminished motion magnitudes and a deviation from the motion observed in reference videos to some extent.To this end, we introduce MoTrans, a customized Motion Transfer method, which mainly focuses on modeling the motion patterns in reference videos while avoiding overfitting to its appearance. Specifically, we adopt a two-stage training strategy, with an appearance learning stage and a motion learning stage respectively modeling appearance and motion. To alleviate the coupling issue between appearance and motion, we undertake comprehensive explorations in both stages. 1) During the appearance learning stage, a multimodal large language model (MLLM) is adopted as the recaptioner to expand the original textual descriptions of the reference video. 2) During the motion learning stage, before adapting the temporal module to a specific motion, representations of video frame are pre-injected to compel this module to capture motion dynamics. The complementary multimodal information from expanded prompt and video frame promotes the modeling of appearance and decomposition of appearance and motion. Notably, it has been observed that motions in videos are primarily driven by verbs within the prompt. Inspired by this observation, we employ a residual embedding to enhance the token embeddings of the verbs corresponding to motion, thereby capturing the specified motion patterns in the reference video. Extensive experimental results demonstrate that our method effectively mitigates the issue of overfitting to appearance and produces high-quality motion, performing favorably against other state-of-the-art methods. The main contributions of our work can be summarized as follows:• We propose MoTrans, a customized video generation method enabling motion pattern transfer from single or multiple reference videos to various subjects. • By introducing an MLLM-based recaptioner and appearance prior injector, we leverage complementary text and image multimodal information to model the appearance information, effectively mitigating the issue of coupling between motion and the limited appearance. • We introduce the motion-specific embedding, which is integrated with temporal modules to collaboratively represent specific motion within reference videos. • Experimental results demonstrate that our method surpasses other motion customization methods, enabling any motion customization contextualized in different scenes. 
related work 2.1 text-to-video generation: 
 Diffusion models are catalyzing rapid advancements in image generation tasks [8, 31, 35] and have spawned numerous valuable applications [12, 28, 43, 49, 51, 54] . This success has garnered significant interest in extending these capabilities to video generation [16, 32, 44, 46] . Early efforts in T2V domain [19, 21, 37, 44] primarily focus on cascading video frame interpolation and super-resolution models to generate high-resolution videos, which seems to be complex and cumbersome. In contrast, ModelScopeT2V [42] represents a significant shift by incorporating spatio-temporal blocks atop stable diffusion [35] to model motion more effectively. Building on this, ZeroScope [5] expands the training data and utilizes watermarkfree data for fine-tuning, enabling the generation of videos with improved resolution and enhanced quality.Recently, a new wave of high-quality T2V models [14, 17, 52 , 56] has achieved impressive progress. Emu Video [14] generates highquality videos from natural language descriptions by dividing the video generation process into two steps: initially generating a textconditioned image, followed by creating videos conditioned on both the text and the generated image. VideoCrafter2 [7] utilizes lowquality videos to ensure motion consistency while employing highquality images to enhance video quality and conceptual composition ability. Commercial models such as Pika [33] and Gen-2 [11] also exhibit exceptionally strong generative capabilities. Moreover, OpenAI's recent launch of the Sora model [3] , capable of generating high-quality videos up to 60 seconds in length, marks a significant milestone in video generation. Although the above foundation T2V models can generate appealing videos, they face challenges in precisely controlling the generated motion.Existing T2V models [5, 7, 42] excel at generating simple motions or camera movements, struggling to produce specific human-centric motions that align with user preferences. To this end, some models have been introduced to synthesize specific motion pattern and transfer it to diverse subjects. For customized motion transfer [6, 22, 48] , some methods employ additional pose maps [4] or dense poses [15] as guidance and require substantial amounts of training data. During the inference stage, it is possible to animate static characters by merely providing initial noise, a reference image, and a set of pose sequences as additional guiding conditions. These approaches allows to produce animations without any need for fine-tuning once they are adequately trained. However, they primarily focus on human-to-human motion transfer and often struggle to transfer motion to subjects that significantly deviate from the human domain, such as animals.We aim to learn specific motion patterns rather than precisely replicate every frame's action. This task [29, 34, 45, 47, 53, 55] requires only a minimal amount of training data sharing the same motion concept. Similar to the T2I method DreamBooth [36] , these approaches necessitate individual training for each type of motion. Since the generation process does not require additional control conditions such as pose, the resulting motions are more flexible and do not need to follow each frame of the reference video strictly. MotionDirector [55] learns both camera movement and motion, adopting a dual-path way framework to separately learn appearance and motion. During motion learning, the spatial layers trained for appearance learning are frozen to inhibit the temporal layers from learning appearance. DreamVideo [45] introduces structurally simple identity and motion adapters to learn appearance and motion, respectively. To decouple spatial and temporal information, it proposes injecting appearance information into the motion adapter, forcing the temporal layers to learn motion.Although some methods [34, 45, 53, 55] realize the issue of appearance-motion coupling, they are still prone to synthesizing videos overfitting to the appearances of training data to a certain extent, thereby exhibiting insufficient learning of motion patterns. Furthermore, some methods [47, 53] learn motions that are easier to model. In this paper, we are more concerned with challenging actions with larger ranges of motion, such as sports actions. 
method 3.1 overview: 
 Given a single video or multiple videos with similar motions, the goal of our task is to learn the specific motion or the common motion pattern contained in reference videos. Subsequently, the learned motion can be adapted to new subjects contextualized in different scenes. As illustrated in Fig. 2 , the overall training pipeline is divided into the appearance learning stage and the motion learning stage. In the appearance learning stage, we employ an MLLM-based recaptioner to expand the initial prompt of the reference videos. It could promote the modeling capabilities of the spatial attention modules for appearance information. At this stage, we only train spatial low-rank adaptions (LoRAs) and share the weights with the second stage to fit the appearance of the corresponding reference video. As shown in Fig. 4 (a) , to preserve the textual alignment capability of the pretrained T2V model, we freeze the parameters of the cross-attention layer and only inject LoRAs into the self-attention and feed-forward layers (FFN). In the motion learning stage, before adapting temporal modules to a specific motion, image embeddings are injected to introduce appearance priors, thereby forcing the temporal LoRAs to focus on motion modeling. Additionally, we employ a multilayer perceptron (MLP) to augment the token embeddings corresponding to verbs, which is jointly trained with temporal LoRAs to capture specific motion pattern. For temporal A person is playing the guitar, wearing a cap and striped shirt, seated by a white door.A person is playing the guitar. During the inference stage, we integrate the temporal LoRAs and the residual embedding into pretrained video generation models to transfer the specific motion to new subjects.User Recaption Source ImageThe primary objective of our task is to learn motion patterns specified by several reference videos. Due to the inherent characteristics of the diffusion model's training loss, the leakage of some appearance information is inevitable in the motion learning stage. To seperate motion from appearance to a certain extent during the motion learning process, we propose an MLLM-based recaptioner and an appearance injector. In this manner, the complementary multimodal appearance information provided by text and video facilitates the decoupling of appearance and motion information. MLLM-based recaptioner. MLLMs like LLaVA 1.5 [24] or GPT4 [30] have robust in-context reasoning and language understanding capabilities, which can be used for image recaptioning [2, 50] . As illustrated in Fig. 3 , let V = {𝑓 𝑖 |𝑖 = 1, ..., 𝑙 } denote the reference video with 𝑙 frames, given a carefully crafted task instruction, the recaptioner can perform text-to-text translation and expand the base prompt c 𝑏 based on a random frame 𝑓 𝑖 . In this manner, the recaptioned prompt c 𝑟 can comprehensively describe the appearance information contained within the video frames. Through training, the spatial attention module will adapt to the appearance information of the reference video and remains frozen in the subsequent During the appearance learning stage, we adopt a frozen MLLM and only spatial LoRAs need to be trained. The optimization process of this stage is defined as follows:EQUATIONHere, a VQ-VAE [23] initially compresses frame 𝑓 𝑖 into a latent representation z i 0 ∈ R 𝑏 ×1×ℎ×𝑤×𝑐 , where 𝑏, ℎ, 𝑤, 𝑐 represent batch size, height, width, and channel count, respectively. z i t is the noised latent code at timestep 𝑡 ∼ U (0,𝑇 ). 𝜏 𝜃 (•) denotes the pretrained OpenCLIP ViT-H/14 [10] text encoder. Meanwhile, the network 𝜖 𝜃 (•) is trained to predict the noises added at each timestep. Appearance injector. In addition to leveraging recaptioned prompts to enhance the modeling of appearance, integrating embedding information from video frames themselves can also yield significant benefits. These two modalities collaboratively contribute to the effective decomposition of motion from its appearance. Drawing inspiration from [45] , we inject appearance information in the second stage to diminish its impact on motion learning. As shown in Fig. 2 (b ), an image encoder 𝜓 is utilized to obtain embeddings of all video frames, we randomly select an image embedding 𝜓 (f i ) ∈ R 1×𝑑 from the input video, where 𝑑 denotes the dimension of image embedding. Then the appearance information is injected before the temporal transformers, as demonstrated in Fig. 4 (b) .Formally, for each UNet block 𝑙, the spatial transformer produces the hidden states ℎ 𝑙 𝑠 ∈ R (𝑏 ×ℎ×𝑤 ) ×𝑓 ×𝑐 . We employ a linear projection to broadcast the input embeddings across all frames and spatial positions, which are then summed with the hidden states ℎ 𝑙 𝑠 before being fed into the temporal transformer. In this way, the appearance representations from the visual modal are pre-injected. The entire process can be formulated as follows:EQUATIONwhere 𝑊 𝑝 represents the weights of the linear projection layer, with its output dimension adapting to variations in the dimensions of the UNet hidden states. And ⊙ denotes the broadcast operator.An intuitive observation is that motion patterns in videos generally align with verbs in a text prompt. Hence, we posit that emphasizing verbs could potentially encourage the model to enhance its modeling of motion in the reference videos. Motion Enhancer. A heuristic strategy for enhancing the modeling of motion involves leveraging visual appearance information to enrich the textual embedding representation of motion concept. This is achieved by learning a residual motion-specific embedding on top of the base text embedding. The base embedding can be considered a coarse embedding corresponding to a general motion category, whereas the residual embedding is tailored to capture the specific motion within given reference videos. Specifically, we employ a pretrained text encoder 𝜏 𝜃 to extract text embeddings from a sequence of words S = {𝑠 1 , ..., 𝑠 𝑁 }. To locate the position i of the verb 𝑠 𝑖 in the text prompt, we use Spacy for part-of-speech tagging and syntactic dependency analysis. Following this, the base motion embeddings corresponding to the motion concept are then selected. As shown in Fig. 2 (b), video frames are initially processed by an image encoder to generate frame-wise embeddings. To capture temporal interactions, these embeddings are aggregated through a mean pooling operation, resulting in a unified video embedding. This video embedding is concatenated with the base motion embedding and further processed by an MLP. The MLP comprises two linear layers separated by a Gaussian Error Linear Unit (GELU) [18] activation function. Subsequently, we compute a residual embedding, which is added to the base motion embedding to form an enhanced motion-specific representation. Mathematically, let 𝐸 𝑏 and 𝐸 𝑟 represent the base embedding and learnable residual embedding, respectively, the operation can be expressed as follows:EQUATIONEQUATIONHere, [•] refers to the concatenation operation, and 𝑊 1 and 𝑊 2 denote the weights of two Linear layers in MLP. The GELU function is represented by 𝜎 𝐺𝐸𝐿𝑈 . This motion-specific embedding is integrated with the text embeddings of other words in the prompt to serve as the new condition for training temporal transformers.To prevent the learned residuals from becoming excessively large, akin to the strategy in [13] , we introduce an L2 regularization term as a constraint as:L 𝑟𝑒𝑔 = ||𝐸 𝑟 || 2 2(5) Similar to the appearance learning stage, the loss function in the motion learning stage calculates the Mean Squared Error (MSE) loss between the predicted noise of the diffusion model and the ground truth noise, except that the frame dimension is no longer 1. Therefore, the final loss function for this stage is defined as:EQUATIONFor motion learning, the loss function is the combination of temporal loss and a constraint term as follows,EQUATIONwhere 𝜆 controls the relative weight of the regularization term. 
experiments 4.1 experimental setup: 
 Dataset. We collect a dataset that includes 12 distinct motion patterns, sourced from the Internet, the UCF101 dataset [40] , the UCF Sports Action dataset [39] , and NTU RGB+D 120 [25] . Each motion pattern is represented by approximately 4-10 training videos. The dataset consists of various sports motions, such as weightlifting and golf swing, alongside large-scale limb movements like waving hands and drinking water. For evaluation, we employ six base prompt templates that involve variations in subject, motion, and context. An example template is "A {cat} is {motion} {in the living room}", with placeholders indicating dynamic elements. Videos corresponding to each motion are generated based on these six prompt categories. More details of our dataset are available in the supplementary materials. Implementation details. We employ ZeroScope as the base T2V model, which is trained with the AdamW [27] optimizer across approximately 600 steps with a learning rate of 5𝑒 -4. For the spatial and temporal transformers, we specifically fine-tune LoRAs instead of all parameters, with the LoRA rank set to 32. The image encoder used for appearance injection is derived from OpenCLIP ViT-H/14, which is also used to calculate CLIP-based metrics. The regularization loss coefficient for normalizing the verb's residual embedding is 1𝑒 -4. During inference, we employ DDIM [38] sampler with 30-step sampling and classifier-free guidance scale [20] of 12. We generate 24-frame videos at 8 fps with a resolution of 576 × 320. All experiments are conducted on a single NVIDIA A100 GPU.Comparison methods. To investigate the generative capabilities of existing T2V models, we compare our approach with prominent open-source models, including ZeroScope [5] and VideoCrafter2 [7] . Additionally, we explore the effectiveness of directly fine-tuning ZeroScope on a small set of videos containing a specific motion. It is noteworthy that fine-tuning is not applied to the entire diffusion model but specifically targets the LoRAs within the temporal transformers. Our proposed method is adaptable for both single and multiple video customization scenarios. Consequently, we benchmark our approach against open-source methods specialized for one-shot customization, such as Tune-a-Video [46] , and for few-shot customization, like LAMP [47] . Further comparisons are conducted with MotionDirector, which serves as our baseline.Evaluation metrics. The performance of the comparison methods is evaluated by four metrics. CLIP Textual Alignment (CLIP-T) is employed to assess the correspondence between the synthesized video and the provided prompt, while Temporal Consistency (Tem-pCons) measures frame consistency within videos. Due to issues of appearance overfitting observed in some comparison methods, we introduce CLIP Entity Alignment (CLIP-E) metric, which is similar to Textual Alignment but foucses on prompts containing only entities, such as "a panda". This metric evaluates whether the synthesized video accurately generates the entity specified by the new prompt. To the best of our knowledge, there exists no metric capable of measuring the congruence between motion patterns in the synthesized videos and those in the reference videos. Therefore, we propose Motion Fidelity (MoFid), which is based on the video understanding model VideoMAE [41] . Specifically, a video 𝑣 𝑖 𝑚 is randomly selected from the training videos, and a pretrained Video-MAE 𝑓 (•) is used to obtain the embeddings for both the selected video 𝑣 𝑖 𝑚 and the synthesized video v𝑘 . Formally, motion fidelity is calculated as follows:EQUATIONwhere M denotes the set of motions, | v𝑚 | is the number of videos with motion 𝑚 in the generated videos, and 𝑐𝑜𝑠 (•) refers to cosine similarity function. Further details on motion fidelity are available in the supplementary material.Qualitative Evaluation. To validate the motion customization capabilities of our method, we conduct a comparative analysis with several representative open-source methods tailored for one-shot and few-shot motion customization. As depicted in Fig. 5 , direct inference using pretrained T2V models ZeroScope and VideoCrafter fails to synthesize specific motion patterns due to the lack of finetuning on specified videos. Additionally, the synthesized videos exhibit notably small motion amplitudes, suggesting that pretrained T2V models struggle to generate complex, human-centric motion.In particular, these models face significant challenges in generating specific motions without targeted training on specific videos. Furthermore, unconstrained fine-tuning of Zeroscope leads to an undesirable coupling between appearance and motion, and the motions in the generated videos do not sufficiently resemble those in the reference videos, with notably small motion amplitudes. Tune-A-Video, which targets single-video customization and is based on the T2I model, suffers from poor inter-frame smoothness and severe appearance overfitting. Similarly, the few-shot motion customization method LAMP, also leveraging a T2I model, exhibits very poor temporal consistency and heavily relies on the quality of the initial frame. Compared to other methods, LAMP requires more reference videos and training iterations to achieve relatively better results. MotionDirector also encounters challenges with appearance overfitting, often generating unrealistic scenarios such as a panda on a skateboard dressed in human attire. Moreover, it exhibits insufficient modeling of motion patterns, resulting in videos with diminished motion magnitudes and deviations from the observed motion in reference videos.Our method, however, demonstrates superior ability to accurately capture motion patterns in both one-shot and few-shot motion customization scenarios. Additionally, one-shot methods sometimes fail to discern whether to learn camera movement or foreground motion. In contrast, few-shot methods can leverage inductive biases derived from multiple videos, better capturing common motion patterns. This allows the temporal transformer to focus on foreground action rather than camera movements. Quantitative Evaluation. As illustrated in Table 1 , when only a single reference video is provided, both Tune-a-Video and the fine-tuned Zeroscope exhibit higher motion fidelity but lower entity alignment. This is primarily attributed to the severe appearance overfitting, which leads to pronounced similarities in both appearance and motion to the reference video. Consequently, these methods fail to synthesize the new subject specified in the prompt, which is also demonstrated in Fig. 5 . When multiple reference videos are provided, our approach outperforms other methods across all evaluated metrics. Notably, it achieves high levels of text alignment and motion fidelity, showcasing our method's capability to effectively learn motion patterns from reference videos while avoiding overfitting to the appearance information. User study. Automatic metrics like CLIP-T have limitations in fully reflecting human preferences, hence, we conduct user studies to further validate our method. We collect 1536 sets of answers from 32 participants, with each completing a questionnaire containing 48 sets of questions. Participants are asked to pick the best video through answering the following questions: (1) which video better aligns with the textual description? (2) which video is smoother, and with fewer flickering? (3) which video's motion is more similar to that in the reference video without resembling its appearance? Considering the significantly inferior performance of the T2I-based model LAMP, our comparison primarily focused on MoTrans versus the other methods. The results, shown in Fig. 6 , reveal that our method consistently outperforms the others across all metrics, aligning more closely with human intuition.We conduct ablation studies to demonstrate the efficacy of the key modules introduced in this paper. Specifically, the MLLM-based recaptioner and the appearance injector leverage the prior knowledge of multimodal sources, i.e., textual and visual modalities, to address the challenge of coupling between appearance and motion. As illustrated in Table 2 and Fig. 7 (rows 1 and 2), the absence of either the MLLM-based recaptioner or the appearance injector leads to a performance drop in both CLIP-T and CLIP-E, alongside high motion fidelity. This suggests a severe overfitting to both appearance and motion. Additionally, without the motion enhancer, the model struggles to synthesize the specific motion depicted in the reference videos, but with the introduction of the first two modules, it can synthesize the specified subject. In comparison, our method effectively mitigates appearance overfitting while ensuring motion fidelity as much as possible. Each module we propose significantly contributes to the improvement of the final generation results.Video customization with both subject and motion. Benefitting from the two-stage training strategy of our approach, appearance and motion can be learned separately through the spatial and temporal transformers within a UNet. As depicted in Fig. 8 , we simultaneously customize the subject depicted in an image set and the motion specified by a video set. The customization results demonstrate that our method does not suffer from appearance overfitting to the training data and can successfully enable a specific animal or inanimate object to perform a human-centric motion. 
conclusion: 
 We propose MoTrans, a customized motion transfer method that effectively transfers a specific motion pattern from reference videos to diverse subjects. By integrating multimodal appearance priors, encompassing both visual and textual modalities, our approach mitigates the issue of coupling between motion patterns in synthesized videos and the limited appearance contained in reference videos. Additionally, our method employs dedicated residual embeddings to accurately represent the specific motion pattern inherent in the reference videos. Compared with existing methods, our method demonstrates superior capabilities in customizing motion and decoupling appearance, and it also supports the simultaneous customization of subjects and motions. Although our method can synthesize high-quality motion, it is currently optimized for short video clips of 2-3 seconds and faces challenges in generating longer sequences. Moreover, while our method currently supports the customization of motion for a single subject, extending this capability to multiple subjects performing the same motion remains a challenge. Future work will aim to address these limitations and expand the applicability of our method to more complex and practical scenarios.  
motrans: customized motion transfer with text-driven video diffusion models: 
 Anonymous AuthorsExisting pretrained text-to-video (T2V) models have demonstrated impressive abilities in generating realistic videos with basic motion or camera movement. However, these models exhibit significant limitations when generating intricate, human-centric motions. Current efforts primarily focus on fine-tuning models on a small set of videos containing a specific motion. They often fail to effectively decouple motion and the appearance in the limited reference videos, thereby weakening the modeling capability of motion patterns. To this end, we propose MoTrans, a customized motion transfer method enabling video generation of similar motion in new context. Specifically, we introduce a multimodal large language model (MLLM)-based recaptioner to expand the initial prompt to focus more on appearance and an appearance injection module to adapt appearance prior from video frames to the motion modeling process. These complementary multimodal representations 
 
 Appendix:  


Decision: The paper introduces MoTrans, a customized motion transfer method designed for text-driven video diffusion, particularly focusing on capturing and replicating specific motion patterns in new contexts. This method employs a multimodal large language model (MLLM)-based recaptioner to enhance initial prompts with a focus on appearance, and introduces an appearance injection module to incorporate appearance priors from video frames into the motion modeling process.

Strengths:

+ MoTrans effectively addresses the text-to-video generation challenge, leveraging both appearance and motion data to enhance video output.

+ The paper is well-structured and clearly written, making technical details accessible and easy to follow.

+ Experimental validation on datasets like UCF101 and NTU RGB+D 120 demonstrates the method's effectiveness in learning specific motion patterns from reference videos.

Limitations:

+ The method's ability to handle scenarios with minimal temporal data, such as reference videos containing only one frame, is not explored, raising questions about its versatility in appearance modeling. 

+ Some metrics such as TempCons and MoFid in the experimental results do not outperform existing methods, and the reasons behind these outcomes are not thoroughly explained. 

+ The examples provided in the study are limited to scenarios containing a single object, which may not adequately demonstrate the method's effectiveness in more complex, multi-object settings. 

+ The computational cost of generating videos is not detailed, which is crucial for evaluating the practical applicability of MoTrans in real-world scenarios.

Conclusion:

MoTrans introduces innovative techniques for motion and appearance integration in video generation, showing promising results. However, further exploration into its limitations and broader applicability is necessary to fully ascertain its potential. The AC suggests a borderline accept, indicating that while it presents significant advancements, additional clarifications and expansions on its capabilities and performance would strengthen its contribution to the field. The authors might want to carefully address the concerns of Reviewer tL5r.

Reviews: review: paper_summary: paper_summary: This paper focused on the text-driven video diffusion task and proposed a customized motion transfer method named MoTrans, which enables video generation of similar motion in new context.  A multimodal large language model (MLLM)-based recaptioner in applied to expand the initial prompt to focus more on appearance and an appearance injection module is introduced to adapt appearance prior from video frames to the motion modeling process. A motion-specific embedding is proposed to further enhance the modeling of the specific motion. Experimental results on UCF101, UCF Sports Action, and NTU RGB+D 120 datasets validate the effectiveness of the MoTrans in learning specific motion pattern from singular or multiple reference videos.
strength_weakness: strength_weakness: +) Clear motivations and contributions.

+) Good writing and the technical details are easy-to-follow.

+) The promising video generation results.
limitations: limitations: -) What about using the reference videos that contain only one frame? In this setting, the temporal coherece is missing and the apperance information dominates. This experiment is important for comprehensvely evaludate the ability of apperance modeling of the MoTrans.

-) Some results in Table 1-2 are not the best, for example, the TempCons and MoFid, more insights behind this phenomenon are needed.

-) Since all the examples contain single object, the videos containing multiple objects are required to be given. 

-) What is the computational cost (e.g., several seconds or minutes) for generating one video given the reference video?
paper_summary: paper_summary: Given reference videos and a text prompt, the research aims at customized motion generation. This paper proposed a two-stage method to decouple motion and appearance. In the appearance learning stage, they proposed an MLLM-based recaptioner and an appearance injector. In the motion learning stage, they proposed a residual embedding to enhance the effect of verbs. Finally, the experiments on the collected dataset show the performance of the proposed method.
strength_weakness: strength_weakness: 1. This paper explored the customized motion generation under decoupling motion and appearance.

2. The idea to use LLMs to generate the more detailed caption of the source image, is useful to improve the ability of learning appearance.

3. The idea of attention to verbs of the text is also useful in text learning.

4. The paper is easy to follow.
limitations: limitations: 1. Lack of details. Though the paper introduces the two stages of the proposed turn, the details of each stage are still vague. For example, In L446, do you mean the method is a latent diffusion based method? Does equation (1) refer to spatial loss? What is spatial LoRA and temporal LoRA? There are no specific definitions for them.

2. Lack of novelty. The proposed method is similar to MotionDirector. For example, the spatial/temporal LoRAs and spatial/temporal loss are the same with MotionDirector. The difference is only the injection of text-image multi-modal information and video information. Hence, the novelty of the proposed method is limited. It can also be proved by the experimental results, which show little improvement compared with MotionDirector.

3. Correctness of the proposed MoFid metric. As mentioned in L630, MoFid measures the motion patterns. But in Table 1, when the proposed method is low on MoFid, they argue the reasons from the appearance. In Table 2, when w/o appearance injector yields a high score on MoFid, they argue without appearance information they got more consistent motion patterns. A correct metric for evaluating the motion pattern should be robust, and not be so influenced by other factors.

4. The experimental results are on the collected dataset. Could you give the results on one public dataset, not a mixed dataset?
paper_summary: paper_summary: The paper proposes a motion transfer method for video generation which intends to decouple the appearance and motion learning. In the appearance learning branch, the method proposes to use a MLLM-based recaptioner to extend the initial prompts to comprehensive descriptions on the appearance to learn spatial LoRAs. In addition, the method learns a linear layer from the randomly selected image embeddings extracted from video frames as an appearance injector in learning the temporal patterns. In the motion learn branch, the method proposes to use a pretrained text encoders to focus on \emph{verbs} in the text prompt. The motion-specific embedding is combined with the text embeddings to control the video generation within the off-the-shelf ZeroScope T2V model. The experiments show that the proposed method can effectively transfer human motions to animals, such as bear, panda and cat.
strength_weakness: strength_weakness: The proposed motion transfer method is well motivated to extend motion transfer beyond human for general video generation. 

The particular way to decouple appearance and motion, such as the recpationer and spatial LoRA as well as the appearance injector, is reasonable and somewhat novel.

Overall, the paper is well written and sufficient experiments.
limitations: limitations: The motion learning branch employs a simple way to learn motion-specific embeddings, e.g., a mean pooling of the embeddings of verbs over frames, which needs more justification how this scheme can learn motion-specific embeddings.
 
Why a randomly select image embedding from the input video can diminish the impact of appearance to motion learning when being infected before the temporal transformers? Any better scheme to select the appearance injectors? 

Please discuss the limitation of the proposed method transferring motion to different kinds of objects, e.g., 4 limbs required to transfer human motion?

Overall, the paper studied a critical problem in controllable video generation and push a step forward, though there is some gap between the proposed approach and the claim that needs to be elaborated.  Nevertheless, the work deserves to share with the community.
paper_summary: paper_summary: In this paper, the authors propose a customized motion transfer method, MoTrans, to effectively decouple appearance and motion. MoTrans is divided into two learning stages: the appearance learning phase and the motion learning phase. In the appearance learning stage, the Multimodal Large Language Model (MLLM) is employed as the recaptioner to extend the original textual descriptions of the reference video; in the motion learning stage, the temporal module is used to capture motion information from the reference video. Experimental results demonstrate that MoTrans can effectively learn specific motion patterns from singular or multiple reference videos.
strength_weakness: strength_weakness: 1. This paper is structured and easy to understand.
2. In the motor learning stage, the temporal module is injected with appearance information to allow the temporal module to learn effective motor information.
3. The paper is complete with comparative experiments to demonstrate its effectiveness by comparing a number of different methods.
limitations: limitations: 1. When there is a large change in the background, does the motion learning stage learn background motion features that affect the motion of the foreground?

2. In the appearance learning stage, Spatial Transformers receive prompts that have gone through the MLLM-based recaptioner. But in motion learning stage, why prompt does not need to go through the MLLM-based recaptioner? Please explain it.

3. When generating videos on specific subjects. Is it also possible to achieve good results when there is a large difference in the foreground size of the reference video?