MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation
Authors: Chenhui Zhu
Yilu Wu
Shuai Wang
Gangshan Wu
Limin Wang
Venue: NeurIPS 2025 poster ()
Keywords: Image-to-Video Generation
Motion Transfer
Retrieval-Augmented Generation
Primary area: applications
TL;DR: We introduce MotionRAG, a framework that improves video generation by retrieving motion from real videos and injecting it into diffusion models to create more realistic movement.

Abstract: Image-to-video generation has made remarkable progress with the advancements in diffusion models, yet generating videos with realistic motion remains highly challenging. This difficulty arises from the complexity of accurately modeling motion, which involves capturing physical constraints, object interactions, and domain-specific dynamics that are not easily generalized across diverse scenarios. To address this, we propose MotionRAG, a retrieval-augmented framework that enhances motion realism by adapting motion priors from relevant reference videos through Context-Aware Motion Adaptation (CAMA). The key technical innovations include: (i) a retrieval-based pipeline extracting high-level motion features using video encoder and specialized resamplers to distill semantic motion representations; (ii) an in-context learning approach for motion adaptation implemented through a causal transformer architecture; (iii) an attention-based motion injection adapter that seamlessly integrates transferred motion features into pretrained video diffusion models. Extensive experiments demonstrate that our method achieves significant improvements across multiple domains and various base models, all with negligible computational overhead during inference. Furthermore, our modular design enables zero-shot generalization to new domains by simply updating the retrieval database without retraining any components. This research enhances the core capability of video generation systems by enabling the effective retrieval and transfer of motion priors, facilitating the synthesis of realistic motion dynamics.

MotionRAG: Motion Retrieval-A Aggmented Image-to-Video Generation
Chenhui Zhu!
Yilu Wu
Shuai Wang'
Gangshan Wu Limin Wang ..2
State Key Laboratory for Novel Software Technology, Nanjing University
"Shangaai AI Laboratory
Abstract
Image-to-video generation has made remarkable progress with the advancements in diffusion models, yet generating videos with realistic motion remains highly challenging. This difficulty arises from complexity accurately modeling motion, which involves capturing physical constraints, object interactions, and domain-specific dynamics that are not easily generalized across diverse scenarios. framework that
To address this, propose MotionRAG, enhances motion realism by adapting motion priors from relevant reference videos through Context-Aware Motion Adaptation (CAMA), The key technical innova- tions include: (i) retrieval- -base pipeline extracting high-level motion features using video encoder and specialized resamplers distill semantic motion repre- sentations; (ii) an in-context learning approach for motion adaptation implemented through causal transformer architecture; (iii) attention-based motion injection adapter that seamlessly integrates transferred motion features into pretrained video diffusion models. Extensive experiments demonstrate that our method achieves sig- nificant improvements across multiple domains and various base models, all with negligible computational overhead during inference. Furthermore, our modular design enables zero-shot generalization to new domains by simply updating the retrieval database without retraining any components. This research enhances the core capability of video generation systems by enabling the effective retrieval and transfer of motion priors, facilitating the synthesis of realistic motion dynamics.
Introduction
Recent advancements generative models have revolutionized image-to-video synthesis, enabling the creation of short video clips from static images with unprecedented visual quality 2 4
These models, primarily based on diffusion architectures [6. excel at preserving the appearance of input images while introducing temporal dynamics. However, despite their impressive visual fidelity, critical challenge persists: generating physically plausible and semantically coherent motion remains significant and unresolved issue 7
The core challenge stems from the inherent complexity of motion. Unlike appearance, which can be directly inferred from single frame, motion involves capturing physical constraints, object interactions, and -mmmain--pecficc dynamics, making its modeling significantly more difficult. Ex- isting methods typically rely on end-to-end training on large video datasets [5] L. where motion knowledge develops naturally through exposure various examples. While this approach yields im- provements, it struggles with compositional generalization the ability to combine familiar elements novel ways, such as "an astronaut riding a horse on the moon."
Cooresppnniing author: 1mwang@nju. edu. cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ref. Videos
Database
astronaut riding
Motion
Gen. Videos
horse
MotionRAG
Illustration of cross-domain motion transfer. Our approach retrieves videos of people riding horses and transfers their motion priors to generate an astronaut riding horse on the moon. while preserving the appearance characteristics of the input image.
Figure 1:
A key insight driving our research is that motion can be inherently transferred across do- mains [T]] M. For example, the motion of person riding a horse can be applied to generate an astronaut riding horse, despite significant differences in visual appearance, as shown in Figure [ This transferability due to the physical and kinematic constraints governing motion, which remain consistent even when surface appearances change 12 However, effectively extracting and trans- ferring this motion challenging, as motion features are often mixed with appearance information in video representations [I3]. Current image-to-viden models primarily rely on text descriptions to infer motion dynamics, but textual descriptions inherently lack the precise temporal coordination and kinematic details that actual video examples provide. This fundamental limitation motivates retrieval- aaurenel-uugmented approach, which leverages real video motion patterns to guide generation with richer and more physically plausible dynamics.
To tackle these challenges, we introduce a novel retrieval- auggeettd framework MotionRAG for image-to-vides generation that enhances motion realism through explicit cross-domain transfer. Our approach comprises three key components: (i) text-based retrieval system that identifies videos with relevant motion, (ii) a context-aware motion adaptation (CAMA) module that adapts the extracted motion information to the target image, and (iii) a motion-guided generation process that synthesizes the final video while preserving appearance fidelity. The core technical innovation of our work lies in our Context-Aware Motion Adaptation (CAMA) module, which formulates motion transfer in-context learning problem [14 Drawing inspiration from recent advances in LLM, our transformer-base
architecture processes a sequence of retrieved examples to infer appropriate motion priors for the target image. By arranging examples in reverse similarity order and employing causal attention, the model effectively learns to adapt motion features across visual domains without requiring domain-spednt sppecifc fine-tuning.
Through our experiments, we demonstrate that our approach significantly improves motion realism across multiple domains and base models. Our method achieves consistent quality improvements when integrated with various state-of-the-art image-to- to-video models. Quantitative and qualitative evaluations confirm that videos generated with our method exhibit more natural and physically plausible motion compared to existing approaches.
Our contributions can be summarized as follows: (i) We introduce MotionRAG, a retrieval-augmentte framework that extracts and transfers high-level motion representations from semantically relevant videos to guide image animation. (ii) We propose CAMA, novel in-context learning approach for motion transfer that adapts motion patterns across visual domains without requiring per-instance fine-tuning. (iii) Extensive experiments demonstrate our method significantly improves motion quality across multiple baseline models with negligible computational overhead, enhancing even state-of-the--rt models by substantial margins.
2 Related Work
Retrieval-Augmentee Generation. Retrieval-Augmente Generation (RAG) is a powerful approach that improves pretrained models by retrieving relevant information from external sources

---

Ref. Video
RAG
Gen. Video
Database
Ref. Video 2
Ref. Video
Motion Context
Image-to-Vine -odde- Generator
Prompt
Transformer
Target
Input Image
Encoders
Motion
Figure 2: Our MotionRAG framework. Text prompts retrieve relevant videos from a database. Motion information from these references are adapted to the input image via Motion Context Transformer, then injected into an image-to-video generator to produce the final output.
during generation. Originally designed for natural language processing, RAG allows models to access domain-speciin knowledge on demand, leading to more accurate and relevant outputs[[7]. Inspired by its success in language tasks, similar methods have been applied to visual domains. For example, ImageRAG 18 retrieves related images to improve image generation quality. In video generation, methods like search-T2V [19] follow similar idea by using a video database motion prior. This turns text-to-video (T2V) generation into a search-based process, boosting performance without requiring large-scale training. However, search-T2V only supports text-to-video generation and cannot use a reference image as input, limiting its use in image-to-video tasks. It also requires expensive fine-tuning for each generation, which takes several minutes. In contrast, method uses in-context learning to adapt motion patterns quickly and efficiently without extra training.
Video Motion Customization. Video customization aims adapt pre-trained video generation
models to personalized concepts, often by fine-tuning on reference videos 220, 21 22 23 For example, MotionDirector [20) used dual-ppth design with a temporal objective, VMC [21 employed
inter-frame residuals to distill motion, and Dream Video [22] applied adapters to separate motion and appearance. Customize-A A- A-Video 23 enhanced appearance features and used temporal LoRA [24 for motion learning. Unlike these methods, which require model fine-tuning for each video, our retrieval- based approach uses in-context learning adapt motion without changing model parameters. This enables efficient generalization across domains and allows combining motion priors from multiple retrieved examples for more flexible and context-aware generation.
Image Animation. Animating a single image has received increasing attention. Several diffusion- based methods 25 26 27 have been proposed for open-domain image animation. Stable Video Diffusion I introduced a latent diffusion model with multi-stage training for high-resolution video generation. DynamiCrafter (2) projected images into text-aligned space using query transformer to better preserve visual details. 12VGen-X XLL (25) improved clarity and continuity via a two-stage design that decouples semantic and temporal modeling. VideoComposer (26) enabled controllable generation by encoding spatial-temporal conditions. MoG (28 used explicit motion guidance for high-fidelity frame interpolation. Motion-12V 27 relied optical flow improve motion consistency. Unlike Motion-12V which relies on low-level optical flow, our method extracts and injects high-level semantic motion features from retrieved references. These high-level motion representations capture more abstract dynamics that are easier to transfer across different visual domains and subject appearances. Additionally, our approach extracts these high-level motion representations in less than one second, compared to the several minutes required for optical flow generation in Motion-I2V, making our method substantially more efficient for practical applications.
Methodology
3.1 Framework Overview
To tackle the motion realism challenge in image-to-video generation, we propose a novel Motion Retrieval Augmented image-to-video Generation (MotionRAG for brevity) framework. Our approach uses simple yet effective three-stage process(retrieval, adaptation, synthesis) to improve motion quality in generated videos, as shown in Figure 2 Given an input image wx4 and a text prompt T E N with N tokens and embedding dimension de framework operates follows:

---

First, we perform text-based retrieval to identify the most contextually relevant video samples {V,) W from comprehensively indexed database. Subsequently. our Context-Aware Motion Adaptation component transforms these retrieved motion patterns into target image compatible features. Denoting the motion feature extraction function as f' and the image encoder as fi. we compute the adapted motion representation M as:
M =
where F; represents the first frame of retrieved video V,, and T denotes our motion context trans- former that adapts retrieved motion patterns to align with the visual characteristics of the target image. In the synthesis stage, we generate the output video V RT xxxxxxx3 with ''' frames by conditioning a diffusion-bast -aased generator on the input image, text prompt, and adapted motion features:
V G(I
(2)
This principled approach enables the generation of videos with enhanced motion fidelity while maintaining visual coherence with the input image and semantic alignment with the text description. By explicitly incorporating real-works motion priors through strieval-augmente generation, framework improves temporal dynamics and physical plausibility in the synthesized videos.
3.2 Text-based Video Retrieval
Text-based video retrieval provides relevant motion exemplars with high quality as references. Our retrieval pipeline comprises two interconnected components: database construction and semantic retrieval. To achieve robust and accurate retrieval, we chose to construct a retrieval database with text embedding as the retrieval index.
Retrieval Database Construction. We curate a diverse video dataset with associated captions. To keep simplicity, encode the corresponding captions with embedding techniques (e.g., Sentence- BERT 29 to generate a dense representation ej E Rd as the retrieval index.
Semantic Retrieval. These embeddings are indexed using approximate nearest neighbor techniques to facilitate efficient retrieval during inference. Input text prompt undergoes identical encoding to produce query embedding eg. We then compute the semantic similarity between this query and all database entries using cosine similarity:
ej Eg
8im v, T)
||ey ll eell
The system subsequently retrieves the top-K videos [V.) ordered by descending similarity scores, where i = 0 represents the most semantically aligned exemplar. This retrieval mechanism provides a foundation for our motion adaptation process by identifying real-world motion patterns that exhibit strong semantic alignment with the desired video content, thereby establishing a crucial bridge between text prompts and motion representations.
3.3 Context-Aware Motion Adaptation
To effectively transfer motion priors from retrieved videos to the target image, we propose Context- Aware Motion Adaptation. Figure illustrates our approach, which operates through three modules.
Motion Feature Extraction. We leverage the pretrained VideoMAE (30 encoder extract high- level motion representations from each retrieved video Vg. Unlike conventional optical flow that captures low-level pixel trajectories, these features encapsulate semantic motion patterns. The VideoMAE encoder processes each video V to produce dense apatio-temport features, which are then processed through learnable resampler (31] module that distills these representations into compact set of I motion tokens: f.. (V&) RLxd This approach enables us to capture coherent motion patterns while discarding appearance-specifie details that might hinder effective transfer across visual domains.

---

Target Mation
xN
Input Image
Ref. Video
Mation Context Transformer
Ref.
Image
Resampler
Encoder
Image Encoder
Video
Resampler
Encoder
Motion Encoder
OOO Image Features
Frozen
Trainable module
Motion Features
Learnable
module
Token
Context-Aware Motion Adaptation (CAMA) architecture. Appearance and motion features from retrieved videos and the target image are processed through causal transformer, which learns to predict appropriate motion features for the target image through in-context learning.
Figure 3:
Image Feature Extraction. For appearance encoding, utilize the DINO 32 vision transformer process both the target image I and the first frame F& of each retrieved video. These visual inputs are encoded and subsequently compressed through parallel resampler architecture obtain appearance features f (F) E IR Lxd and fi(I) E RR 4xd, We specifically design both appearance and motion resamplers to output the same token count L and feature dimension d, facilitating direct feature addition when constructing the transformer input sequence.
Adaptive Motion Transfer. To transfer motion features to the target image while preserving motion semantics, we introduce a causal transformer architecture that performs in-context learning of motion- rootion-appearance relationships. The retrieval system returns videos ordered by descending relevance (V1, V2...., V* where V1 represents the most semantically relevant video. We arrange these examples in reverse retrieval order: (Fx, Fx-1.-... F1, Fo), where Fo represents the target image. This reverse ordering serves a crucial purpose: it allows the model to first process less relevant examples, gradually transitioning to more relevant ones, and finally the target image. This progressive ordering enables the transformer to build a more refined understanding of motion- appearance relationships before addressing the target image. For each retrieved video frame F, n, and its corresponding motion feature fm (V.). we construct input tokens X,, = f. + fm ( (V.+1) by directly adding feature representations. The Motion Context Transformer (MCT) processes these examples sequentially, with a critical constraint on the attention mechanism: tokens corresponding to each video frame can only attend tokens from the same video and tokens from previously processed videos in the sequence. This causal attention mask ensures that predictions rely only previously observed examples, which essential maintaining the in-context learning paradigm, where the model learns patterns from example pairs without "peeking" at future examples.
By positioning the target image Fo last in the sequence, the model accumulates sufficient context from all retrieved examples before generating motion features for the target. The transformer infers compatible motion features M by extrapolating from the learned examples, effectively adapting motion patterns from retrieved videos the visual characteristics of the target image. This approach enables our model to rapidly adapt new visual domains without requiring explicit fine-tuning, as it learns to transfer motion patterns across varying appearance contexts through in-context learning.
3.4 Motion-Guided Video Generation
At the core of our approach a conditional diffusion model for video generation. Diffusion models [ 33 operate by gradually denoising random Gaussian noise through series of denoising steps. The
5

---

forward process progressively adds noise to the data, while the reverse process learns to denoise and recover the original data distribution. In conditional image-to-video diffusion models, this reverse process is typically guided by an image I and text prompt T, formulated as:
(4) where X₁ represents the noisy latent at diffusion timestep 6, and pa is the predicted noise mean parameterized by UNet incorporating the transferred motion features M an additional conditioning signal:
Pe Xo) Xx. I, T = (X, t, I, T), (x,,,,))), t)),
DiT 5 35 361. Our approach extends this framework by
Po(xo|x;, I, T, M) (X M), Ex(x,, t)),
(5)
To incorporate the transferred motion features into the diffusion process, we employ an adapter-based injection mechanism inspired by -P-Aaptter [37]. Our approach, which we call Motion-Adan integrates motion information without requiring modifications to the underlying generative model architecture. The motion-guidee video generation process starts with pretrained image-to- vi-iidoo diffusion model. We strategically insert our Motion-Adapter modules after each cross-attentic in the UNet or DiT backbone, allowing motion features to guide generation after text conditioning has been applied. These adapters modify the latent representations to follow the desired motion pattern while maintaining appearance consistency with the input image.
Formally, given the hidden features Z; at the i-th layer, which have already incorporated text conditioning through the text cross-attention mechanism, we compute:
Z = Z; + Attention Q; K,, Vi),
(6)
where: (i) Q, Z, ! represents queries derived from the text-conditioned visual tokens. (i)K,, NW provides keys from our motion features. (iiiVV MWT supplies values from the same motion features. The projection matrices W W and W learnable parameters specifie to each layer. Our approach preserves all original pretrained weights of the diffusion model, which remain frozen during training. Only the adapter-specific parameters are optimized, allowing our method to leverage the full generative capabilities of the base model while introducing motion control without catastrophic forgetting of pretrained knowledge.
3.5 Training and Inference
Our training strategy employs two-stage approach to effectively learn motion patterns and their domain-adaptive transfer. In the first stage, we train the Motion-Adapter and motion resampler modules using ground truth videos. Given video V and its first frame FD, we extract motion features using video encoder and train the motion resampler to produce compact, semantically representations. Concurrently, the Motion-Adappe these representations to reconstruct the original video. This stage ensures our adapter can effectively inject motion information into the generation process. In the second stage, we freeze both the motion resampler while training the Motion Context Transformer and image resampler. For each training video, we extract motion tokens fm (V) using the frozen resampler and construct in-context learning sequences by sampling similar videos from our database. The transformer is trained to predict the motion features of the target video using an L.2 loss between predicted features M and ground truth features fm (V):
is trained to condition the diffusion model using
Ciranefer = M - fm (V)
(7)
During inference, our framework operates end-to-end without requiring domain-specific fine-tuning. Given text prompt and image, we retrieve semantically similar videos from our database and process them through the Motion Context Transformer predict adapted motion representations. These features are then injected into the generation model via the MAtion-Adapter to produce the final video that exhibits the desired motion patterns while preserving the appearance of the reference image. A key advantage of our approach is its extensibility to new domains without parameter updates. To generate videos specialized domains (e.g., instructional videos, scientific visualizations). one only needs to construct a new retrieval database containing examples from the target domain. The model's in-context learning capabilities enable to adapt motion patterns across substantial domain shifts without explicit fine-tuning of any parameters, significantly enhancing its practical utility.

---

Experiments
4.1 Implementation Details
Video Generation Models. We implement our method on three image-to-video generation models: Stable Video Diffusion (SVD) M. Dynamicrafter (2), and Cog C5 [5]. For SVD and Dy- namicrafter, we generate 16-frame videos at x resolution, while Cog VideoX-S0 produces 17-frame videos at x20x480 resolution. We insert our Motion Adapter modules after every text cross-attention layer in SVD and Dynamicrafter, and after each MMDiT layer in Cog CogVideoX-5b All adapter modules are trained using Adam W optimizer for approximately 50,000 steps with a batch size of 16 the Open Vid- 1M dataset (38].
Context-Aware Motion Adaptation. Our CAMA module utilizes a VideoMAF base model 30 as the video encoder and DINOv2-large [32 as the image encoder. Both Resampler modules employ 4-layer Transformer architectures that distill features into 25 tokens with dimension 1024. The Motion Context Transformer consists of 4-layer causal Transformer with hidden dimension 1024, trained with A optimizer for 50,000 steps with a batch size of 64. During training, we retrieve the most similar videos from Open Vid- 1M to construct the in-context learning examples. Additional implementation details are provided in the supplementary material.
Retrieval Database. We construct three video retrieval databases using the G.E-v1.5 model [39 to encode video captions into embedding vectors, Retrieval is performed efficiently via cosine similarity between the query text and the pre-computed caption embeddings. Leveraging high-performance vector database, this retrieval step is extremely fast, taking only approximately 40ms to query 1- million-ente cellionn-enty database on a standard CPU, ensuring it is not bottleneck in the generation pipeline. Our databases include: (1) Open Vid-1M [38] A large-scale, general-domain video dataset. To create more motionn-ooussee
captions suitable for retrieval, we performed one-time, automated preprocessing step on the original detailed descriptions. We utilized the Llama3. 1-8B model to generate concise, motion-centric summaries for each video, a process that both scalable and easily reproducible. This refined dataset will be made publicly available to facilitate future research. (2) SkillVid [4]]: A specialized dataset containing instructional and skill-based vidcos. This database is used to demonstrate our framework's 's ability to adapt to domain-specific motions. (3) Intern Vid- 10M 41 A massive-scale video-text dataset originally curated for video understanding tasks. Its data distribution (e.g., content, style, captioning focus) differs significantly from our cnneerationfocussee datasets. We use as a challenging out-of-distribun (OOD) database to test the generalization capabilities of our framework.
4.2 Datasets and Metries
We evaluate our method on two datasets: (1) Open Vid-1K, diverse test set of 1,000 videos sampled from Open Vid- M (3 with no overlap with the training data, representing general video domains;
and (2) SkillVid 40 test set, the test set of SkillVid that we use to assess zero-shot capabilities.
While existing benchmarks like VBench (42) focus primarily detecting visual artifacts such as flickering, they lack ground truth videos and cannot effectively measure whether generated content exhibits realistic motion patterns. More recent work on physical realism evaluation 48 utilizes low-level metrics like spatial loU, which are highly effective for simple, controlled scenarios but are less robust for the complex, open-domain videos in our evaluation, where minor, physically plausible variations in motion could be unfairly penalized. Therefore, following prior work [42] 40), we adopt suite of complementary metrics that holistically assess quality by comparing to ground-truth videos. These include: Action Score (semantic motion correctness). CLIP Score (frame-level semantic alignment), DINO Score (frame-level identity preservation), FID
44 (frame visual quality), and FVD (55 (overall video quality). Among these, the Action Score is particularly crucial for our task as it evaluates motion from a high-level, semantic perspective, which is more indicative of physical plausibility in complex scenes than pixel-level metrics.
4.3 Results on General Domain
Table | presents quantitative results on the Open IK test set. Our retrreeal-augmeeneed approach significantly enhances performance across all baseline models. Notably, when applied to Video.
7

---

Table I: Quantitative comparison on the Open Vid- 1K test set. Higher is better for Action, CLIP, and DINO scores; lower is better for FVD, FID, and Time (minutes).
Table 2: Zero-shot transfer evaluation on the SkilIVid. Our method demonstrates gen- eralization to instructional videos without any fine-tuning.
Action?
CLIP
DINO!
FVDJ
FID.
Time!
Model
Action
CLIP)
DINO;
FVD]
FID.
Cog 5
59,9
91.2
87.8
87.1
11.8
0.99
Cog
51.5
87.7
78.7
91.8
10.0
Cog+RAG
65.8
92.1
89,4
80.2
11.4
1,08
Cog+RAG
53.5
89.1
83.2
DC 2
53.5
91.0
85.8
88.4
10.9
1,46
DC 2
49.6
90.0
82,9
108.5
8.1
DC+RAG
DC-RAG
50.1
88.9
80.5
89.4
SVD 0
57,5
87.2
83.6
98.0
15.7
0.74
SVD
48.0
86.1
79.6
127.1
12.6
SVD+RAG
60.0
91.8
89.6
167.1
134
0.75
SVD+RAG
49.1
90.2
85.2
130.5
10.1
our method yields the highest overall Action similarity score of 65.8, representing substantial 5.9-point improvement that demonstrates RAG's ability to enhance state-of-the-art DiT models.
For Dynamicrafter, our approach increases the Action score by 8.6 points, while SVD shows 2.5- point improvement. Both CLIP and DINO scores improve consistently across all models, indicating better semantic alignment and identity preservation. The FVD score improvements are particularly striking for Cog VideoX (8.0% reduction) and Dynamicrafter (22.0% reduction).
Crucially, these substantial improvements come with negligible computational overhead. Our RAG approach adds only 0.01-0.06 minutes (less than 4 seconds) to inference time across all models, demonstrating the practical viability of our approach for real- weal-world applications. The consistently positive results across diverse model architectures, coupled with the negligible computational cost, underscore that our RAG framework is highly effective and practical plug-and-play module for enhancing state-of-the-ar video generation models.
4.4 Zero-Shot Transfer to Specialized Domains
Table demonstrates our method's zero-shot generalization the SkillVid dataset. Without any fine-tuning, RAG improves motion realism across all models, with Cog VogVi showing the most substantial gain (+2.0 points in Action score). Overall video quality improves significantly, with FVD reductions of 9.4% for CogVideoX and 17.6% for Dynamicrafter. SVD benefits particularly in semantic alignment (CLIP +3.8, DINO +5.4), while Dynamicrafter shows modest trade-off's between motion quality and semantic preservation. These results show that our approach effectively transfers to specialized domains by simply changing the retrieval database, requiring no parameter updates.
4.5 Ablation Studies
We conduct detailed analysis of our framework's components using the Dynamicrafter model on the Vid-I K dataset. The ablation studies, presented in Tables 3 and systematically validate our design choices. Specifically, significantly outperforms simpler feature integration strategies, (2) the framework is remarkably robust noisy context, and (3) it generalizes effectively to out-of-distribution retrieval database.
demonstrate that (1) Motion Context Transformer (MCT)
Impact of Motion Adaptation Method. As shown in Table 3 our Context-Aware Motion Adapta- tion (CAMA) module is the key to superior performance. While naive approaches like using the top- retrieved video 1) or averaging features from videos (Avg-9) provide moderate gains over the baseline, our Motion Context Transformer (MCT-9) dramatically outperforms them. It achieves an Action score of 62.1 (+8.6 over baseline and +4.2 over Avg-9) and reduces FVD to 69.0 (-19.4 from baseline and -9.7 from Avg-9). This highlights that our method's strength lies not just in retrieval but in the intelligent, in-context synthesis of motion. Remarkably, our performance is very close to the theoretical ceiling established by using ground-truth motion features (Dracle), achieving 97.2% of the oracle's ' Action score.
Effect of Reference Video Quantity. Comparing the results for 5 and 9 reference videos in Table 3 shows that more context beneficial. Increasing the number of references from boosts the Action score for our method by 1.1 points (MCT-5 vs. MCT-9). In contrast, the improvement for

---

Table 3: Comparison of motion adaptation methods. Our MCT outperforms simpler fea- ture integration strategies and approaches the performance of using ground-truth motion.
Table 4: Robustness and Generalization. Our method demonstrates strong robustness to noisy (random) retrieval and generalizes effectively to an out-of-distributi (OOD) retrieval database.
Method Actiont
CLIP
FVD.
FIDJ
Method
Action
CLIP
DINO
FVD.
FIDI
Baseline
53.5
91.0
85.8
$8.4
10.9
Baseline
53.5
91.0
85.8
88.4
10.9
Top-1
86.6
Avg-5
$7.5
91.8
88.1
78.2
11.1
Rand-1
52.2
90.3
85.5
88.4
11.9
Avg-9
$7.9
92.0
88.3
78.7
10.9
Avg-Rand-9
56.4
92.2
MCF-5
MCT-Rand-9
60.7
91.6
87.6
77.5
10.9
MCT9 (Ours)
62.1
92.3
88.4
69.0
9.7
MCT-Open
62.1
92.3
69.0
9.7
Oraclo (GT)
63.9
90.6
85.3
71.5
10.7
MCT-InieenNidd99
60.9
91.7
87.4
70.7
10.5
simple averaging is minimal (+0.4 for Avg-5 vs. Avg-9) 9), indicating that our CAMA module is more effective leveraging richer contextual information refine motion synthesis.
Robustness to Noisy Retrieval. To rigorously test the model'ss robustness, simulate worst-case
scenario with noisy retrieval by providing completely random videos as context, as shown in Table While using single random video (Rand-1) slightly degrades performance, our CAMA module (MCT-Rand-3) still achieves a remarkable Action score of 60.7, far surpassing the baseline (53.5). This demonstrates that CAMA is not fragile; when faced with irrelevant context, it relies on its strong, learned priors of plausible motion to generate high-quality result, effectively filtering out noise.
4
Generalization to Duttr--Distribution Data. We evaluate out-of-distributtin (OOD) generaliza- tion by switching the retrieval database from the in-domain Open Vid- IM (38) to the large-scale InternVid-10M 44 dataset, which has a significantly different data distribution. As seen in Table the performance of MCT-InternVid-9 remains exceptionally strong (Action 60.9, FVD 70.7), closely tracking the in-distribution results. This demonstrates that our framework is not simply interpolating between similar examples within familiar dataset. Instead, it has learned extract and adapt fundamental, transferable motion principles, making a truly generalizable solution.
4.6 Qualitative Results
Figure comparre our retrieval-augmentte approach against baseline models across diverse scenarios. The results show consistent improvements in motion plausibility and temporal coherence.
When enhancing Dynamicrafter, our method corrects fundamental motion errors. For example, transforms the unphysical floating of the Newton's 's cradle balls into a realistic swinging motion. It also animates subjects that are nearly static in the baseline, instilling natural movement the gesturing man and the jumping person. Similar improvements are seen with Cog VideoX. Our approach induces clear progressive motion for the tram, which is largely static and distorted in the baseline video. also replaces the unnatural "flickering" motion of the runner and the horse with proper gait cycles.
These qualitative results, consistent with our quantitative findings, demonstrate our method's ability to transfer motion semantics to generate more convincing videos. The approach proves especially effective for complex physical phenomena, biological motion, and expressive human actions. Visual- izations of the retrieved reference videos that guide the generation process, along with more video results, are available in our supplementary material and on our project website.
Despite its robustness, our method can falter when retrieved videos contain directly conflicting motion cues. For example, to generate "person jumping." the retrieved might include both the upward and downward phases of a jump. In such cases, the model may "average" these opposing motions, resulting in a nearly static video with minimal vertical movement. This outcome is known characteristic of models trained with objectives like MSE, which tend to find a mean solution when faced with conflicting targets.
Conclusion and Limitations
We have introduced MotionRAG. a novel aaugentl-augmentedd framework that enhances motion realism in image-to-video generation by effectively transferring motion patterns across domains.

---

suspended
a red and whie tram Traveling down smowyy streel
DC
Coo
DC+RMG
CoouRAS
panda bear is eatnng piece bamboo
back
DC+RAG DC
Co
CoPARAG
person jumping
man running on dirt road
DC-RAG DC
CooRAA
man gesturing with hands, explaining the recie
pereon is pouning water into a teacup
DC+RAG DC
Cog4RAB Cog
Figure 4: Qualitative comparison between baseline models and our Aurieval-augmentter ap- proach across diverse scenarios. Our method generates more physically plausible and coherent motion, such as realistic object physics, natural animal/human movements, and corrects static or artifacts found in baseline models. Video results available
website
Our Context-Aware Motion Adaptation module formulates motion transfer as an in-context learning problem, enabling robust adaptation without domain- sppcciff fine-tuning. Experiments demon- strate the framework's strong robustness and zero-shot transfer capabilities, showing its consistent improvements across different base models and domains with negligible inference overhead.
Despite these promising results, several limitations still remain. First, primary limitation is the potential for motion cancellation when retrieved references contain contradictory movements, which can lead to static-like outputs. In addition, while our metrics indicate improved semantic motion, quantitatively evaluating physical plausibility in open-domain videos remains challenging open research problem. In fact, the development of robust, automated metrics for physical realism is an important issue that would benefit the entire field. On a more practical note, our current framework. constrained by the base models, is designed for short clips, though it could be extended to longer videos by integrating it with models possessing stronger temporal modeling capabilities. Finally, generating fine-graie fine-grained motions, such subtle hand gestures, would require both more specialized retrieval corpus and video encoder with higher spatiotemporal resolution, marking clear path for enhancing motion detail in future iterations.
From a broader perspective, our research demonstrates the potential of integrating retrieval mech- anisms into generative models, establishing a promising direction for enhancing video generation beyond end-to-end training alone. We believe this retrieval-aus augmentte paradigm represents an important step toward more realistic and controllable video generation systems that can effectively leverage existing motion knowledge.
Acknowledgement
This work is supported by the National Key R&D Program of China (No. 20227D0 2022ZD0160000,, the Basic Research Program of Jiangsu (No. BK20250009), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
10

---

References
[1] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint
2023.
[2] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In European Conference on Computer Vision, pages 399-417. Springer, 2024.
[3] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:24/2.03666 2024.
[4] Guojun Lei, Chi Wang, Hong Li, Rong Zhang, Yikai Wang, and Weiwei Xu. Animateanything: Consistent and controllable animation for video generation. arXiv preprint arXiv:2411.10836. 2024.
[5] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 2xiv:24408.060722 2024.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-685 2020.
[7] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guider video synthesis with diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7346-7356, 2023.
[8] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video Text-to-video generation without text-video data, arXiv preprint arXiv:2209. /4792, 2022.
[9] Yixin Liu, Kai Zhang. Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17117, 2024.
[10] Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Learning character-agnostic motion for motion retargeting in 2d. arXiv preprint urXiv:/905.0// 0000000 2019.
[11] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5933-5942, 2019.
[12] Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4116-4125. 2020.
[13] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018
[14] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[15] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234 2022.
11

---

[16] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. aetrieval-augmentt generation for snowledge-intenstee nlp tasks. Advances in neural information processing systems, 33:9459-9474 2020.
[17] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval -etmentllaugmmented generation for large language models: survey, arXiv preprint urXiv:2312. 10997, 2, 2023.
[18] Rotem Shalev-Arkushine Rinon Gal, Amit Bermano, and Ohad Fried. Imagerag: Dynamic image retrieval for reference-guided image generation. arXiv preprint arXiv:2502.09411 2025.
[19] Haoran Cheng, Liang Peng, Linxuan Xia, Yuepeng Hu, Hengjia Li, Qinglin Lu, Xiaofei He, and Boxi Wu. Searching priors makes text-to-video synthesis better. arXiv preprint a4Xiv:2406.03215, 2024.
[20] Rui Zhao, Yuchao Gu, Jay Zhangjic Wu, David Junhao Zhang. Jia-Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. In European Conference Computer Vision, pages 273-290. Springer, 2024.
[21] Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. Vme: Video motion customization using temporal attention adaption for text-to-vides diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9212-9221 1229221, 2024.
[22] Yujie Wei, Shiwei Zhang, Zhiwu Qing. Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with cus- tomized subject and motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6537-6549, 2024.
[23] Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, and Abhinav Shrivastava. Customize-a --vioiii----idooo One-shot motion customization of text-to-video diffusion models. In European Conference on Computer Vision, pages 332-349. Springer, 2024.
[24] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
[25] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.041 2023.
[26] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang. Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer. Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36:7594-7611, 2023.
[27] Xiaoyu Shi, Zhaoyang Huang, Fu- Yum Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin. et al. Motion-i2v: Consistent and controllable todge-to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pages 1024,
[28] Guozhen Zhang, Yuhan Zhu, Yutao Cui, Xiaotong Zhao, Kai Ma, and Limin Wang. Motion- aware generative frame interpolation. arXiv preprint urXiv:2501.03699, 2025.
[29] Nils Reimers and Iryna Gurevych. Sentence-ber Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 10084, 2019.
[30] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for supervised video pre-training. Advances in neural information 10093, 2022.
processing systems,
[31] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention, In International conference on machine learning, pages 4651-4664, PMLR, 2021.
12

---

[32] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 07/93, 2023.
[33] Shuai Wang, Zexian Li, Tianhui Song, Xubin Li, Tiezheng Ge. Bo Zheng, Limin Wang, et al. Differentiable solver search for fast diffusion sampling. arXiv preprint arXiv:2505.21114 2025.
[34] Shuai Wang, Zexian Li, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, and Limin Wang. Exploring den-like architecture for fast image generation with arbitrary resolution. Advances in Neural Information Processing Systems, 37:87959-87977 2024.
[35] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[36] Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. arXiv preprint arXiv:2504.05741, 2025.
[37] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0672) 2023.
[38] Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- m: A large-scale high-quality dataset for text-to-video generation. arXiv preprini urXiv:2407.02371, 2024.
[39] Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, al. mgte: Generalized tation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407.19669, 2024.
text represen-
[40] Yilu Wu, Chenhui Zhu, Shuai Wang, Hanlin Wang. Jing Wang, Zhaoxiang Zhang, and Limin Wang. Learning human skill generators at key-step levels. arXiv preprint arXiv:2502.08234 2025.
[41] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.066: 2023.
[42] Ziqi Huang, Yinan Hc, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11807-21812 2024.
[43] Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles? arXiv preprint arXiv:2201.0900 2025.
[44] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[45] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marein Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. ICLR, 2019.
[46] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842-
5850, 2017.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We claim our contribution in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
The code and weights will be open-sourced when this paper accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pullc//uuuii wwwtii/gguie///eeeeceoocc CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: computation resources limitation
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is technical enhancement to existing video generation methods that improves motion quality without introducing new capabilities applications that would create additional societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is technical enhancement to existing video generation methods that improves motion quality without introducing new capabilities or applications that would create additional societal impacts.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy hateop//meerrpppc.cc///offeerere for what should or should not be described.
20

---

A Implementation Details
This section provides comprehensive technical details about our MotionRAG framework implementa- tion, covering network architectures, training procedures, and inference pipeline configurations.
Video and Image Encoders. We employ VideoMAEE 330] pre-trained Something-Somettingg
v2 [46] as our video encoder. We process 16 frames at 224x224 resolution and extract features from all tokens of the final layer. For image encoding, we utilize DINOv2-Large [32] which employs
ViT-L/14 architecture with a hidden dimension of 1024.
Resamplers. Our framework employs two separate resamplers for motion and appearance features. These resamplers compress the encoder outputs into a compact set of tokens for efficient processing. The configuration details are provided in Table 5
Table 6: Configuration for Motion Context Transformer.
Table 5: Configuration for Resamplers.
Configuration
Motion Resampler
Image Resampler
Configuration
Motion Context Transformer
Architecture
Transformer
Transfrmer
Architecture
Causal Transformer
Layers
Layers
Attention heads
Attention heads
Hidden dimension
768
768
Hidden dimension
Foed-forward dimension
4096
4096
Feed forward dimension
4096
Maximum sequence length
500
Oulput tokens
Attention mask
Causal
Input feature dimension
768 (VideoMAF)
1024 (DINOv2)
Dropout rate
0.0
Output feature dimension
1024
1024
Position embodding
Simusoid LayerNorn GELU
Dropout rate
0.0
0.0
Normalization
Trainable parameters
48.0M
48.3M
Activation
Initialization
Random
Random
Trainable parametirs
Motion Context Transformer. Our Context-Award Motion Adaptation (CAMA) module uses a causal transformer architecture to facilitate in-context learning for motion transfer. The detailed specifications are provided in Table
Motion Adapters. We implement separate Motion Adapters for SVD, DynamiCrafter, and Cog Vib, inserting them after text cross-attention layers in the respective UNet architec- tures for SVD and DynamiCrafter, and all MMDiT layers for Cog VideoX-5b. The configuration details for all adapters are provided in Table 7
Table 7: Configurations for Motion Adapters across different video generation models.
Configuration
SVD Motion Adapter
DC Motion Adapter
Cog VidexX Motion Adapter
Architecture
Cross Attention
Cross-Attention
Cross-Attention
Insertion points
After text cross-atlention layers
Afler text cross-atiention layers
After MMDiT self-attention
Number of adapters
16
16
42
Attention heads
8
48
Key/Value dimension
1024
1024
3072
Scale factor
1.0
1.0
1.0
Trainable parameters Initialization
38M
38M
660M
Random
Random
Rundom
Training Protocol. We employ a two-stage training approach for our MotionRAG framework across all three models (SVD, Dynamicrafter, and Cog -5b) the first stage, we train the Motion Adapter and Resampler modules, followed by training the Motion Context Transformer in the second stage. The training hyperparameters for all configurations are detailed in Table g
Table 8: Training hyperparameters for the two-stage approach across different models.
Hyperparameter
Stage (SVD)
Stage (DC)
Stage (Cog)
Stage 2 (Transformer)
Dataset
OpenVid-1M IM 38 AdamnW 10-1
OpenVid-IM 38
OpenVid-IM 38 Adamn
OpenVid-IM 88 Adain
Optimizer
Adamn
Learning
5 x 10-3
10-2
x 10
Resolution
x 576
480 720
224 64 (8 per GPU) S0K MSE (motion features)
Batch size
16 (2 per GPU)
16 (2 per GPU)
8 per GPU) 60K
Training steps
90K
60K MSE (dennised prediction)
Loss function
MSE (denoised prediction) NVIDIA RTX A600 GPUs 48 hours
MSE (denoised prediction)
Hardware
NVIIII A60DD GPUs
NVIDIA RTX A6000 GPUs 108 hours
NVVIDI RTX A60UD GPUs hours
Training time
90 hours
21

---

Video Retrieval System. Our text- extt-baed retrieval system uses TTE--base-1.5-rr 13 to encode text queries and video captions into embedding vectors. For all experiments, we retrieve the top-9 most relevant videos based on cosine similarity between text embeddings.
Video Generation. We implement approach on three state-of-the image-to-video generation models: Stable- Gaoble-Video-Daee "iddeo-Diffusioniimee (SVD) M. Dynamicrafter- 1024 (DC) [2]. and Cog V The hyperparameters used for video generation are provided in Table
5b-12V [5]
Table 9: Generation hyperparameters for Open Vid- IK and SkillVid dataset.
Hyperparameter
SVD
DC
CogVideoX
Resolution
376 1024
1024
Frame count
16
16
Sumpler
EDM
DDIM
DPM
Steps
30
25
CFG scale
1.0-3.0
2.0
3
FPS/Mation Strength
Inference time (A6100)
44s
90s
60s
For video preprocessing during training, extract 16-frame clips at FPS with random temporal crops during training, and for each vidco, we use the first frame as the reference image.
B Extended Visualization Results
This section presents additional qualitative results generated by our MotionRAG framework across diverse range of scenarios.
B.1 Retrieval Visualization
To illustrate how our retrieval mechanism influences motion generation, Figure 5 shows examples of the retrieval process and resulting generated videos. For each query prompt, our system retrieves semantically relevant videos that contain similar motion patterns, which then guide the generation process.
These examples demonstrate how our approach transfers motion characteristics across visual domains:
Physics- based motion: For "metal balls suspended in the air" the system retrieves videos of Newton's 'ss cradles, magnetic balls, and physics experiments. The generated video exhibits realistic pendulum- like oscillations derived from these references.
For person pouring water into a teacup" retrieved videos show various pouring actions with different teapots and cups. The generated video captures the natural flow of liquid and the subtle hand movements during pouring.
Fluid dynamics:
Human locomotion: For "a man running on a dirt road" the system retrieves videos of people jogging in various environments. The generated video reproduces natural running gait and body mechanics.
Animal-humal interaction: For "a person riding on the back of a horse led by another person" retrieved videos show various horse-riding scenarios. The generated video captures the coordinated movement between horse and riders.
Despite differences in background, lighting, and specific object arrangements, the retrieved videos provide valuable motion priors that guide the generation process. The resulting videos exhibit realistic motion while maintaining the visual appearance specified in the input images.
B.2 Additional Generation Results
F showeases video sequences generated using Dynamicrafter+RAG and CogVIGeo++RAG models, demonstrating their ability to produce realistic motion patterns across various domains.
These results highlight our methods' ability to transfer motion patterns across visual domains while maintaining physical plausibility and semantic consistency. The generated videos preserve the appearance specifications while introducing temporally coherent motion that aligns with the described actions. For the best view, please refer the videos in our website (https://gthub.com// NJU/MotionRAG).
22

---

suspendied
C
acco
videos
man running on dirt road
person riding back horse led by another person
retrieved videce
relrieved videos
Figure 5: Retrieval and generation examples. Each panel shows a different scenario: (top-left) metal balls suspended in air with pendulum-like motion, (top-right) person pouring water into teacup, (bottom-left) man running on dirt road, and (bottom-right) a person riding on horse led by another person. For each example, the top row displays frames from our generated video, while the rows below show frames from retrieved reference videos. Note how our system extracts relevant motion patterns from visually different but semantically similar videos.
23

---

motorcycle driving down road
bunch of cars driving on highway
a penguin walking on beach near the water
red double decker bus driving down street
city bus driving down snowy street at night
an older man jogging by the water
a red panda eating bamboo in zoO
A yellow boat cruising in front of bridge
a zebra walking across a dirt road near a field
Figure 6: Additional video generation results. Each row displays five frames from a generated video sequence. The first four rows show results from Cog Vide while the remaining rows present Dynamicrafter4RAG outputs. Our approach successfully captures motion characteristics across these diverse scenarios.


Reviews:
[{"review_id": "o3mkR5fXGX", "reviewer": "Reviewer_v6PA", "summary": "This paper proposes a retrieval-augmented framework, MotionRAG, to enhance motion realism in video generation by transferring motion priors from relevant reference videos. The framework addresses the challenge of generating physically plausible and semantically coherent motion. MotionRAG contains a key component, Context-Aware Motion Adaptation (CAMA), that adapts retrieved motion priors to the target image context with in-context learning. Experiments demonstrate that MotionRAG can improve motion quality across multiple base models and domains with negligible computational overhead.", "questions": "- The captions given by the existing video captioners are bound to be rough and unable to accurately describe the physical motions. When constructing a dataset, is it necessary to manually select homogeneous reference videos?\n- I highly doubt the robustness of filtering out reference videos based solely on text similarity. Text cannot convey the details of physical motion and may lead to incorrect retrieval. Even for the examples shown in the supplementary material, the text descriptions are inaccurate. There are also large differences in physical motion between the retrieved reference videos. What about the reference video retrieved when the text description is out of domain?", "limitations": "I would suggest adding more analysis and visualization of the dataset in the revised version.\n\nAnd more metrics for measuring physical motion need to be considered.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- This paper applies the RAG technique to video generation to improve motion realism, which is meaningful for the video generation task.\n- This paper achieves zero-shot effective motion transfer across different domains with negligible computational overhead.\n- The writing of this paper is clear and easy for the reader to follow.\n\nWeaknesses\n- The RAG technique itself is not novel and has been proven to be effective in other related generation tasks. Therefore, the contribution of applying RAG to motion transfer is relatively weak.\n- One of the crucial points of RAG is to construct the dataset, but this paper lacks the necessary data analysis. How such datasets are constructed is important to the community.\n- This paper lacks a detailed analysis and visualization of the semantic similarity of the retrieved reference videos. Based on the current version, MotionRAG relies on the existence of videos with highly similar motion in the dataset and cannot accurately achieve motion transfer.\n- Existing evaluation metrics cannot fully reflect the quality of motion, especially the physical constraints claimed in this paper.", "was_revised": true, "final_justification": "Most of my concerns have been addressed in the rebuttal.\n- The dataset construction is automated and scalable.\n- MotionRAG outperforms the baseline when processing OOD data.\n\nBut the experiments related to physical constraints are still insufficient.", "rebuttal": "We sincerely thank Reviewer v6PA for the detailed and constructive feedback. We appreciate that the reviewer recognized the significance of our work in improving motion realism, its zero-shot capabilities, and the clarity of our writing. We would like to address the raised concerns, which we believe stem from a potential misunderstanding of our core technical contribution—particularly the robustness and learned priors of our CAMA module.\n\n**1. Regarding Novelty: The Core Innovation is CAMA's In-Context Adaptation, Not Just the Application of RAG.**\n\nWe agree that RAG is an established paradigm. However, our main contribution is not simply *applying* RAG, but proposing a novel method for how motion priors are adapted and synthesized from imperfect, real-world data. This is the specific function of our Context-Aware Motion Adaptation (CAMA) module.\n\nOur ablation study in Table 3 provides direct evidence of this. A naive RAG approach that simply retrieves and averages features (\"Avg-9\") offers only a moderate improvement, yielding an Action score of 57.9 and an FVD of 78.7. In contrast, our CAMA module (\"MCT-9\") achieves significantly superior results, with an Action score of 62.1 and a much lower FVD of 69.0. It demonstrates that the innovation lies in the intelligent, in-context synthesis of motion, not just the retrieval framework. CAMA learns to *adapt* motion by reasoning over multiple examples, a far more complex and novel task than simple feature blending.\n\n**2. Regarding Dataset Construction and Robustness to Imperfect Retrieval.**\n\nThis is the reviewer's primary concern, and we thank them for the insightful questions. Our framework is explicitly designed to be robust to imperfect and diverse retrieval results, which is a key strength that we wish to clarify.\n\n- **Dataset Construction is Automated and Scalable:** We apologize for not making this clearer. A key strength of our method is that it does not require any manual curation or selection of reference videos. For the OpenVid-1M dataset, we used an Llama3-8B model in a one-time, automated preprocessing step to generate concise motion-focused descriptions from the original detailed captions. This makes our approach practical, easily reproducible, and scalable.\n  \n- **CAMA's Robustness is Due to Learned Priors:** The reviewer rightly doubts the reliability of text-based retrieval. Our CAMA module is specifically designed to handle this challenge. During its training on millions of video, CAMA learns a powerful internal prior about plausible motion for given objects and scenes. This learned knowledge makes it robust, as demonstrated by the following experiments:\n  \n  | Retrieval Method | Action↑ | FVD↓ | FID↓ | CLIP↑ | DINO↑ |\n  | --- | --- | --- | --- | --- | --- |\n  | No Retrieval (Baseline) | 53.5 | 88.4 | 10.9 | 91.0 | 85.8 |\n  | Random Retrieval | 60.7 | 77.5 | 10.9 | 91.6 | 87.6 |\n  | OOD Retrieval | 60.9 | 70.7 | 10.5 | 91.7 | 87.4 |\n  | In-Distribution Retrieval | **62.1** | **69.0** | **9.7** | **92.3** | **88.4** |\n  \n  The data above reveals two critical insights into CAMA's robustness:\n  \n  1. **It excels even with random, irrelevant context.** The \"Random Retrieval\" row represents the worst-case scenario. Even here, CAMA dramatically improves the Action score from 53.5 to 60.7 and reduces the FVD from 88.4 to 77.5. This proves that CAMA does not blindly trust its inputs. Instead, it falls back on its strong, learned priors to generate a physically plausible motion.\n  2. **It generalizes across different data distributions.** The \"OOD Retrieval\" experiment serves as a stringent test of this capability. We retrieved from InternVid-10M, a large-scale dataset curated for video *understanding*, whose distribution (content, style, captions) differs substantially from our generation-focused dataset. Even with this domain shift, performance remains remarkably strong (Action 60.9, FVD 70.7), closely tracking the in-distribution results. It proves that CAMA is not merely interpolating between similar examples but has learned to extract and adapt fundamental motion principles, even from a out-of-distribution source. This directly addresses the concern that our method might be fragile and overly reliant on the retrieval database's domain.\n\nIn essence, CAMA does not simply copy motion; it *infers* it, using retrieved examples as context but relying on its own learned understanding of motion physics and semantics to produce a robust final result.\n\n**3. Regarding Evaluation Metrics and Physical Plausibility.**\n\nWe concur with the reviewer that quantitatively measuring physical plausibility is a challenging, open research problem.\n\nOur evaluation philosophy is conceptually aligned with recent, dedicated work on this topic (e.g., Motamed et al., \"Do generative video models understand physical principles?\", 2025). Both our approach and theirs rely on the core principle of comparing a generated video to a ground-truth (GT) video, under the assumption that the GT video is physically plausible.\n\nThe primary difference lies in the specific metrics chosen, which reflects the different nature of the evaluation data.\n\n- Metrics like Spatial/Spatiotemporal IoU and MSE, as used in the cited work, are highly effective for their dataset of simple, single-object interactions in controlled settings.\n- However, these low-level, coordinate-based metrics are not robust for the complex, open-domain scenarios in our evaluation, which often feature articulated human/animal motion, multiple interacting objects, and camera movement. In these cases, a minor, physically plausible variation (like a slightly different camera angle or running gait) could unfairly penalize a low-level metric score.\n- Therefore, we opted for a suite of metrics that are better suited for our setting. The Action Score, in particular, uses a high-level action recognition model to assess if the *semantic* nature of the motion is correct. The consistent and significant improvements across our entire suite of metrics (Action Score, FVD, etc.) provide strong, holistic evidence of enhanced motion quality.\n\nIn summary, we believe the reviewer's primary concerns are centered on the perceived fragility of a retrieval system. We hope our explanation clarifies that our core contribution, CAMA, is a robust motion adaptation mechanism that leverages in-context learning to thrive on diverse, imperfect, and even out-of-distribution reference data. We respectfully request the reviewer to reconsider their evaluation in light of these clarifications."}, {"review_id": "McpLOuNOmr", "reviewer": "Reviewer_SsZX", "summary": "This manuscript presents MotionRAG, a retrieval-augmented framework designed to improve motion realism in image-to-video generation. The core contributions are:\n\n-A pipeline that uses sentence embeddings to retrieve semantically relevant reference videos from a large corpus, providing real-world motion priors for generation.\n-Lightweight adapters inserted after cross-attention layers in frozen diffusion models, seamlessly integrating adapted motion features to guide video synthesis.", "questions": "see comments above.\nIt will be necessary to address the comments above to maintain positive score.", "limitations": "see comments above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n-By explicitly leveraging retrieved video examples, MotionRAG grounds motion dynamics in real data \n-The framework’s modular design allows for zero-shot transfer.\n-The paper thoroughly investigates retrieval quality (random vs. semantic), number of references (5 vs. 9), and integration strategies (Top-1, Avg-9, MCT-9)\n\nWeaknesses:\n-Experiments focus on short clips (~2 seconds) and a fixed set of general and instructional domains; the approach’s scalability to long-range motions or fine-grained action nuances (e.g., subtle gestures) remains untested.\n-Large-scale retrieval (e.g., nearest-neighbor search over millions of embeddings) can be nontrivial; reporting end-to-end latency including retrieval indexing would be necessary.\n-MotionRAG’s performance hinges on the diversity and relevance of the retrieval corpus; can rare or unique motions perform well? better to add analysis and failure cases\n-Will the work be open-source or public? It will be necessary to provide temporary/anonymous repo and demo code during the review.", "was_revised": true, "final_justification": "After going through the response, the reviewer tends to maintain the original score as it is.", "rebuttal": "We sincerely thank Reviewer SsZX for the positive and insightful review. We are delighted that the reviewer recognized the effectiveness of our retrieval-based approach, the flexibility of our modular design, and the thoroughness of our experimental investigation. We appreciate the constructive feedback and would like to address the raised points to further strengthen our paper.\n\n**1. Regarding Scalability to Longer Videos and Fine-Grained Motions (Weakness 1).**\n\nWe agree that extending our work is a valuable future direction.\n\n- **Current Scope on Short Clips:** Our experiments focus on ~2 second clips, which is inherent architectural limitations of the baseline models we used, such as SVD and DynamiCrafter, which are themselves designed primarily for short video generation. Within this established setting, we have demonstrated significant improvements.\n- **Potential for Longer Videos:** Our framework is not fundamentally limited to short clips. When integrated with base models possessing stronger temporal modeling capabilities, such as CogVideoX, we observed that our approach can readily extend to generate coherent videos of up to 6 seconds. We focused on 2-second clips in our reported results primarily due to computational constraints but will mention this extensibility in the paper.\n- **Fine-Grained Motions:** We agree that fine-grained motions (e.g., subtle gestures) pose a challenge. Performance would depend on both a specialized retrieval corpus containing such examples and a video encoder capable of capturing these nuances. We will explicitly state these points as promising avenues for future research in our Limitations section.\n\n**2. Regarding End-to-End Latency Including Retrieval (Weakness 2).**\n\nThis is an excellent point regarding practical deployment. Our framework is designed for high efficiency.\n\n- **Highly Efficient Retrieval:** We use LanceDB, a modern, high-performance vector database. On our CPU, retrieving the top-9 examples from our 1 million entry database takes only 40ms on average. This is highly scalable; even if the database were expanded to 10 million entries, the retrieval time would only increase to approximately 100ms. This confirms that retrieval is not a bottleneck.\n\n**3. Regarding Dependency on the Retrieval Corpus and Failure Cases (Weakness 3).**\n\nWe agree that analyzing the performance boundaries is crucial.\n\n- **Generalization to Rare Motions:** The performance on rare motions is not solely dependent on finding a perfect match in the database. During its training on a large and diverse dataset, our CAMA module learns a generalized prior of physical motion and object-scene interactions. This means that for a rare motion, even if the retrieved videos are only loosely related, CAMA can leverage this learned prior to synthesize a plausible motion.\n  \n- **Robustness to Imperfect Retrieval:** As shown in our ablation study (Table 4, \"MCT-Rand-9\"), our CAMA module can distill meaningful motion priors even from randomly selected videos, demonstrating a degree of robustness.\n  \n- **Failure Case Analysis:** The primary failure mode we identified occurs when retrieved videos contain directly opposing motions. For example, when generating a video of \"a person jumping up,\" the retrieval set may contain both the upward and downward phases of a jump. In this case, our CAMA module might average these conflicting motion vectors, resulting in a nearly static output where the person barely moves. This highlights a limitation in resolving contradictory motion priors.\n  \n\n**4. Regarding Open-Sourcing (Weakness 4).**\n\n- **Commitment to Open Source:** We are fully committed to open science. We will make our code and pre-trained model weights publicly available upon publication to ensure full reproducibility.\n\nWe believe that by incorporating these clarifications, additional analyses, and a commitment to releasing our code, we have thoroughly addressed the reviewer's constructive points. We thank the reviewer again for their valuable feedback, which will significantly improve our paper."}, {"review_id": "Bx0Y0VdH5q", "reviewer": "Reviewer_rjfD", "summary": "This paper proposes MotionRAG, a retrieval-augmented framework for video generation that enhances motion realism by transferring motion priors from semantically similar videos retrieved via text prompts.\nExperiments across general and domain-specific datasets (OpenVid-1K and SkillVid) show MotionRAG consistently improves motion realism, semantic alignment, and video quality with minimal computation overhead. Ablations further validate the design.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "stengths:\n1. This paper is well-written and easy to understand.\n2. The paer propose a RAG based video generation method which is interesting and meaningful for video generation community.\n3. The expertiments are extensive enough to demonstrate the performance.\n\nweaknesses:\nOne of the potential weaknesses of the proposed method is its strong dependence on the quality of the retrieved reference videos. It is unclear whether the model will be able to automatically filter or reduce the impact of these sub-optimal references if there are errors in the search results. Although the Motion Context Transformer is designed to have context-aware motion adaptation capabilities, its robustness to low-quality or erroneous search results is not explicitly analyzed in this paper, and mechanisms such as confidence weighting or retrieval screening are not proposed to alleviate this problem. Further exploration of denoising or robustness mechanisms related to retrieval quality will help improve the reliability of the system in practical applications.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer rjfD for the positive and encouraging review. We are delighted that the reviewer found our paper well-written, our approach interesting and meaningful, and our experiments extensive. The reviewer raised a very important point about the system's robustness to sub-optimal retrieval results, which we would like to address comprehensively.\n\n**Regarding the Robustness to Low-Quality or Erroneous Search Results.**\n\nWe agree that robustness to imperfect retrieval is critical for any RAG-based system's practical application. While we did not explicitly frame it as a \"denoising\" mechanism, our Context-Aware Motion Adaptation (CAMA) module is inherently designed to be robust to such scenarios, thanks to the motion priors it learns during training. Our ablation studies provide strong, quantitative evidence for this capability.\n\n1. **CAMA's Inherent Filtering Capability through Learned Priors:** The reviewer correctly identifies that we did not add an explicit filtering or weighting mechanism. This is by design. Our CAMA module, a causal transformer, learns to perform this function implicitly. By training on millions of diverse video-text pairs, CAMA develops a strong internal prior of plausible physical motion. When processing a sequence of multiple retrieved examples, it leverages this prior to identify and amplify common, relevant motion patterns while down-weighting or ignoring outlier features from sub-optimal references. It learns to reason about motion, not just copy it.\n  \n2. **Strongest Evidence: Robustness to Random Retrieval (Worst-Case Noise).** The most compelling evidence of this robustness is presented in our ablation study in Table 4. The \"MCT-Rand-9\" experiment uses 9 completely random videos as context—the worst-case scenario for retrieval quality. Even with this entirely irrelevant context, our method's performance is remarkable.\n  \n  | Method | Action↑ | FVD↓ | FID↓ | CLIP↑ | DINO↑ |\n  | --- | --- | --- | --- | --- | --- |\n  | Baseline | 53.5 | 88.4 | 10.9 | 91.0 | 85.8 |\n  | MCT-9 (Ours) | **62.1** | **69.0** | **9.7** | **92.3** | **88.4** |\n  | MCT-Rand-9 | 60.7 | 77.5 | 10.9 | 91.6 | 87.6 |\n  \n  As the table shows, the Action score for \"MCT-Rand-9\" (60.7) is not only far superior to the baseline (53.5) but is also impressively close to the performance with high-quality retrieval (\"MCT-9\", 62.1). This directly demonstrates that CAMA is not fragile. When faced with erroneous inputs, it falls back on its powerful learned priors about how things should move, ensuring a plausible and high-quality output. This directly addresses the reviewer's core concern.\n  \n\nRegarding the **Paper Formatting Concerns**, we thank the reviewer for pointing this out. The color choice was purely for aesthetic reasons and had no intention of implying any affiliation.\n\nWe are confident that these clarifications, supported by direct quantitative evidence from our experiments, fully address the reviewer's valid concerns and further strengthen the paper. We thank the reviewer again for their positive assessment and constructive feedback."}, {"review_id": "aZtnJA6H6G", "reviewer": "Reviewer_GD23", "summary": "This paper introduces MotionRAG, a retrieval-augmented framework designed to improve motion realism in image-to-video generation. The authors propose several contributions, including Retrieval-based motion adaptation and Seamless motion injection. The method enhances motion realism across diverse domains without retraining, supporting zero-shot generalization by updating the retrieval database.", "questions": "1. Please address the two concerns in the weakness section.\n2. Further disussion between the proposed implicit motion conditioned mechanism and explicit optical-flow-based motion condition mechanism (Motion-I2V and MOFA-Video) would be helpful to understand the contribution of this submission.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The concept that \"motion can be inherently transferred across domains\" is well-justified and intuitive.\n- The authors test their method on three different base models, demonstrating broad applicability.\n\nWeaknesses:\n- The retrieval database and test sets both come from OpenVid-1M and SkillVid, meaning the test videos likely share a similar distribution with the retrieved references. This could inflate performance compared to real-world scenarios where videos may differ significantly from the database. A more convincing evaluation would involve out-of-distribution or in-the-wild videos.\n- Lack of physics-based evaluation: While the abstract and introduction emphasizes physical realism, the paper does not quantitatively assess whether generated videos adhere to physical laws. Recent benchmarks (e.g., \"Do generative video models understand physical principles? arxiv '25\") could provide more rigorous metrics for evaluating motion plausibility and dynamics.", "was_revised": true, "final_justification": "After rebuttal, I lean towards borderline accept for this submission.", "rebuttal": "We sincerely thank Reviewer GD23 for the constructive feedback. We are glad the reviewer found our core concept intuitive and our multi-model validation a key strength. We would like to address the important concerns raised regarding evaluation and comparison to other methods.\n\n**1. Regarding In-Distribution Evaluation and Out-of-Distribution Testing (Weakness 1).**\n\nThis is a very valid point. To rigorously test our method's generalization ability beyond just swapping specialized datasets, we conducted a new **out-of-distribution (OOD) experiment**.\n\n- **New OOD Experiment:** We tested our method on the OpenVid-1K test set while using a completely different, large-scale video understanding dataset, **InternVid-10M**, as our retrieval database. It is crucial to note that InternVid-10M is curated for video *understanding* tasks and has a significantly different data distribution (e.g., content, caption, camera work) compared to video *generation* datasets like OpenVid. The results are as follows (on Dynamicrafter):\n  \n  | Retrieval DB | Action↑ | FVD↓ | FID↓ | CLIP↑ | DINO↑ |\n  | --- | --- | --- | --- | --- | --- |\n  | Baseline (None) | 53.5 | 88.4 | 10.9 | 91.0 | 85.8 |\n  | OpenVid-1M (In-Dist.) | **62.1** | **69.0** | **9.7** | **92.3** | **88.4** |\n  | InternVid-10M (OOD) | 60.9 | 70.7 | 10.5 | 91.7 | 87.4 |\n  \n\n- **Conclusion:** The results clearly show that even when retrieving from a large-scale, out-of-distribution database, our method still achieves a dramatic improvement over the baseline, with performance nearly on par with using an in-distribution database. This provides strong evidence that our method is not simply \"memorizing\" patterns from a specific dataset distribution but is capable of robustly generalizing across diverse data sources. We will add this new experiment and analysis to our paper.\n\n**2. Regarding Physics-Based Evaluation.**\n\nWe agree that quantitative, physics-based evaluation is an important and emerging area.\n\n- **Alignment with Evaluation Principles:** Our evaluation philosophy is conceptually aligned with the one proposed in the benchmark cited by the reviewer (\"Do generative video models understand physical principles?\"): both methodologies compare the generated video to a ground truth (GT) video, operating under the assumption that the GT video is physically plausible.\n- **Suitability of Metrics for Open-Domain Content:** The primary difference lies in the specific metrics used, which reflects the different nature of the evaluation data. The cited work employs low-level metrics like Spatial IoU and MSE, which are highly effective for their dataset of **simple, single-object interactions in controlled settings** (e.g., a ball bouncing). However, these coordinate-based metrics are less suitable for the **complex, open-domain scenarios** in our evaluation, which often feature articulated human/animal motion, multiple interacting objects, and unconstrained camera movement. In such cases, these low-level metrics could penalize a generated video for being physically plausible but not pixel-perfectly identical to the GT (e.g., a slightly different but equally valid running gait).\n- **Our Metrics Capture Semantic Plausibility:** For this reason, we chose a suite of metrics better suited for open-domain content. The Action Score, in particular, uses a high-level action recognition model to assess if the *semantic nature* of the motion is correct (e.g., does it look like \"running\" or \"jumping\"?). The consistent and significant improvements across our entire suite provide strong, holistic evidence of enhanced motion realism.\n\n**3. Regarding Comparison with Explicit Motion-Conditioned Methods.**\n\nThis is an excellent question that helps clarify our contribution.\n\n- **High-Level Semantics vs. Low-Level Pixels:** The key difference is the level of abstraction. Methods like Motion-I2V rely on **low-level, pixel-based motion representations like optical flow**. While precise, optical flow is **tightly coupled with appearance and structure**, making it very difficult to transfer across domains (e.g., the optical flow of a person walking cannot be easily applied to a robot walking). It is also **computationally expensive** to generate.\n- **Our Advantage: Cross-Domain Transfer and Efficiency.** Our approach uses **high-level, semantic motion features** that capture the *essence* of a motion (e.g., the concept of \"walking\"). This abstraction is what enables **robust cross-domain transfer** and is the core reason our method works. Furthermore, our motion feature extraction is **highly efficient**, taking less than one second. We will expand the related work section to explicitly detail these key differences.\n\nWe believe these clarifications, supported by new experimental evidence, thoroughly address the reviewer's concerns. We appreciate the feedback, which has prompted us to further strengthen our paper's evaluation and positioning."}]

Author rebuttal:

Meta-review: All the reviewers unanimously voted for acceptance. After checking the review, the rebuttal, the manuscript, and the discussion, the AC recommended acceptance on the condition that the manuscript would be revised in the camera-ready version by taking the reviewers' suggestions into consideration, such as the experiments on physical constraints, code open source, etc.
Decision: Accept (poster)