MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation
Authors: name: Chenhui Zhu, name: Yilu Wu, name: Shuai Wang, name: Gangshan Wu, name: Limin Wang

Image-to-video generation has made remarkable progress with the advancements
in diffusion models, yet generating videos with realistic motion remains highly
challenging. This difficulty arises from the complexity of accurately modeling
motion, which involves capturing physical constraints, object interactions, and
domain-specific dynamics that are not easily generalized across diverse
scenarios. To address this, we propose MotionRAG, a retrieval-augmented
framework that enhances motion realism by adapting motion priors from relevant
reference videos through Context-Aware Motion Adaptation (CAMA). The key
technical innovations include: (i) a retrieval-based pipeline extracting
high-level motion features using video encoder and specialized resamplers to
distill semantic motion representations; (ii) an in-context learning approach
for motion adaptation implemented through a causal transformer architecture;
(iii) an attention-based motion injection adapter that seamlessly integrates
transferred motion features into pretrained video diffusion models. Extensive
experiments demonstrate that our method achieves significant improvements
across multiple domains and various base models, all with negligible
computational overhead during inference. Furthermore, our modular design
enables zero-shot generalization to new domains by simply updating the
retrieval database without retraining any components. This research enhances
the core capability of video generation systems by enabling the effective
retrieval and transfer of motion priors, facilitating the synthesis of
realistic motion dynamics.