README.md

---
license: apache-2.0
language:
- en
---

# MotionCtrl Model Card

<div align="center">

[**Project Page**](https://wzhouxiff.github.io/projects/MotionCtrl/) **|** [**Paper (ArXiv)**](https://arxiv.org/pdf/2312.03641.pdf) **|** [**Code**](https://github.com/TencentARC/MotionCtrl)

[🤗 **Gradio demo (MotionCtrl+VideoCrafter)**](https://huggingface.co/spaces/TencentARC/MotionCtrl) **|** [🤗 **Gradio demo (MotionCtrl+SVD)**](https://huggingface.co/spaces/TencentARC/MotionCtrl_SVD)

</div>

## Introduction

MotionCtrl, when provided with specific camera movements indicated by camera poses or several trajectories, 
can independently control both the **camera** and **object** motion in a generated video. 
It can be deployed on both Text-to-Video generation models, such as **LVDM/VideoCrafter** and **AnimateDiff**, 
as well as Image-to-Video generation models like **SVD**.

## MotionCtrl + SVD

- Model: [motionctrl.pth](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl.pth)
- Results: 

<div align="center">
    <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/00_ibzz5-dxv2h.gif", width="49.7%", height="100%", style="float: left"> 
    <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/svd/01_5guvn-0x6v2.gif", width="49.7%", height="100%", style="float: right">
</div>

&ensp;

## MotionCtrl + LVDM/VideoCrafter

- Model:
  - [motionctrl_svd.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt)
  - [motionctrl_videocrafter2_cmcm.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_videocrafter2_cmcm.ckpt)
- Results:

<div align="center">
    <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/hpxvu-3d8ym.gif", width="100%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/w3nb7-9vz5t.gif", width="100%", height="100%", style="float: right">
    <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/62n2a-wuvsw.gif", width="100%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/TencentARC/MotionCtrl/main/assets/ilw96-ak827.gif", width="100%", height="100%", style="float: right">
</div>

&ensp;

## MotionCtrl + AnimateDiff

- Model:
  - [motionctrl_animatediff_cmcm.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_animatediff_cmcm.ckpt)
  - [motionctrl_animatediff_omcm.ckpt](https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_animatediff_omcm.ckpt)
- Results:

<div align="center">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.png", width="25%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/1424acd0007d40b5.gif", width="25%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/camera_Round-R_ZoomIn.png", width="25%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/complex/round_in.gif", width="25%", height="100%", style="float: left">
</div>

<div align="center">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/traj_curve_1.gif", width="25%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_1.gif", width="25%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1_2.gif", width="25%", height="100%", style="float: left">
    <img src="https://raw.githubusercontent.com/wzhouxiff/wzhouxiff.github.io/master/projects/MotionCtrl/assets/videos/ours_animatediff/traj/curve_1.gif", width="25%", height="100%", style="float: left">
</div>

&ensp;

## Usage

- Download directly in this repository OR Download with python script

```python
from huggingface_hub import hf_hub_download
motionctrl_lvdm_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl.ckpt", repo_type="model")
motionctrl_svd_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl_svd.ckpt", repo_type="model")
```

- Then generate the controllable video following the instructions in our [GitHub repository](https://github.com/TencentARC/MotionCtrl/).

## Citation

```bibtex
@inproceedings{wang2023motionctrl,
  title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
  booktitle={arXiv preprint arXiv:2312.03641},
  year={2023}
}
```