Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,50 @@
|
|
1 |
-
---
|
2 |
-
license: apache-2.0
|
3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
---
|
4 |
+
---
|
5 |
+
license: apache-2.0
|
6 |
+
---
|
7 |
+
# VFIMamba: Video Frame Interpolation with State Space Models
|
8 |
+
This is the official checkpoint library for [VFIMamba: Video Frame Interpolation with State Space Models](https://arxiv.org/abs/2407.02315).
|
9 |
+
Please refer to [this repository](https://github.com/MCG-NJU/VFIMamba) for our code.
|
10 |
+
|
11 |
+
## Model Description
|
12 |
+
VFIMamba is the first approach to adapt the SSM model to the video frame interpolation task.
|
13 |
+
1. We devise the Mixed-SSM Block (MSB) for efficient inter-frame modeling using S6.
|
14 |
+
2. We explore various rearrangement methods to convert two frames into a sequence, discovering that interleaved rearrangement is more suitable for VFI tasks.
|
15 |
+
3. We propose a curriculum learning strategy to further leverage the potential of the S6 model.
|
16 |
+
|
17 |
+
Experimental results demonstrate that VFIMamba achieves the state-of-the-art performance across various datasets, in particular highlighting the potential of the SSM model for VFI tasks with high resolution.
|
18 |
+
|
19 |
+
## Usage
|
20 |
+
We provide two models, an efficient version (VFIMamba-S) and a stronger one (VFIMamba). You can choose what you need by specifying the parameter model.
|
21 |
+
|
22 |
+
### Manually Load
|
23 |
+
Please refer to [the instruction here](https://github.com/MCG-NJU/VFIMamba/tree/main?tab=readme-ov-file#sunglassesplay-with-demos) for manually loading the checkpoints and a more customized experience.
|
24 |
+
```bash
|
25 |
+
python demo_2x.py --model **model[VFIMamba_S/VFIMamba]** # for 2x interpolation
|
26 |
+
python demo_Nx.py --n 8 --model **model[VFIMamba_S/VFIMamba]** # for 8x interpolation
|
27 |
+
```
|
28 |
+
|
29 |
+
|
30 |
+
### Hugging Face Demo
|
31 |
+
For Hugging Face demo, please refer to [the code here](https://github.com/MCG-NJU/VFIMamba/blob/main/hf_demo_2x.py).
|
32 |
+
```bash
|
33 |
+
python hf_demo_2x.py --model **model[VFIMamba_S/VFIMamba]** # for 2x interpolation
|
34 |
+
```
|
35 |
+
|
36 |
+
|
37 |
+
## Citation
|
38 |
+
If you think this project is helpful in your research or for application, please feel free to leave a star⭐️ and cite our paper:
|
39 |
+
```
|
40 |
+
@misc{zhang2024vfimambavideoframeinterpolation,
|
41 |
+
title={VFIMamba: Video Frame Interpolation with State Space Models},
|
42 |
+
author={Guozhen Zhang and Chunxu Liu and Yutao Cui and Xiaotong Zhao and Kai Ma and Limin Wang},
|
43 |
+
year={2024},
|
44 |
+
eprint={2407.02315},
|
45 |
+
archivePrefix={arXiv},
|
46 |
+
primaryClass={cs.CV},
|
47 |
+
url={https://arxiv.org/abs/2407.02315},
|
48 |
+
}
|
49 |
+
```
|
50 |
+
|