Hugging Face's logo

TMElyralab
/
MuseV

Text-to-Video
Diffusers
ONNX
Safetensors
English
AIGC
text2video
image2video
infinite-length
human
Model card Files Community
5
anchor commited on
Commit
dea34a1
•
1 Parent(s): 10cde77

update readme

Browse files
Files changed (1) hide show
  1. README.md +318 -0
README.md CHANGED
@@ -1,3 +1,321 @@
1
  ---
2
  license: creativeml-openrail-m
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: creativeml-openrail-m
3
+ language:
4
+ - en
5
+ library_name: diffusers
6
+ pipeline_tag: text-to-video
7
+ tags:
8
+ - AIGC
9
+ - text2video
10
+ - image2video
11
+ - infinite-length
12
+ - human
13
  ---
14
+
15
+ # MuseV
16
+
17
+ <font size=5>MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Parallel Denoising
18
+ </br>
19
+ Zhiqiang Xia <sup>\*</sup>,
20
+ Zhaokang Chen<sup>\*</sup>,
21
+ Bin Wu<sup>†</sup>,
22
+ Chao Li,
23
+ Kwok-Wai Hung,
24
+ Chao Zhan,
25
+ Wenjiang Zhou
26
+ (<sup>*</sup>co-first author, <sup>†</sup>Corresponding Author)
27
+ </font>
28
+
29
+ **[project](comming soon)** **Technical report (comming soon)**
30
+
31
+
32
+ We have setup the world simulator vision since March 2023, believing diffusion models can simulate the world. `MuseV` was a milestone achieved around July 2023. Amazed by the progress of Sora, we decided to opensource `MuseV`, hopefully it will benefit the community.
33
+
34
+ Our next move will switch to the promising diffusion+transformer scheme. Please stay tuned.
35
+
36
+ We will soon release `MuseTalk`, a diffusion-baesd lip sync model, which can be applied with MuseV as a complete virtual human generation solution. Please stay tuned!
37
+
38
+ # Intro
39
+ `MuseV` is a diffusion-based virtual human video generation framework, which
40
+ 1. supports infinite length generation using a novel Parallel Denoising scheme.
41
+ 2. checkpoint available for virtual human video generation trained on human dataset.
42
+ 3. supports Image2Video, Text2Image2Video, Video2Video.
43
+ 4. compatible with the Stable Diffusion ecosystem, including `base_model`, `lora`, `controlnet`, etc.
44
+ 5. supports multi reference image technology, including `IPAdapter`, `ReferenceOnly`, `ReferenceNet`, `IPAdapterFaceID`.
45
+ 6. training codes (comming very soon).
46
+
47
+
48
+ ## Model
49
+ ### overview of model structure
50
+ ![model_structure](data/models/musev_structure.png)
51
+ ### parallel denoise
52
+ ![parallel_denoise](data/models/parallel_denoise.png)
53
+
54
+ ## Cases
55
+ All frames are generated from text2video model, without any post process.
56
+
57
+ Bellow Case could be found in `configs/tasks/example.yaml`
58
+ ### Text/Image2Video
59
+
60
+ #### Human
61
+ <!-- 2 columns, one image, one video -->
62
+
63
+ <table class="center">
64
+ <tr style="font-weight: bolder;text-align:center;">
65
+ <td>image</td>
66
+ <td>video </td>
67
+ <td>prompt</td>
68
+ </tr>
69
+
70
+ <tr>
71
+ <td>
72
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/cTQX49v7GT7GA-NEHj5vK.jpeg width="250">
73
+ </td>
74
+ <td >
75
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/U4sHVv_fYVbFHveS7Sw7h.mp4"></video>
76
+ </td>)
77
+ <td>(masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
78
+ soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
79
+ </td>
80
+ </tr>
81
+
82
+ <tr>
83
+ <td>
84
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/SPSgJpptVM4Qm11nqD07C.jpeg width="250">
85
+ </td>
86
+ <td>
87
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/XMya1FCmRs6USzKp9qrAy.mp4"></video>
88
+ </td>
89
+ <td>
90
+ (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
91
+ soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
92
+ </td>
93
+ </tr>
94
+ <tr>
95
+ <td>
96
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/l6chBPhUKeOLbnXnX-ewG.jpeg width="250">
97
+ </td>
98
+ <td>
99
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/pPnU6QXgWuWxw5SWZdl8N.mp4"></video>
100
+ </td>
101
+ <td>
102
+ (masterpiece, best quality, highres:1), peaceful beautiful sea scene
103
+ </td>
104
+ </tr>
105
+
106
+ <tr>
107
+ <td>
108
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/xKJnKg7AQEJCTzt4DW4n3.jpeg width="250">
109
+ </td>
110
+ <td>
111
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/KezbR3_yWus5JFJdMlaV7.mp4"></video>
112
+ </td>
113
+ <td>
114
+ (masterpiece, best quality, highres:1), peaceful beautiful sea scene
115
+ </td>
116
+ </tr>
117
+ <tr>
118
+ <td>
119
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/VQMEeGc1wTuiATtQLJjer.jpeg width="250">
120
+ </td>
121
+ <td>
122
+ <video width="200" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/_9ZQEebUlmSNtXMJKiGPu.mp4"></video>
123
+ </td>
124
+ <td>
125
+ (masterpiece, best quality, highres:1), peaceful beautiful sea scene
126
+ </td>
127
+ </tr>
128
+ <!-- guitar -->
129
+ <tr>
130
+ <td>
131
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/Fk_eec7vqq4NfAYVPNLI-.jpeg width="250">
132
+ </td>
133
+ <td>
134
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/p0gWRwZTDOrbPf8mZOphG.mp4"></video>
135
+ </td>
136
+ <td>
137
+ (masterpiece, best quality, highres:1), playing guitar
138
+ </td>
139
+ </tr>
140
+ <tr>
141
+ <td>
142
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/Zcc5xj1-lA_EPS7gvJu99.jpeg width="250">
143
+ </td>
144
+ <td>
145
+ <video width="200" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/4mT4KL3q4FzyQQKJfgXVG.mp4"></video>
146
+ </td>
147
+ <td>
148
+ (masterpiece, best quality, highres:1), playing guitar
149
+ </td>
150
+ </tr>
151
+ <tr>
152
+ <td>
153
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/iT5OsCpRNnntuS0TH1cG5.jpeg width="250">
154
+ </td>
155
+ <td>
156
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/6RWBw73-oE4rJH808FzIK.mp4">
157
+ </td>
158
+ <td>
159
+ (masterpiece, best quality, highres:1), playing guitar
160
+ </td>
161
+ </tr>
162
+ <tr>
163
+ <td>
164
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/Ym582ZF-MbYkRW1sAE5r3.jpeg width="250">
165
+ </td>
166
+ <td>
167
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/lYqpeRcIiK7WEqRe4d8dZ.mp4"></video>
168
+ </td>
169
+ <td>
170
+ (masterpiece, best quality, highres:1), playing guitar
171
+ </td>
172
+ </tr>
173
+ <!-- famous people -->
174
+ <tr>
175
+ <td>
176
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/JsAjbl4AeYz089kWHjjUJ.jpeg width="250">
177
+ </td>
178
+ <td>
179
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/KCjF9HutBo7el15gm3YV3.mp4"></video>
180
+ </td>
181
+ <td>
182
+ (masterpiece, best quality, highres:1),(1man, solo:1),(beautiful face,
183
+ soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
184
+ </td>
185
+ </tr>
186
+
187
+ <tr>
188
+ <td>
189
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/v-X3Wrkm14YwLGGloNlMK.jpeg width="250">
190
+ </td>
191
+ <td>
192
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/P_Y5jUO1EJ6n3Z4qd1xh1.mp4"></video>
193
+ </td>
194
+ <td>
195
+ (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
196
+ soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
197
+ </td>
198
+ </tr>
199
+ <tr>
200
+ <td>
201
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/cNe41NV1OfLF5AmMKD6mi.jpeg width="250">
202
+ </td>
203
+ <td>
204
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/L6mA8uuckRJhzhAJHayJT.mp4"></video>
205
+ </td>
206
+ <td>
207
+ (masterpiece, best quality, highres:1),(1man, solo:1),(beautiful face,
208
+ soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
209
+ </td>
210
+ </tr>
211
+ <tr>
212
+ <td>
213
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/6iHIaa15eBgop7BsE0Nps.jpeg width="250">
214
+ </td>
215
+ <td>
216
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/foPX3iRk2TzjRl_V52T21.mp4"></video>
217
+ </td>
218
+ <td>
219
+ (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
220
+ soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
221
+ </td>
222
+ </tr>
223
+ <tr>
224
+ <td>
225
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/R7rp7t4DPkws0dXRxi0bf.jpeg width="250">
226
+ </td>
227
+ <td>
228
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/tGTSNe9i08pvTMNe6SOBg.mp4"></video>
229
+ </td>
230
+ <td>
231
+ (masterpiece, best quality, highres:1),(1girl, solo:1),(beautiful face,
232
+ soft skin, costume:1),(eye blinks:{eye_blinks_factor}),(head wave:1.3)
233
+ </td>
234
+ </tr>
235
+ </table >
236
+
237
+ #### scene
238
+
239
+ <table class="center">
240
+ <tr style="font-weight: bolder;text-align:center;">
241
+ <td width="35%">image</td>
242
+ <td width="50%">video</td>
243
+ <td width="15%">prompt</td>
244
+ </tr>
245
+
246
+ <tr>
247
+ <td>
248
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/EIembLBwySZTBjFZStFr_.jpeg width="250">
249
+ </td>
250
+ <td>
251
+ <video width="200" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/jfHV6a186BzAu-Tz0ET1o.mp4"></video>
252
+ </td>
253
+ <td>
254
+ (masterpiece, best quality, highres:1), peaceful beautiful waterfall, an
255
+ endless waterfall
256
+ </td>
257
+ </tr>
258
+
259
+ <tr>
260
+ <td>
261
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/u2_mzl5m-Z0nwSYFcTLxs.jpeg width="250">
262
+ </td>
263
+ <td>
264
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/eXrRVejCZs3QaA4-JK6Le.mp4"></video>
265
+ </td>
266
+ <td>(masterpiece, best quality, highres:1), peaceful beautiful river
267
+ </td>
268
+ </tr>
269
+
270
+ <tr>
271
+ <td>
272
+ <img src=https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/NIHfIi7onyJ5ELetE2f_Z.jpeg width="250">
273
+ </td>
274
+ <td>
275
+ <video width="300" controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/65f9352ed760cfdf5eb80e16/eaZyhukcfaoY7dIGp2cs6.mp4"></video>
276
+ </td>
277
+ <td>(masterpiece, best quality, highres:1), peaceful beautiful sea scene
278
+ </td>
279
+ </tr>
280
+ </table >
281
+
282
+ ### VideoMiddle2Video
283
+
284
+
285
+
286
+ # News
287
+ - [03/22/2024] release `MuseV` project and trained model `musev`, `muse_referencenet`.
288
+
289
+
290
+ # Quickstart
291
+ please refer to [MuseV](https://github.com/TMElyralab/MuseV)
292
+
293
+ ### Gradio demo
294
+ MuseV provides gradio script to generate GUI in local machine to generate video conveniently.
295
+
296
+ ```bash
297
+ cd scripts/gradio
298
+ python app.py
299
+ ```
300
+
301
+ # Acknowledgements
302
+
303
+ MuseV builds on `TuneAVideo`, `diffusers`. Thanks for open-sourcing!
304
+
305
+ <!-- # Contribution 暂时不需要组织开源共建 -->
306
+
307
+ # Citation
308
+ **paper comming soon**
309
+ ```bib
310
+ @article{musev,
311
+ title={MuseV: Infinite-length and High Fidelity Virtual Human Video Generation with Parallel Denoising},
312
+ author={Xia, Zhiqiang and Chen, Zhaokang and Wu, Bin and Li, Chao and Hung, Kwok-Wai and Zhan, Chao and Zhou, Wenjiang},
313
+ journal={arxiv},
314
+ year={2024}
315
+ }
316
+ ```
317
+ # Disclaimer/License
318
+ 1. `code`: The code of MuseV is released under the MIT License. There is no limitation for both academic and commercial usage.
319
+ 1. `model`: The trained model are available for non-commercial research purposes only.
320
+ 1. `other opensource model`: Other open-source models used must comply with their license, such as `insightface`, `IP-Adapter`, `ft-mse-vae`, etc.
321
+ 1. `AIGC`: This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.