File size: 3,173 Bytes
920184b c902eba 920184b 50f6abe 920184b 50f6abe 920184b 50f6abe 920184b 50f6abe 920184b 50f6abe 920184b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
---
license: creativeml-openrail-m
base_model: nitrosocke/mo-di-diffusion
training_prompt: A bear is playing guitar.
tags:
- tune-a-video
- text-to-video
- diffusers
inference: false
---
# Tune-A-Video - Modern Disney
## Model Description
This is a diffusers compatible checkpoint. When used with DiffusionPipeline, returns an instance of TuneAVideoPipeline
>df-cpt is used to indicate that its a diffusers compatible equivalent of Tune-A-Video-library/mo-di-bear-guitar .
- Base model: [nitrosocke/mo-di-diffusion](https://huggingface.co/nitrosocke/mo-di-diffusion)
- Training prompt: a bear is playing guitar.
## Samples
Test prompt: "A princess playing a guitar, modern disney style"
## Usage
### Loading with a pre-existing Text2Image checkpoint
```python
import torch
from diffusers import TuneAVideoPipeline, DDIMScheduler, UNet3DConditionModel
from diffusers.utils import export_to_video
from PIL import Image
# Use any pretrained Text2Image checkpoint based on stable diffusion
pretrained_model_path = "nitrosocke/mo-di-diffusion"
unet = UNet3DConditionModel.from_pretrained(
"Tune-A-Video-library/df-cpt-mo-di-bear-guitar", subfolder="unet", torch_dtype=torch.float16
).to("cuda")
pipe = TuneAVideoPipeline.from_pretrained(pretrained_model_path, unet=unet, torch_dtype=torch.float16).to("cuda")
prompt = "A princess playing a guitar, modern disney style"
generator = torch.Generator(device="cuda").manual_seed(42)
video_frames = pipe(prompt, video_length=3, generator=generator, num_inference_steps=50, output_type="np").frames
# Saving to gif.
pil_frames = [Image.fromarray(frame) for frame in video_frames]
duration = len(pil_frames) / 8
pil_frames[0].save(
"animation.gif",
save_all=True,
append_images=pil_frames[1:], # append rest of the images
duration=duration * 1000, # in milliseconds
loop=0,
)
# Saving to video
video_path = export_to_video(video_frames)
```
### Loading a saved Tune-A-Video checkpoint
```python
import torch
from diffusers import DiffusionPipeline, DDIMScheduler
from diffusers.utils import export_to_video
from PIL import Image
pipe = DiffusionPipeline.from_pretrained(
"Tune-A-Video-library/df-cpt-mo-di-bear-guitar", torch_dtype=torch.float16
).to("cuda")
prompt = "A princess playing a guitar, modern disney style"
generator = torch.Generator(device="cuda").manual_seed(42)
video_frames = pipe(prompt, video_length=3, generator=generator, num_inference_steps=50, output_type="np").frames
# Saving to gif.
pil_frames = [Image.fromarray(frame) for frame in video_frames]
duration = len(pil_frames) / 8
pil_frames[0].save(
"animation.gif",
save_all=True,
append_images=pil_frames[1:], # append rest of the images
duration=duration * 1000, # in milliseconds
loop=0,
)
# Saving to video
video_path = export_to_video(video_frames)
```
## Related Papers:
- [Tune-A-Video](https://arxiv.org/abs/2212.11565): One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
- [Stable Diffusion](https://arxiv.org/abs/2112.10752): High-Resolution Image Synthesis with Latent Diffusion Models
|