ali-vilab

/

text-to-video-ms-1.7b

like 482

Here are code blocks with small tweaks that will allow you to use the model without any problems.

IMPORTANT NOTE: Keep Setup and Generation in separate blocks, and run Setup only once. Otherwise Google Colab will load model with each run and crash the instance after two-three times.

• First example from documentation, only generates videos up to 2 seconds

[1] Setup

!pip install diffusers transformers accelerate

import torch
from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.utils import export_to_video

pipe = DiffusionPipeline.from_pretrained("damo-vilab/text-to-video-ms-1.7b", torch_dtype=torch.float16, variant="fp16")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

[2] Generation

prompt = "Spiderman is surfing"
video_frames = pipe(prompt, num_inference_steps=25).frames
video_path = export_to_video(video_frames)
video_name = video_path.replace('/tmp/', '')
print('Name:', video_name)
torch.cuda.empty_cache()

• Second example, generates videos with custom length

[1] Setup

!pip install diffusers transformers accelerate

import torch
from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.utils import export_to_video

# load pipeline
pipe = DiffusionPipeline.from_pretrained("damo-vilab/text-to-video-ms-1.7b", torch_dtype=torch.float16, variant="fp16")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

# optimize for GPU memory
pipe.enable_model_cpu_offload()
pipe.enable_vae_slicing()

[2] Generation

prompt = "Spiderman is surfing. Darth Vader is also surfing and following Spiderman"
video_frames = pipe(prompt, num_inference_steps=25, num_frames=200).frames
video_path = export_to_video(video_frames)
video_name = video_path.replace('/tmp/', '')
print('Name:', video_name)
torch.cuda.empty_cache()

num_frames is variable responsible for video length (1 = 0.123 second)

How to access generated videos:

• Select folder icon on left pane (4th icon)
• Double click on two dots above sample_data folder
• Find tmp folder and expand it, this is where the videos are saved by default
  All video file names start with tmp and appear on the bottom of file list. Just right click and download or refresh if you don't see it (folder with refresh icon above the file view)

Tweaks I've added on top of code from documentation:

• Printing out the name in clean way
  video_name = video_path.replace('/tmp/', '')
print('Name:', video_name)
• Flushing CUDA cache to prevent it from clogging after couple of uses
  torch.cuda.empty_cache()