README.md

metadata

license: apache-2.0

ExVideo

ExVideo is a post-tuning technique aimed at enhancing the capability of video generation models. We have extended CogVideoX-5B to generate videos up to 129 frames long.

This is our second publicly released model, incorporating LoRA into the structure of CogVideoX-5B.

• Project Page
• Source Code
• Technical report

Usages

from diffsynth import ModelManager, CogVideoPipeline, save_video, download_models
import torch


download_models(["CogVideoX-5B", "ExVideo-CogVideoX-LoRA-129f-v1"])
model_manager = ModelManager(torch_dtype=torch.bfloat16)
model_manager.load_models([
    "models/CogVideo/CogVideoX-5b/text_encoder",
    "models/CogVideo/CogVideoX-5b/transformer",
    "models/CogVideo/CogVideoX-5b/vae/diffusion_pytorch_model.safetensors",
])
model_manager.load_lora("models/lora/ExVideo-CogVideoX-LoRA-129f-v1.safetensors")
pipe = CogVideoPipeline.from_model_manager(model_manager)

torch.manual_seed(6)
video = pipe(
    prompt="an astronaut riding a horse on Mars.",
    height=480, width=720, num_frames=129,
    cfg_scale=7.0, num_inference_steps=100,
)
save_video(video, "video_with_lora.mp4", fps=8, quality=5)

Please refer to DiffSynth for more information.

Examples

Prompt: an astronaut riding a horse on Mars.