Viggle-Animate

Simple, fast, accurate, high-quality character animation.

Repaint one frame. The model propagates that edit across the whole clip, in 4 sampling steps.

_{30 s of renders from this model — one take each, --steps 4, no pose estimation and no masks. Unmute for sound.}

Left panel is the driving video, right panel is this model at --steps 4. The only other input is that same clip's own first frame with the character painted over — same camera, same pose, same background, same light.

Try it in your browser → — upload a clip and a character still; the Space repaints frame 0 for you and renders. No install, no GPU.

26 s

per render
124 frames at 24 fps on one B200 · 4 sampling steps, 3 forward passes

2

inputs
a driving clip and its own first frame, repainted · no pose, mask, depth or prompt

anything

you can paint
animals, machines, flat 2D, two at once — the model is never told what it is

1

take each
every clip on this page is one paint, one render, --steps 4 --seed 42 · no best-of-N

driving.mp4
the performance

frame 0 →

any image editor
gpt-image-2 · not this model

→

ref.png
frame 0, repainted

driving.mp4  +  ref.png
↓

Viggle-Animate
33.1 B ref2va full finetune  +  DMD2 LoRA, rank 128
4 sampling steps · 3 forward passes · frozen text embedding, encoder never loads

↓

out.mp4
same length, same camera, new character

Never computed: pose skeleton · segmentation mask · face crops · background plate · depth · text prompt

python inference/sample.py --cond driving.mp4 --ref repainted_first_frame.png --out out.mp4

The repaint comes from an image editor, not from this model — these used gpt-image-2. This model is the second half of that pipeline, not the whole of it.

Compared to Wan2.2-Animate-14B

One B200, 480×832, 124 frames at 24 fps, bf16, no compile, no offload. Wan ran its documented replacement recipe — 20 steps, sample_shift 5.0, --refert_num 1 --replace_flag --use_relighting_lora, --w_len 1 --h_len 1 — after its own preprocessing pass.

	Viggle-Animate	Wan2.2-Animate-14B
Inputs	driving video + one repainted frame	driving video + character image, then a preprocessing pass producing pose, face, mask and background tracks
Render, after weights load	26 s	160 s
— of which sampling	13.6 s	140 s
Forward passes	3	40 (20 steps × 2 chunks)
Parameters	33.1 B	17.3 B

6.1× faster per render, 10.3× on sampling alone, on a model with 1.9× the parameters. Wan's preprocessing pass is not counted in its 160 s.

Four panels each: painted reference · driving video · this model · Wan2.2-Animate-14B.

What you can paint in

The model is never told what the new character is — it sees one edited frame and carries it through five seconds of motion it did not choreograph. So the question is how far from a person that character can get. Three panels each: painted reference · driving video · this model.

Animals. Ears, eye patches and flippers move on limbs the driving clip does not have — the paint places them, the render animates them as if they were arms and a head.

Not humanoid. The airliner is the hardest case: the paint binds wings to arms and landing gear to legs, and the model only has to keep that binding. The robot has to relight specular metal as it turns.

Stylised. Plasticine figure, photographic kitchen — the style boundary holds for the whole clip.

Two characters, one edit. The paint prompt has to bind each character to a position — "the one on the left" — or the editor swaps one and leaves the other. All four held, including the wide arena shot where each figure is a few dozen pixels tall: landscape framing and two subjects are both outside the training distribution.

What this is

Two parts, both derived from MiniMaxAI/MiniMax-H3's ref2va transformer:

transformer/	33.1 B, bf16, 14 shards. A full finetune of the base transformer_ref on a character-replacement objective
lora/	rank 128 over 302 linear layers, 2.5 GB. A DMD2-distilled delta on that finetune — this is what turns 30 sampling steps into 4

The LoRA is a delta on the finetuned transformer — loading it onto stock transformer_ref produces garbage. The frozen conditioning embedding is assets/fixed_prompt.txt.

Quickstart

This repository ships only the transformer and the LoRA — the VAE, audio VAE and schedulers load from your own copy of the base model. Inference touches 11 GB of its 269 GB:

hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \
  --include "modular_model_index.json" "vae/*" "audio_vae/*" \
            "scheduler/*" "audio_scheduler/*" "assets/ref2va.mp4"

hf download Viggle/Viggle-Animate --local-dir ./Viggle-Animate

pip install torch "git+https://github.com/huggingface/diffusers@d6726f3" av

python Viggle-Animate/inference/sample.py \
  --model-dir ./MiniMax-H3 \
  --cond driving.mp4 --ref repainted_first_frame.png --out swapped.mp4

d6726f3 is the tested diffusers commit; the upstream minimax_h3 modular pipeline is enough, no fork or patch. The last --include is the clip examples/demo.sh needs.

One 80 GB card is not enough at bf16 — the transformer is 62 GiB resident and a 480×832 / 124-frame render peaks at 80.1 GiB allocated. Use a card with ≥ 96 GB, or pass --offload to stream blocks from CPU (~12 GB resident, much slower).

It also runs quantized on consumer hardware. We deploy it on a single RTX 5090 (32 GB): NVFP4 weights — 4.5 bits/param, dispatching to the real sm_120 cutlass block-scaled kernel, 2.70× bf16 per compiled linear — plus a low-rank adaln_proj and torch.compile. Quantization alone is not enough for 32 GB: 13.0 B of the 33.1 B parameters sit in adaln_proj, which the linear-layer quantizer does not touch. That deployment path is not shipped in this repository.

Defaults are the evaluated configuration: --steps 4 --flow-shift 3 --num-frames 124 (≈ 5.2 s at 24 fps) --seed 42. Output geometry follows the driving clip and must be a multiple of 32 on both axes. Weights load in ~21 s, once per process.

Make the reference with ffmpeg -i driving.mp4 -frames:v 1 first_frame.png, then edit that frame at the same resolution. Name the change, and pin down what must not change — pose, hands, props, framing, background, light. An editor that quietly reframes the shot will fight the driving motion. examples/demo.sh reproduces the clip above and needs no media from you.

Limitations

• A character photograph is not a reference, and the failure is silent. Pass a standalone portrait — different scene, pose, background — and the model swaps nobody: it reconstructs the driving clip almost unchanged and reports no error. When reference and driving video disagree, the driving video wins.
• Everything you want must survive the paint step. The model will not invent a costume detail, a prop or a face the reference does not show.
• More steps is not better. --steps 4 is the operating point, not a degraded --steps 12. It renders sharper than its teacher (Laplacian variance 16–50 % higher) — detail sometimes, over-sharpening other times, worse the further you push it.
• The paint carries appearance; the driving video carries geometry. Two consequences. A view the reference never shows is filled from the model's prior: a cat painted from behind is a cat while the skater's back is turned, a generic woman at the frame she faces camera, a cat again after. And a silhouette the driving pose cannot explain survives as colour only — a LEGO minifigure kept its palette and yellow claw hands but reverted to human anatomy, while the airliner held because the paint bound its wings to the arms. Prefer clips that keep one side to camera, and bind new limbs to real ones.
• No facial-landmark channel, so tight close-up dialogue is the weakest case — Wan drives the face from explicit landmarks and is visibly more expressive there.
• Out of distribution is not the same as broken. Training was one subject filling a portrait frame. The last four gallery clips are two subjects in landscape framing, including a wide arena shot, and they hold — but that is untuned territory with no evaluation behind it, and crowds remain untested.
• The driving clip's audio is dropped at input; the model emits its own track, and the fixed prompt asks for silence.

License

The weights are a Model Derivative of MiniMax H3, so the MiniMax H3 Community License applies to them — read it before you redistribute them or ship a product on them. Our changes are listed in MODIFICATIONS.md.

The code in inference/ and examples/ is Apache 2.0 (LICENSE-CODE).

Music in the teaser at the top of this page: "Electrodoodle" by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0.

Intended use

This model exists to put a consenting performer into footage they did not shoot, and it will just as readily put someone into footage they never agreed to appear in. Note where that decision is made: the identity comes from the frame you paint, so an image editor's safeguards are upstream of this model and none of them are in it. It cannot verify identity or consent. Do not run it on people who have not agreed to it, label what you generate as AI-generated, and see Section V.5 of the Agreement if you offer this as a service.

Powered by MiniMax H3.