LiveWan: streaming text 2 video, 3000 steps

Unofficial community project. Not affiliated with, endorsed by, or produced by Alibaba Group or the Wan-Video team. Built on their Apache-2.0 Wan2.1 release.

A 1.3 B student distilled from a Wan2.1-14B teacher that generates video continuously rather than as a fixed clip: 750 ms of 640x368 video per block, generated in 428 ms on one A100 80GB, extended block by block for as long as you let it run. The text conditioning can be changed mid-stream without clearing the K/V cache, so the scene continues instead of cutting.

Code, setup and documentation: JonathanColetti/LiveWan. This repository holds weights and data only. It is not usable on its own.

Not audio

The model is text-to-video and was never trained to articulate speech. There is no audio, no lip sync and no speech anywhere in this project. Text steers the scene. A face that appears to talk is generating plausible mouth motion, not saying your sentence.

Files

To run the model — 6.0 GB. This is what setup.sh fetches.

path	size	what
checkpoints/t14b_b64/milestones/step002250_noema.pt	5.3 GB	the model — 825 tensors, bf16, no EMA copy
checkpoints/t14b_b64/history.json	100 KB	the full training history of the run
data/prompts.pt	388 MB	umt5-xxl embeddings for 96 prompts, indices 0–95
out/world_p{0,44,60,82}.pt	4 × 59 MB	the four cached evaluation worlds — skip base-model generation entirely
samples/	16 MB	reference clips and analysis filmstrips from those worlds

To continue the run Not needed for inference. do not download these unless you intend to train.

path	size	what
checkpoints/t14b_b64/latest.pt	10.6 GB	step 3000, carries its EMA copy. The comparison arm, and the weights a resume starts from
checkpoints/t14b_b64/opt_rank00{0..7}.pt	8 × 2.2 GB	FSDP-sharded AdamW state at step 3000, one shard per rank

Nothing here duplicates a public download. The Wan2.1 base model, its VAE and the umt5-xxl encoder are not mirrored in this repository. setup.sh pulls Wan-AI/Wan2.1-T2V-1.3B (17 GB) from its own repo, and the 14B teacher is only needed if you are distilling.

The layout mirrors the code repository, so setup.sh downloads straight into place. To take only part of it:

hf download JonathanColetti/LiveWan --local-dir . \
  --exclude "checkpoints/t14b_b64/latest.pt" "checkpoints/t14b_b64/opt_rank*"

Use

git clone https://github.com/JonathanColetti/LiveWan && cd LiveWan
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
./setup.sh                       # pulls this repo, checksum-verified

python scripts/demo.py \
  --weights checkpoints/t14b_b64/milestones/step002250_noema.pt \
  --world-cache out/world_p60.pt --prompt-idx 60 \
  --latent-norm 1.0 --block 3 --steps 2 --window 6 --units 60 --fps 16 \
  --out out/demo_w60

Needs ~10 GiB of VRAM. scripts/live_demo.py serves the same model as a steerable browser stream.

--prompt-idx must match the world: world_p60.pt goes with --prompt-idx 60. Indices run 0–95; anything above fails.

Training

3000 iterations of SF-DMD distillation from a Wan2.1-T2V-14B teacher into a Wan2.1-T2V-1.3B student, effective batch 64 (8×H200, accum 8, FSDP), 41.6 hours at 66.5 s/it, zero interventions. Losses do not decrease in this trainer and should not (the critic is retrained every step, so the generator holds position against a strengthening opponent).

Continuing the run

The optimizer shards make extending the run a resume rather than a restart. Put latest.pt, all eight opt_rank*.pt and history.json in the same run directory and:

ITERS=4000 ./run_b64_resume.sh

Four things decide whether this works:

• Exactly 8 ranks. These are per-rank FSDP shards, not a consolidated state dict. World size 8 is what wrote them and world size 8 is what can read them; fewer ranks means larger shards, not smaller, so there is no repacking shortcut. run_b64_resume.sh checks for all eight up front and refuses rather than half-starting.
• You resume from step 3000, not from the released step 2250. Optimizer state and the EMA copy exist only for step 3000 — 2250 was archived _noema and its EMA is gone. The released checkpoint is the better model; latest.pt is the only resumable one.
• --resume overwrites the checkpoint it resumed from. It reads OUT/latest.pt and writes that same path at the first save. run_b64_resume.sh preserves it as step003000.pt first — if you drive the trainer directly instead, copy it aside yourself.
• ~30 GiB must be free transiently at each save, because the shards are written as .tmp and renamed, so old and new coexist. Prune before the save, not after; a post-save janitor cannot help, since the save is what runs out of space.

The resume path is verified rather than assumed: killing a run at step 12 brought AdamW's internal counters back as 16/18 rather than reset, and the real 750→3000 resume held loss_gen, loss_critic, dmd_grad_norm and gn_gen within one standard deviation of the 60-point pre-crash reference. A subtly broken restore shows up as a shifted mean or changed variance; neither appeared.

Verified

• all 825 tensors load, zero non-finite values, weight norms 0.018–115.4
• three correctness gates pass with their deliberately broken controls failing, both before and after training
• 24 streaming cells across every arm and world met the real-time budget
• same box, same seed → byte-identical output

Limits

• Quality holds for roughly a minute. By 160 s sharpness has fallen to ~73% of the world's.
• Maximum 1024 latent frames (4.3 minutes at block 3), set by WanModel.freqs.
• The pixel check has not been done. Checkpoint selection rests on proxy metrics, and in this project nine automatic measurements have pointed the wrong way — every one caught by looking at pixels at 1:1.
• World 82 is unexplained: every trained checkpoint drifts on it while the untrained baseline sits at −0.001.
• The batch-64 premise is unsettled — confounded with having simply trained longer.

Licence

Apache-2.0. Derived from Wan2.1 (Apache-2.0, Alibaba Group).