anywaylabs/Wan2.1-VACE-14B-adapter-sft

Wan2.1-VACE-14B Adapter SFT — AnywayLabs

Private VACE adapter weights (the pipe.vace sub-network only, ~3B parameters). The 14B DiT backbone is frozen. Load on top of the public Wan-AI/Wan2.1-VACE-14B checkpoint.

Checkpoint keys have the pipe.vace. prefix stripped so they can be merged into pipe.vace directly via load_state_dict(..., strict=False).

---

Training task

RGB dashcam render → India-city photorealistic video.
The control input (vace_video) is a synthetic RGB driving clip from the Cosmos Drive Dreams pipeline. The target output is a photorealistic Indian highway/city re-render from the same viewpoint. No reference image is used; condition is control video only.

Dataset: private anywaylabs/cosmos-synthetic-200 — 200 paired clips (synthetic RGB control + Indian-city GT + per-clip captions).

---

Files

Run 1 — 17-frame SFT (step-3500)

Path	Notes
step-3500.safetensors	17-frame run, end of training — 20 epochs, 3 500 optimizer steps
training_args.json	Trainer config for this run

Run 2 — 117-frame SFT (117frames/)

Path	Notes
117frames/step-500.safetensors	117-frame run, end of training — 20 epochs, 500 optimizer steps (recommended)
117frames/step-100.safetensors	117-frame run, epoch 4 — lowest single-step validation loss (0.018); useful for comparison
117frames/training_args.json	Trainer config for this run
117frames/loss.csv	Per-step loss log for the full 117-frame run

---

Training details

Run 1 — 17 frames

Param	Value
Geometry	480 × 832, 17 frames (4k+1)
Control	RGB vace_video; no reference image
Dataset	1 400 non-overlapping 17-frame windows from 200 clips
Hardware	8 × A100 80 GB
Training	DeepSpeed ZeRO-2 + gradient-checkpointing offload, global batch 8
Optimizer	AdamW, LR 5e-5, weight decay 0.1
Epochs / steps	20 epochs, 3 500 steps (save every 175)
Trainable	vace only
Trainer	DiffSynth-Studio

Run 2 — 117 frames

Param	Value
Geometry	480 × 832, 117 frames (4k+1 ≈ 4.9 s @ 24 fps)
Control	Horizontally-flipped RGB vace_video; no reference image
Dataset	200 non-overlapping 117-frame windows (one per clip)
Hardware	8 × A100 80 GB
Training	DeepSpeed ZeRO-2 + gradient-checkpointing offload, global batch 8
Optimizer	AdamW, LR 5e-5, weight decay 0.1
Epochs / steps	20 epochs, 500 steps (save every 25)
Trainable	vace only
Trainer	DiffSynth-Studio

Loss summary (117-frame run):
Epoch means range 0.083–0.122 across all 20 epochs. The run trains stably with no divergence. step-500 (epoch 20 final) is the recommended checkpoint. step-100 (epoch 4) recorded the single lowest step loss (0.018) and is included for ablation.

---

Inference (DiffSynth)

import torch
from diffsynth.core import load_state_dict
from diffsynth.pipelines.wan_video import ModelConfig, WanVideoPipeline
from diffsynth.utils.data import VideoData, save_video

# Load base model
pipe = WanVideoPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="Wan-AI/Wan2.1-VACE-14B", origin_file_pattern="diffusion_pytorch_model*.safetensors"),
        ModelConfig(model_id="Wan-AI/Wan2.1-VACE-14B", origin_file_pattern="models_t5_umt5-xxl-enc-bf16.pth"),
        ModelConfig(model_id="Wan-AI/Wan2.1-VACE-14B", origin_file_pattern="Wan2.1_VAE.pth"),
    ],
    tokenizer_config=ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="google/umt5-xxl/"),
)

# Merge adapter (17-frame or 117-frame checkpoint)
ckpt_path = "path/to/step-3500.safetensors"  # or 117frames/step-500.safetensors
missing, unexpected = pipe.vace.load_state_dict(load_state_dict(ckpt_path), strict=False)

# Run
control = VideoData("rgb_control.mp4", height=480, width=832)
vace_video = [control[i] for i in range(17)]   # or 117 for the 117-frame adapter
video = pipe(
    prompt="A car drives along a busy Indian highway at dusk, realistic, high quality",
    negative_prompt="...",
    vace_video=vace_video,
    num_frames=17,    # match the adapter's training frame count
    height=480,
    width=832,
    seed=1,
    tiled=True,
)
save_video(video, "output.mp4", fps=24)

Note on frame counts: The 17-frame adapter was trained with num_frames=17; the 117-frame adapter with num_frames=117. Use the matching count at inference time for best results. Wan requires num_frames = 4k+1.