albertalonghini8/weaver-robolab-51task-10k-left

WEAVER fine-tuned on 51 diverse RoboLab tasks (10k campaign) — model card

Fine-tunes of the released WEAVER action-conditioned multi-view world model on 729 Pi0.5 demos spanning 51 diverse RoboLab tasks (pick-place, bin, plate, stacking, reorientation, tool-use, sorting, food-packing, shelf-placement, cleanup — all 10 benchmark tag categories), collected on a fleet of A100s (plan 009). ~3.5× the count and vastly more task-diverse than the earlier 207-episode weaver-robolab-pi05-ft.

Two variants differing only in which over-shoulder camera is the exterior_1 input:

Repo	exterior_1 camera	Use
…-10k-right	over_shoulder right	general world-model prediction
…-10k-left	over_shoulder left	aligned with the Pi0 policy (its exterior_image_1_left is fed from over_shoulder_left) → the one for policy-in-the-dream / intention work

Results (held-out val, base WEAVER → fine-tuned, 51-task set)

Right (over_shoulder_right):

View	FVD ↓	FID ↓	LPIPS ↓
Exterior	362 → 85 (−77%)	39.5 → 20.6	0.157 → 0.074
Wrist	696 → 309 (−56%)	67.9 → 41.2	0.418 → 0.247

Per-frame (held-out seed): exterior +5.5 dB PSNR / +0.10 SSIM; wrist +5.7 dB / +0.10.

Left (over_shoulder_left — policy-aligned):

View	FVD ↓	FID ↓	LPIPS ↓
Exterior	→ 83.2	→ 22.1	→ 0.076
Wrist	→ 289	→ 36.0	→ 0.256

Left in-loop val at step 16 000 (base-vs-FT panels + per-frame PSNR/SSIM added by the eval pass). The left variant essentially matches the right on exterior fidelity (FVD 83 vs 85) and is the one to use for policy-in-the-dream, since Pi0's exterior_image_1_left is fed from over_shoulder_left.

Training: batch 6, 16 000 steps, single H100, use_compile=False. See plans/009-scale-10k.md, memory/weaver-finetune.md.

Download & use

huggingface-cli login   # + accept the SD3 license (VAE loaded at runtime, not redistributed)
huggingface-cli download <HF_REPO> --local-dir ./weaver-10k
python -m weaver.generate_views --checkpoint ./weaver-10k --output-dir out --split val \
    --num-videos 4 --start-idx 0 --overrides dataset.path=<your_dataset> \
    dataset.norm_stats_path=<your_dataset>/norm_stats_relabel.json dataset.annotation_dir=annotation_rewards

Fine-tune further with PRETRAINED=./weaver-10k scripts/finetune-weaver-singlegpu.sh. Full pipeline (collect → encode → train → eval → dream) in NVLabs RoboLab + this repo's scripts/.

License & attribution

Derived from WEAVER (arnavkj1995 / NVLabs) — respect the upstream license. Checkpoint = fine-tuned FlowWM + reward/critic only; SD3 VAE + CLIP not redistributed (obtain from their gated sources).