Dreamer-VLA/dreamer-vla-extra-backbones-libero

Extra encoder backbones for the dreamer-vla cross-architecture comparison

Three additional encoder backbones finetuned on LIBERO with the InvDynAuxHead inverse-dynamics auxiliary loss (λ=1), used in Section 3.9 of the dreamer-vla paper.

These are companion ckpts to:

• Scuttie/dreamer-vla-libero-vjepa2-aux — V-JEPA 2 (paper main)
• Scuttie/dreamer-vla-dreamer4-libero — Dreamer V4 (negative result)
• Scuttie/dreamer-vla-lapa-aux-libero — LAPA

Cross-architecture probe table (LIBERO task-OOD, mean of 3 seeds)

Backbone	params	frozen R²	+ aux R²	lift
V-JEPA 2 ViT-L	304M	+0.40	+0.85	+0.45
V-JEPA 2.1 ViT-B	87M	+0.44	+0.82	+0.38
DIFF (Stage A best)	91M	—	+0.57	—
LAPA	344M	+0.41	+0.51	+0.10
SigLIP 2 ViT-L	316M	+0.045	+0.17	+0.12
Web-DINO ViT-L	304M	−0.007	+0.16	+0.17
SDXL VAE	34M	−0.55	−0.41	+0.14
Dreamer 4 (any variant)	64-276M	−0.04	−0.04	0

Mechanism hierarchy (paper finding)

The 8-backbone matrix lands in 4 tiers:

1. Video-temporal masked latent prediction (V-JEPA family) — +0.82-0.85
2. Action-aware / pixel-temporal (LAPA, DIFF) — +0.51-0.57
3. Image-only semantic pretraining (Web-DINO, SigLIP 2) — +0.16-0.17
4. Reconstruction / one-step world model (SDXL VAE, Dreamer 4) — −0.04 to −0.41

Image-level semantic pretraining (3) plateaus at +0.17 regardless of training data scale, contrastive vs masked, or vision-only vs vision-language. This isolates video-temporal masked latent prediction (V-JEPA's specific objective) as the dominant lever — not "large-scale pretraining" generically, not "masked prediction" generically, not "self-supervised learning" generically.

Files

Subdir	Description	Size	Test R²
webdino_aux/	Web-DINO ViT-L (DINOv2 arch, 304M) unfrozen + InvDynAuxHead λ=1 finetune	0.61GB	+0.159
siglip2_aux/	SigLIP 2 ViT-L vision tower (316M) unfrozen + InvDynAuxHead λ=1 finetune	0.64GB	+0.166
sdxl_vae_aux/	SDXL VAE encoder (34M) unfrozen + InvDynAuxHead λ=1 finetune (proxy for SD3 VAE; gated)	0.17GB	-0.407

Training recipe (shared)

• Loss: λ * MSE(InvDynAuxHead(features), action[:7]), λ=1.0 (V-JEPA recipe parity)
• aux head: [2D → 512 → 256 → 7] GELU, trained jointly with encoder
• Encoder unfrozen (full finetune)
• 20k steps, batch 4 per GPU × 2 GPUs DDP = effective batch 8 (SDXL VAE used batch 8 per GPU × 2 = effective 16 — encoder is small)
• AdamW lr=1e-4, cosine schedule with 2k warmup, weight_decay=0.05
• Single GPU type: NVIDIA B200

Caveats

• SD3 VAE substitution: stabilityai/stable-diffusion-3-medium-diffusers is gated on HF. We use stabilityai/sdxl-vae as a faithful proxy (same architecture lineage, also Stability AI). Document explicitly in the paper if reviewers ask.
• Web-DINO 1B unavailable: spec primary facebook/webssl-dino1b-light2b-224 not found; used facebook/webssl-dino300m-full2b-224 (still ViT-L scale). This is closer to V-JEPA ViT-L (304M) in param count — actually a better param match than the 1B would have been.
• Encoder finetuning is destructive; these ckpts no longer match their upstream pretrained features.

How to load

import torch
# Web-DINO / SigLIP 2 (Transformers AutoModel)
from transformers import AutoModel
state = torch.load("webdino_aux/ckpt_last.pt", weights_only=False)
model = AutoModel.from_pretrained("facebook/webssl-dino300m-full2b-224")
model.load_state_dict(state["encoder"])

# SDXL VAE (Diffusers AutoencoderKL)
from diffusers import AutoencoderKL
state = torch.load("sdxl_vae_aux/ckpt_last.pt", weights_only=False)
vae = AutoencoderKL.from_pretrained("stabilityai/sdxl-vae")
vae.load_state_dict(state["encoder"])

aux head state (state["aux_head"]) is included but training-only; the probe protocol ignores it.

Citation

@misc{Fang2026WebSSL,
  title={Web-DINO Scaling Self-Supervised Vision},
  author={Fang, Alex and others}, year={2026},
}
@misc{Tschannen2026SigLIP2,
  title={SigLIP 2}, author={Tschannen, Michael and others}, year={2026},
}
@misc{Esser2024SD3,
  title={Scaling Rectified Flow Transformers for High-Resolution Image Synthesis},
  author={Esser, Patrick and others}, year={2024},
}