Kavin60606/cimse-so100-molmoact2-ckpts

MolmoAct2 × SO-100 sim — checkpoint zoo for CI-MSE ↔ success correlation

Twelve MolmoAct2 fine-tunes (LoRA-VLM + full action expert) on scripted SO-100 pick-cube demos in ManiSkill3, saved every 250 steps, each with a closed-loop success rate and offline CI-MSE scores. Purpose: test whether Critical Interval MSE (arXiv 2606.29898) predicts closed-loop success in simulation, using one policy family (MolmoAct2, arXiv 2605.02881).

Layout

runA/000250 … runA/002000   # 100 % data (360 eps), 8 checkpoints, LeRobot pretrained_model dirs
runB/000250 … runB/001000   # 20 % data (72 eps), 4 checkpoints
zeroshot_config/            # lerobot/MolmoAct2-SO100_101-LeRobot config patched for lerobot 0.6.2 (weights: use that repo)

Each pretrained_model = config.json, model.safetensors (bf16, 12 GB), pre/post-processor files, train_config.json.

Task / environment

ManiSkill3 custom env PickCubeSO100Cam2-v1 (30 Hz control, two 224×224 cams, visible green goal sphere, cube ±5 cm, goal 6–11 cm high). Success = cube within 2.5 cm of marker and arm nearly static. Demos from a screw/RRT motion-planning oracle. Data card: so100_pickcube_train.

Training config

base	allenai/MolmoAct2 (post-trained foundation)
VLM	LoRA r=64 α=16; action expert fully trained; action_mode=continuous, 8 flow timesteps
chunk	30 actions @ 30 Hz, absolute joint degrees, normalize_gripper=true, q01/q99 normalization from train stats
prompt	setup_type="single so100/so101 robotic arm in molmoact2", control_mode="absolute joint pose"
images	cam0, cam1; RandomResizedCrop 0.95 / ±5° / color jitter
optim	bs 16, LR 1e-5 VLM / 5e-6 ViT / 5e-5 expert, warmup 200, bf16, grad-ckpt; 3.7 s/step on one A100-80GB

Closed-loop success (30 episodes, fixed seeds 200000+, open-loop 30-step chunks)

ckpt	SR	ckpt	SR
runA/000250	0.10	runB/000250	0.13
runA/000500	0.27	runB/000500	0.27
runA/000750	0.17	runB/000750	0.33
runA/001000	0.33	runB/001000	0.37
runA/001250	0.37	zero-shot SO-101	0.00
runA/001500	0.57
runA/001750	0.53
runA/002000	0.57

Note: re-planning every 10 steps instead of 30 halved success (stall behaviour); use n_action_steps=30.

Offline metric ↔ success correlation

Spearman ρ between offline metric on the val set and closed-loop SR, 13 checkpoints (12 without zero-shot):

metric	ρ (n=13)	ρ (n=12, no zero-shot)
raw MSE, all frames	−0.90	−0.88
raw MSE, interval frames	−0.90	−0.87
CI only (GT intervals)	−0.84	−0.80
CI + temporal ensemble H=8	−0.84	—
CI + ens8 + DTW W=1 (paper CI-MSE)	−0.81	−0.76
CI + DTW1	−0.88	—
same, Gemini-2.5-Pro intervals	−0.84 … −0.88	—
Every metric selects a top checkpoint (A@2000, SR 0.57 = true best A@1500). In this clean single-task sim regime the
CI-MSE refinements do not improve on plain MSE. Segment-wise: grasp-interval error predicts SR (ρ −0.8); place-interval
error is weak; predicting which phase fails matched deployment in only 8/13. Full write-up: REPORT.md in the code folder.

Reproduce

# eval (ManiSkill env + lerobot 0.6.2 @ 3f2c29e, see setup_box.sh)
python eval_sim.py --ckpt runA/001500 --episodes 30 --num-envs 10 --n-action-steps 30 --out results/A1500
# offline predictions for CI-MSE
python dump_predictions.py --ckpt runA/001500 --val-root so100_pickcube_val --frames-json frames_val.json --out preds/A1500/predictions.h5
python correlate.py

Code: fd-studio/eval/sim_so100/ (env, oracle, dataset build, eval, dump, correlate) + fd-studio/eval/ci_mse (metric). Full technical notes: DATA.md in the same folder.