Stevenshuqing/LW-Bench-SmolVLA

LW-Bench SmolVLA 5K for X7s

LeRobot SmolVLA policy fine-tuned from lerobot/smolvla_base on the audited X7s strict compositional split in LW-Compositional-Bench.

Training contract

• Dataset: LightwheelAI/Lightwheel-Tasks-X7S
• Train split: 91 clean tasks; tasks 7, 119, and 149 held out
• Inputs: three RGB cameras, 25D robot state, and canonical task language
• Output: native 21D X7s absolute action
• Hardware: 4 GPUs
• Seed: 1000
• Effective global batch: 64
• Checkpoint: 5,000 global optimizer updates
• Evaluation horizon: first 16 valid action steps

Offline 5K results

Metric	Value
H16 MAE	0.117617
H16 RMSE	0.227890
Gripper balanced accuracy	87.08%
Boundary-proxy MAE	0.147732
Interior MAE	0.111848
Boundary / interior MAE	1.321x

These results cover three strict held-out tasks and are descriptive. Offline action error does not replace closed-loop success evaluation.

Load with LeRobot

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

policy = SmolVLAPolicy.from_pretrained("Stevenshuqing/LW-Bench-SmolVLA")
policy.eval()

Use LeRobot 0.4.3 and the included preprocessor/postprocessor files. The repository contains the complete pretrained_model directory produced by LeRobot.

Provenance

• Base model: https://huggingface.co/lerobot/smolvla_base
• Code and protocol: https://github.com/stevenqing/LW-Compositional-Bench
• Full report: https://github.com/stevenqing/LW-Compositional-Bench/blob/main/artifacts/model-eval/x7s/rotation-v0/budget-5k/report.md
• Code revision: 2d0e26dd2fe2e91046f416bd65bc369f8568d7f0
• model.safetensors SHA256: f70959474aad109134f799e49dd313abf209380c8e1c808d8650e6d92287e242

Limitations

This checkpoint is specific to the X7s observation/action schema. The benchmark is offline because the available A100 GPUs cannot provide Isaac Sim RT camera rendering. Transition boundaries are action-space proxies based on gripper sign changes and robust continuous-action peaks, not human semantic annotations.