Twu31/smolvla-cross-embodiment-mps

SmolVLA across 16 LeRobot datasets and 4 embodiments — trained entirely on a MacBook

Sixteen SmolVLA checkpoints, one per LeRobot v3 dataset, spanning bimanual ALOHA (sim and real robot), single-arm xArm, and the 2-DOF PushT planar pusher. Every run was trained on one M5 MacBook (Apple MPS, no NVIDIA GPU) — 16 SmolVLA runs plus the 5 RT-1 runs in the companion repo come to roughly 20 hours of total compute.

Code: https://github.com/twu3202/SmolVLA_ALOHA Companion releases: Twu31/rt1-lerobot-mps (same datasets, RT-1 architecture) · Twu31/smolvla-libero-eeg

---

Results

Open-loop action prediction against held-out episodes. L2 is in the dataset's native action units (radians for ALOHA, normalised position for xArm, pixels for PushT — not comparable across groups).

Rank	Checkpoint	L2 ↓	MAE ↓	Final-step loss	Notes
1	xarm_push	0.216	0.105	0.636	3-DOF, no gripper — simplest task in the set
2	xarm_push_replay	0.529	0.269	1.457	replay variant
3	aloha_static_ziploc_slide ★real	0.556	0.106	0.114	best real-robot result
4	aloha_static_battery ★real	0.651	0.121	0.115	real-robot insertion
5	aloha_static_cups_open ★real	0.694	0.139	0.111	real-robot
6	aloha_insertion	0.716	0.154	0.118	sim, human demos
7	aloha_transfer	0.729	0.149	0.119	sim, human demos
8	aloha_static_coffee ★real	0.802	0.168	0.110	real-robot
9	aloha_insertion_scripted	0.864	0.186	0.040	lowest train loss, worse eval
10	xarm_lift_replay	1.042	0.448	1.513	84×84 images, gripper timing
11	aloha_transfer_scripted	1.156	0.214	0.060	sim, scripted
12	xarm_lift	1.324	0.565	1.503	84×84 resolution bottleneck
13	aloha_multitask	1.327	0.276	0.456	2 tasks, 5000 steps
14	aloha_static_towel ★real	1.479	0.250	0.108	deformable object — hardest real-robot task
15	aloha_multitask_scripted	1.531	0.282	3.547	⚠️ normalisation mismatch, see below
—	pusht	40.4 px	25.7 px	0.239	pixel units, not comparable

Seven findings

1. Real-robot data is not harder than sim. The top four real ALOHA datasets (ziploc 0.556, battery 0.651, cups 0.694, coffee 0.802) match or beat sim ALOHA (insertion 0.716, transfer 0.729). SmolVLM2's frozen vision tower handles real lighting and texture without degradation.
2. Deformable objects are the real bottleneck. aloha_static_towel (1.479) is the worst real-robot run — a towel's shape changes through the episode, so visual state stops mapping cleanly onto joint targets.
3. Scripted demos overfit differently from human ones. aloha_insertion_scripted reaches 3× lower train loss than the human version (0.040 vs 0.118) and evaluates worse (L2 0.864 vs 0.716). Human inconsistency acts as regularisation.
4. Image resolution is a silent bottleneck for grasping. xarm_push (84×84, no gripper) is the best result in the set; xarm_lift (same 84×84, with gripper) is near the worst. The images are too coarse to time the gripper's open/close transitions.
5. Multi-task costs ~2× at this data scale. aloha_multitask L2 = 1.33 against ~0.72 single-task, even with the step budget raised to 5000.
6. Normalisation mismatch is a real failure mode. aloha_multitask_scripted trains at loss 3.5–55 because the two scripted datasets have std[joint0] differing 40× (0.004 vs 0.161); z-scoring both with one dataset's stats blows up the other. This checkpoint is published as evidence of the failure, not as a usable model.
7. The gripper-binary problem is architecture-independent. The worst dimension on every ALOHA task is the right-arm gripper (dim 13), MAE 0.14–0.27 here and 0.13–0.24 for RT-1 in the companion repo. Open/close transitions are ~1% of frames — a data problem no amount of capacity fixes.

---

Contents

One directory per dataset, each holding the final checkpoint plus the normalisation statistics it was trained with:

<dataset>/step_003000.pt        # or step_005000.pt for aloha_multitask
<dataset>/dataset_stats.json    # per-key mean/std used for z-scoring — required for inference
eval_output/                    # per-dataset .npz + .png, and the cross-dataset figure

aloha_transfer was produced by an earlier trainer and stores its stats under a stats key inside the checkpoint instead of loss_log/dataset/cfg; the other fifteen share the train_generic.py schema. Its dataset_stats.json is present either way.

---

Model

SmolVLA — SmolVLM2-500M VLM plus a flow-matching action expert, 450.0 M parameters total.

Component	Params	Trained?
VLM text tower	204.6 M	✅ yes
VLM vision tower	86.4 M	❌ frozen
VLM connector	11.8 M	✅ yes
I/O embeddings & heads	48.9 M	✅ yes
Flow-matching action expert	98.2 M	✅ yes

Built with load_vlm_weights=True, freeze_vision_encoder=True, train_expert_only=False. Note that this means the checkpoints are not fine-tunes of lerobot/smolvla_base: the VLM backbone starts from HuggingFaceTB/SmolVLM2-500M-Video-Instruct and the action expert is randomly initialised. Only the 197 frozen vision tensors remain bit-identical to lerobot/smolvla_base; the other 249 differ.

Training: 3000 steps (5000 for aloha_multitask), batch 4–16, images resized to 512×512 with padding, 48-token language prompt, chunk size 50 (ALOHA) / 10 (xArm, PushT). Weights are stored in bfloat16.

Input / output per embodiment

ALOHA (sim + real) — 14-dim bimanual joint space
  observation.images.cam_high    (3, 480, 640)   overhead RGB
  observation.state              (14,)           joint positions, z-scored
  language tokens                (48,)           e.g. "Place the coffee capsule inside..."
  → action chunk                 (50, 14)        1 s of joint targets @ 50 fps

xArm — single arm, tiny images
  observation.image              (3, 84, 84)
  observation.state              (4,)            x, y, z, gripper
  → action chunk                 (10, 4)         lift; push is 3-dim, no gripper

PushT — 2-DOF planar pusher
  observation.image              (3, 96, 96)     top-down
  observation.state              (2,)            agent x, y in pixels
  → action chunk                 (10, 2)         pixel coordinates, raw (not z-scored)

Internally: image → frozen SmolVLM2 vision encoder → transformer with the language tokens → context; state joins at the flow-matching expert, which denoises the whole chunk in 10 steps. One VLM forward pass produces 10–50 actions.

Wall-clock on M5 MPS

Run	Steps	Time
aloha_transfer, aloha_static_*	3000	25–35 min
aloha_insertion_scripted, aloha_transfer_scripted	3000	~30 min
aloha_multitask	5000	~50 min
xarm_lift	3000	~20 min
xarm_push, xarm_*_replay	3000	~48 min
pusht	3000	~85 min

Roughly 8 hours for the sixteen SmolVLA runs; about 20 hours including the RT-1 arm.

---

Usage

Plain torch.save dicts — you need dataset_configs.py and train_generic.py from the GitHub repo to rebuild the policy config.

import torch, json
from huggingface_hub import hf_hub_download

REPO = "Twu31/smolvla-cross-embodiment-mps"
ck = torch.load(hf_hub_download(REPO, "aloha_static_ziploc_slide/step_003000.pt"),
                map_location="cpu", weights_only=False)

ck["step"]          # 3000
ck["loss"]          # final-step training loss
ck["dataset"]       # "aloha_static_ziploc_slide"
ck["cfg"]           # image/state/action keys, dims, fps, chunk size
ck["loss_log"]      # per-step loss, length == step
ck["policy_state"]  # SmolVLAPolicy state_dict, 500 tensors, bfloat16

stats = json.load(open(hf_hub_download(REPO, "aloha_static_ziploc_slide/dataset_stats.json")))

Rebuilding for inference:

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
from train_generic import make_smolvla_config    # from the GitHub repo

cfg = make_smolvla_config(ck["cfg"], device="mps")
cfg.load_vlm_weights = False                     # the checkpoint carries the full VLM state
policy = SmolVLAPolicy(cfg).to("mps")
policy.load_state_dict(ck["policy_state"])

Actions come out z-scored; multiply by stats["action"]["std"] and add stats["action"]["mean"] to recover native units. pusht is the exception — it stores raw pixel coordinates (raw_gt in the registry).

---

Training data

All sixteen datasets are public LeRobot v3 releases under the lerobot organisation, listed in the metadata above. They are not redistributed here — training auto-downloads them.

One exception, so it is on the record: each eval_output/eval_*.npz stores a gts array alongside preds — the ground-truth action vectors for the held-out frames the evaluation ran on (250–12,000 rows of 2–14 floats per dataset, ~9 MB in total). No images, no video, no other observations. They are included so the reported MAE and L2 can be recomputed and checked independently rather than taken on trust; the source datasets are Apache-2.0.

Group	Datasets	Episodes each
ALOHA sim, human	aloha_transfer, aloha_insertion	50
ALOHA sim, scripted	aloha_transfer_scripted, aloha_insertion_scripted	50
ALOHA sim, multi-task	aloha_multitask, aloha_multitask_scripted	100 (2 datasets)
★ ALOHA real robot	aloha_static_coffee, _battery, _cups_open, _towel, _ziploc_slide	50–56
xArm	xarm_lift, xarm_push, xarm_lift_replay, xarm_push_replay	—
PushT	pusht	206

---

Limitations

• Open-loop evaluation only. Every number here is next-action-chunk prediction error against recorded demonstrations. No simulator rollouts, no success rates, no real-robot deployment. Low L2 does not establish that a policy completes the task.
• Small data: 50–206 episodes per task, 3000 steps. Conclusions are about the small-data behavior-cloning regime, not about SmolVLA at scale.
• Single seed per dataset; no confidence intervals.
• L2 is not comparable across embodiments (different action units and dimensionalities).
• aloha_multitask_scripted is a documented failure and should not be used as a model.

---

Related releases

Repo	What it holds
Twu31/rt1-lerobot-mps	RT-1 on 5 of these datasets — the architecture head-to-head
Twu31/smolvla-libero-eeg	SmolVLA + LIBERO + EEG as a fourth modality
Twu31/so101_hand_blue_napkin	Real SO-ARM101 handover demonstrations

Architecture > scale > data quantity. Read together with the RT-1 repo, the headline of this project is that at 50–800 episodes the choice of action head and pretrained backbone matters more than parameter count — a 244 M RT-1 beats a 450 M SmolVLA on 3 of 5 shared tasks, and loses badly on the hardest one.

---

Citation

@software{smolvla_cross_embodiment_2026,
  author = {Twu31},
  title  = {SmolVLA across 16 LeRobot datasets and 4 embodiments on Apple MPS},
  year   = {2026},
  url    = {https://github.com/twu3202/SmolVLA_ALOHA}
}

Built on SmolVLA (Shukor et al., 2025) and the LeRobot dataset ecosystem.