Harrysunshine/so101-smolvla-9task

SO-ARM101 pick-and-place — SmolVLA fine-tune (9 tasks)

A SmolVLA policy fine-tuned on a 9-task SO-ARM101 tabletop pick-and-place dataset. Language-conditioned: one checkpoint serves all nine tasks, selected by the instruction string passed at inference.

Trained with LeRobot (lerobot-train) on the same merged dataset and the same evaluation harness as Harrysunshine/so101-pi0-9task, so the two are directly comparable on cost and offline accuracy.

Tasks

Nine tabletop tasks. The instruction must be passed verbatim — the strings below are exactly the ones present in the training data, including the article oddity in a eraser and the missing it in place in the bin.

Instruction	Episodes	Frames
Pick up a battery and place in the bin	200	56,295
Pick up a can and place in the bin	250	82,792
Pick up a cube and place in the bin	300	106,085
Pick up a eraser and place in the bin	300	109,899
Pick up a golf and place in the bin	250	77,409
Pick up a medicine bottle and place in the bin	300	116,342
Pick up a plush toy and place in the bin	200	80,811
Stack the cube on the can	200	74,425
Stack the smaller cube on the larger one	200	80,905
Total	2,200	784,963

Merged into one training set with no re-balancing / no up-sampling.

Data

• Source: zhuzhuangtian/so101-pick-place-tasks on ModelScope (Apache-2.0), LeRobot v3.0 format.
• Robot: SO-ARM101 follower arm, 6 DoF.
• Cameras: two RGB streams, 480×640 each, 30 fps.
• observation.state and action are both float32(6): shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper (joint positions, absolute targets).

Training

Base	lerobot/smolvla_base; VLM backbone HuggingFaceTB/SmolVLM2-500M-Video-Instruct
Trainable	99.3 M of 459.5 M — vision encoder frozen, action expert only (both upstream defaults)
Steps	30,000 (≈ 2.4 epochs at effective batch 64)
Batch	8 per device × 8 devices = 64 effective
Optimizer	AdamW, lr 1e-4, wd 1e-10, betas (0.9, 0.95)
Schedule	cosine decay, 1,000 warmup steps, decay to 2.5e-6
Action chunk	50 (chunk_size = n_action_steps = 50)
Augmentation	same as the π0 run: brightness/contrast 0.7–1.3, hue ±0.05, saturation 0.5–1.5, sharpness 0.5–1.5, affine ±5° / translate 0.05; up to 5 of 6 sampled per frame
Normalization	state/action MEAN_STD from real dataset statistics; visual IDENTITY
Wall clock	≈ 2 h 40 min, i.e. ≈ 3.4× faster than the π0 full fine-tune on the identical data

Published checkpoint = step 30,000 (the final step). All 12 saved checkpoints were evaluated offline; the curve is monotonically improving with no turning point, so the last step is the best one available — see the caveat below.

Camera key remapping

The base model was trained with three camera keys (camera1/2/3); this dataset has two (top, wrist). Training used:

--rename_map='{"observation.images.top":"observation.images.camera1",
               "observation.images.wrist":"observation.images.camera2"}'

The missing third camera needs no filler — LeRobot's visual-feature check passes when one camera set is a subset of the other. Consequence for inference: this checkpoint expects camera1 and camera2 keys, mapping to the top and wrist views respectively.

Offline evaluation

No simulator exists for this rig, so evaluation is offline only, against a copy-the-current-joint-angles baseline:

• ratio = MAE(prediction, action) / MAE(current_state, action) — must be **< 1**. A raw corr > 0.9 test is useless on this data: the naive baseline itself scores 0.962–0.999.
• delta_corr = corr(prediction − state, action − state) — must be > 0.
• delta_std — guards against output-magnitude collapse.

Published checkpoint (200 frames sampled uniformly across the whole dataset):

Joint	ratio ↓	delta_corr ↑	delta_std
shoulder_pan	0.745	0.742	0.854
shoulder_lift	0.554	0.860	0.961
elbow_flex	0.484	0.837	0.875
wrist_flex	0.886	0.412	0.642
wrist_roll	0.929	0.691	0.952
gripper	0.528	0.763	0.960
worst	0.929	0.412

All six joints beat the baseline. Weakest axes are the two wrist DoF — the same weak spot the π0 run shows on the same data, which points at the data rather than at either architecture.

Under-trained, on purpose of the comparison

Only the last two of twelve checkpoints clear worst_ratio < 1, and the curve is still descending at step 30,000 with no U-turn. With 99.3 M trainable parameters at 2.4 epochs this model is nowhere near saturation — the upstream recipe suggests ~200 k steps. These numbers are an under-trained reading, not SmolVLA's ceiling. Training longer should improve it.

For reference, on the identical dataset at the identical 30 k steps, the π0 full fine-tune is more accurate offline on 6/6 joints. That is a comparison at equal step count under each model's own recommended recipe, not at equal convergence, so it does not show that SmolVLA is architecturally weaker. What SmolVLA wins by an order of magnitude is cost: ≈ 3.4× faster wall clock, 33× fewer trainable parameters, ≈ 15× smaller checkpoints.

Real-robot status

Not validated on hardware. Only offline metrics exist for this checkpoint.

Usage

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

policy = SmolVLAPolicy.from_pretrained("Harrysunshine/so101-smolvla-9task")
# batch: observation.state (6,), observation.images.camera1 (top view),
#        observation.images.camera2 (wrist view),
#        task = one instruction string from the table above
action = policy.select_action(batch)

Or as a LeRobot policy server for a low-power robot host:

# on the GPU host
lerobot-policy-server --host=0.0.0.0 --port=8080
# on the robot host
lerobot-robot-client \
  --policy_type=smolvla \
  --pretrained_name_or_path=Harrysunshine/so101-smolvla-9task \
  --task="Pick up a plush toy and place in the bin" \
  --actions_per_chunk=50

Note that SmolVLAPolicy.__init__ pulls a second model besides this checkpoint — the VLM backbone HuggingFaceTB/SmolVLM2-500M-Video-Instruct (≈ 7 GB). Fully offline (HF_HUB_OFFLINE=1) the first run fails with LocalEntryNotFoundError inside __init__ and it looks like this repo downloaded badly; it is the backbone that is missing from cache. Warm it once with network access.

The two *normalizer_processor.safetensors files are required. Without them LeRobot silently falls back to identity normalization: loss looks fine, but every action comes out at the wrong scale. Load the repo as a whole, do not cherry-pick model.safetensors.

Only inference assets are published here. Optimizer/scheduler/RNG state is not included, so this repo cannot be used to resume training.

Limitations

• Under-trained at 30 k steps (see above). Do not read these metrics as the architecture's limit.
• Rig-specific top camera. The training data comes from a different physical setup than ours: the wrist view matches pixel-for-pixel (same SO-ARM101 follower hardware), but the top camera extrinsics differ. Dropping this checkpoint onto another SO-ARM101 will not work without camera adaptation or a short fine-tune on target-rig data.
• Offline metrics do not predict real-world success.
• Absolute joint-position action space; sending these as deltas will diverge in closed loop.
• Expects camera1 / camera2 observation keys (top / wrist) at 480×640 source resolution.
• Tabletop objects and bin from the training scene only; no generalization claims beyond them.

Related

• Harrysunshine/so101-pi0-9task — π0 full fine-tune, same data, same evaluation harness.
• Harrysunshine/so101-act-cube — ACT on the single cube task.

License

Apache-2.0, following the lerobot/smolvla_base base model and the source dataset.

---

中文说明

SO-ARM101 桌面抓放 SmolVLA 微调权重，9 个任务合并训练，语言条件。与 so101-pi0-9task 同一份数据、同一套验收代码， 可直接做成本/精度对照。指令必须与上表逐字一致。

• 配方＝SmolVLA 官方默认：冻视觉塔、只训动作专家（459.5M 里 99.3M 可训），LR 1e-4，30k 步，有效 batch 64。
• 相机键要接：基座按 camera1/2/3 训的，本数据只有 top/wrist → 训练时 --rename_map 接前两路， 第三路不用补（判据是两边相机集合有一方是子集即通过）。**因此推理时这个 checkpoint 认 camera1/camera2**。
• 发布末点 30,000：12 存点逐点验收，单调下行、无 U 形拐点，末点即当前最优。
• 六关节 ratio 全部 < 1；短板同为腕部两自由度（wrist_roll 0.929、wrist_flex 0.886）—— 与 π0 在同一份数据上的短板一致，说明是数据的难点，不是某个架构的弱点。
• 这是欠训读数：只有末两点过硬底线、曲线仍在下行，99.3M 可训参数在 2.4 epoch 远未饱和 （官方建议 ~200k 步）。同 30k 步下 π0 六关节全胜，但那是等步数不是等收敛，不能推出 SmolVLA 架构更弱； SmolVLA 赢的是成本：快 3.4×、可训参数少 33×、存点小约 15×。
• 真机未验证，只有离线指标。
• 上手坑：SmolVLAPolicy.__init__ 会另外单独拉 VLM 骨干 SmolVLM2-500M-Video-Instruct（约 7G）。 离线首跑报 LocalEntryNotFoundError，看着像本仓没下好，其实是第二个模型没缓存。
• 两个 *normalizer_processor.safetensors 必须一起加载，缺了静默退回恒等归一化、动作尺度整体错。
• 边界：top 相机外参与我们自己的臂不同，换机器需相机适配或少量目标机器数据再微调。