SmolVLA, 30 real demos, full resolution — SO101 candy sort

Task: teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: "pick up the blue candy". Built during the B1S Hacktober Physical AI 2026 hackathon with LeRobot 0.6.2.

Robot: SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: wrist (USB webcam on the gripper) and phone (Android phone beside the table).

Training

Dataset: MadmanMischief/so101_candy_sort
Fine-tuned from lerobot/smolvla_base on the L40S, batch 64, 15k steps, chunk size 50. Checkpoints at 5k, 10k, 15k; repo root = 15k.

Results

Real arm

15k: run on the arm, no result recorded.

Sim (digital twin)

Not evaluated in sim.

Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.

Run it with LeRobot

lerobot-rollout \
  --strategy.type=base \
  --policy.path=MadmanMischief/smolvla_candy_sort \
  --policy.device=cuda \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
  --task="pick up the blue candy" \
  --duration=40

The policy expects the image sizes listed under Inputs below (wrist 640x480, phone 1280x720). Other checkpoints are under checkpoints/<step>/pretrained_model (download with hf download MadmanMischief/smolvla_candy_sort --include 'checkpoints/<step>/*').

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
`observation.state`	STATE	`(6,)`
`observation.images.camera1`	VISUAL	`(3, 256, 256)`
`observation.images.camera2`	VISUAL	`(3, 256, 256)`
`observation.images.camera3`	VISUAL	`(3, 256, 256)`
`observation.images.empty_camera_0`	VISUAL	`(3, 480, 640)`