MadmanMischief/smolvla_candy_grid

SmolVLA, 71 grid demos (new camera view) — SO101 candy sort

Task: teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: "pick up the blue candy". Built during the B1S Hacktober Physical AI 2026 hackathon with LeRobot 0.6.2.

Robot: SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: wrist (USB webcam on the gripper) and phone (Android phone beside the table).

Training

• Dataset: MadmanMischief/so101_candy_grid
• Fine-tuned from lerobot/smolvla_base on the L40S, batch 64, 15k steps, chunk size 50. Checkpoints at 5k, 10k, 15k; repo root = 15k.

Results

Real arm

• 10k: 2 of 6 runs put a blue candy on the towel (picked out of the mixed scene).
• 5k: 0 of 3; one run carried the blue candy toward the towel but let go just short of it. Loop ran ~25 Hz with a ~0.4 s pause every 50 steps to compute the next chunk.

Sim (digital twin)

• 5k: 0 / 5 (one rollout lifted a blue candy 6.5 cm)
• 10k: 0 / 5. The new-view twin only approximates the new camera position and under-rates this model compared with the arm.

Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.

Run it with LeRobot

lerobot-rollout \
  --strategy.type=base \
  --policy.path=MadmanMischief/smolvla_candy_grid \
  --policy.device=cuda \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
  --task="pick up the blue candy" \
  --duration=40

The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a downscale: 2 option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under checkpoints/<step>/pretrained_model (download with hf download MadmanMischief/smolvla_candy_grid --include 'checkpoints/<step>/*').

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
observation.state	STATE	(6,)
observation.images.camera1	VISUAL	(3, 256, 256)
observation.images.camera2	VISUAL	(3, 256, 256)
observation.images.camera3	VISUAL	(3, 256, 256)
observation.images.empty_camera_0	VISUAL	(3, 480, 640)

Outputs

Feature	Type	Shape
action	ACTION	(6,)

Related

Datasets: so101_candy_sort · so101_candy_sort_half_127 · so101_candy_sort_sim_half · so101_candy_sort_mix_half · so101_candy_grid. RL policy and digital twin: so101_candy_rl.