MadmanMischief/act_so101_candy_sort

ACT, 30 real demos, full resolution — SO101 candy sort

Task: teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: "pick up the blue candy". Built during the B1S Hacktober Physical AI 2026 hackathon with LeRobot 0.6.2.

Robot: SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: wrist (USB webcam on the gripper) and phone (Android phone beside the table).

Training

• Dataset: MadmanMischief/so101_candy_sort
• ACT trained from scratch on the laptop (RTX 4060 8 GB), batch 4, 30k-step schedule; checkpoints at 5k, 10k, 15k, 20k. The repo root holds the 10k checkpoint.

Results

Real arm

• 10k steps: first candy picked and placed cleanly, then it went back and closed on nothing.
• 15k steps: no successful picks.

Sim (digital twin)

• 20k steps: 0 / 3, but run before the twin had assisted grasp and with an early arm mapping, so not comparable.

Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.

Run it with LeRobot

lerobot-rollout \
  --strategy.type=base \
  --policy.path=MadmanMischief/act_so101_candy_sort \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
  --task="pick up the blue candy" \
  --duration=40

The policy expects the image sizes listed under Inputs below (wrist 640x480, phone 1280x720). Other checkpoints are under checkpoints/<step>/pretrained_model (download with hf download MadmanMischief/act_so101_candy_sort --include 'checkpoints/<step>/*').

Related

Datasets: so101_candy_sort · so101_candy_sort_half_127 · so101_candy_sort_sim_half · so101_candy_sort_mix_half · so101_candy_grid. RL policy and digital twin: so101_candy_rl.