MadmanMischief/act_candy_sort_real128

ACT, 127 real demos, half resolution (best ACT) — SO101 candy sort

Task: teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: "pick up the blue candy". Built during the B1S Hacktober Physical AI 2026 hackathon with LeRobot 0.6.2.

Robot: SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: wrist (USB webcam on the gripper) and phone (Android phone beside the table).

Training

• Dataset: MadmanMischief/so101_candy_sort_half_127
• ACT on the L40S, batch 16, 70k steps, chunk size 100, on all 127 real demos at half resolution. Checkpoints every 10k; repo root = 70k.

Results

Real arm

• 50k steps, 20 s run: picked a blue candy, placed it on the towel, started back for another.
• 50k steps, 40 s run: 4 grasps, 3 closed beside a candy (it retried on its own); the one it carried to the towel was green.
• 70k: not run on the arm.

Sim (digital twin)

• 70k: 3 / 5 (4 / 10)
• 50k: 1 / 5 (1 / 10)
• 40k: 0 / 5
• 30k: 0 / 5

Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.

Run it with LeRobot

lerobot-rollout \
  --strategy.type=base \
  --policy.path=MadmanMischief/act_candy_sort_real128 \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
  --task="pick up the blue candy" \
  --duration=40

The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a downscale: 2 option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under checkpoints/<step>/pretrained_model (download with hf download MadmanMischief/act_candy_sort_real128 --include 'checkpoints/<step>/*').

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
observation.state	STATE	(6,)
observation.images.wrist	VISUAL	(3, 240, 320)
observation.images.phone	VISUAL	(3, 360, 640)

Outputs

Feature	Type	Shape
action	ACTION	(6,)

Related

Datasets: so101_candy_sort · so101_candy_sort_half_127 · so101_candy_sort_sim_half · so101_candy_sort_mix_half · so101_candy_grid. RL policy and digital twin: so101_candy_rl.