MadmanMischief/act_so101_candy_sort_half

ACT, 30 real demos, half resolution — SO101 candy sort

Task: teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: "pick up the blue candy". Built during the B1S Hacktober Physical AI 2026 hackathon with LeRobot 0.6.2.

Robot: SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: wrist (USB webcam on the gripper) and phone (Android phone beside the table).

Training

• Dataset: MadmanMischief/so101_candy_sort_half
• ACT on the L40S, batch 16, 20k steps, chunk size 100, on the 30 first demos shrunk to half resolution (the dataset repo has since grown to 35 episodes). Repo root = final step.

Results

Real arm

• 5k steps: run only to check loop speed (29 Hz at half resolution vs. 40 ms/forward at full res); no successful pick.

Sim (digital twin)

• Not evaluated in sim.

Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.

Run it with LeRobot

lerobot-rollout \
  --strategy.type=base \
  --policy.path=MadmanMischief/act_so101_candy_sort_half \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
  --task="pick up the blue candy" \
  --duration=40

The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a downscale: 2 option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under checkpoints/<step>/pretrained_model (download with hf download MadmanMischief/act_so101_candy_sort_half --include 'checkpoints/<step>/*').

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
observation.state	STATE	(6,)
observation.images.wrist	VISUAL	(3, 240, 320)
observation.images.phone	VISUAL	(3, 360, 640)

Outputs

Feature	Type	Shape
action	ACTION	(6,)

Related

Datasets: so101_candy_sort · so101_candy_sort_half_127 · so101_candy_sort_sim_half · so101_candy_sort_mix_half · so101_candy_grid. RL policy and digital twin: so101_candy_rl.