MadmanMischief/act_candy_sort_sim

ACT, 55 sim demos only — SO101 candy sort

Task: teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: "pick up the blue candy". Built during the B1S Hacktober Physical AI 2026 hackathon with LeRobot 0.6.2.

Robot: SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: wrist (USB webcam on the gripper) and phone (Android phone beside the table).

Training

• Dataset: MadmanMischief/so101_candy_sort_sim_half
• ACT on the L40S, batch 16, 50k steps, on 55 scripted pick-and-place demos recorded in the Isaac Sim twin. Repo root = 50k.

Results

Real arm

• Not run on the arm.

Sim (digital twin)

• 50k: 0 / 5 (0 / 10), never lifted a blue candy.

Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.

Run it with LeRobot

lerobot-rollout \
  --strategy.type=base \
  --policy.path=MadmanMischief/act_candy_sort_sim \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
  --task="pick up the blue candy" \
  --duration=40

The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a downscale: 2 option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under checkpoints/<step>/pretrained_model (download with hf download MadmanMischief/act_candy_sort_sim --include 'checkpoints/<step>/*').

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
observation.state	STATE	(6,)
observation.images.wrist	VISUAL	(3, 240, 320)
observation.images.phone	VISUAL	(3, 360, 640)

Outputs

Feature	Type	Shape
action	ACTION	(6,)

Related

Datasets: so101_candy_sort · so101_candy_sort_half_127 · so101_candy_sort_sim_half · so101_candy_sort_mix_half · so101_candy_grid. RL policy and digital twin: so101_candy_rl.