MikelRubi/act-pick

Pick — ACT policy for picking the part (SO-101)

Picks one bracket from the crate, grasping it according to its orientation, lifts it and holds it up to the inspection camera. Stage 1 of the pipeline below: it leaves the part ready for MikelRubi/pin-classifier and for one of the sorting policies.

Recommended checkpoint

Checkpoint	pick_v3_super/checkpoints/006000/
Why	Error 15.16 on the held-out episodes, 27 % below "stay still" (20.71); trained on the most data (398 episodes)
Caveat	Tied with pick_v3_strict/002400 (15.14), within noise. Every run gets worse after about one pass over its data
Status	No pick model has been tested on the robot yet

If there is little robot time, try v3_super 6K → v3_strict 2.4K → v3_1-2-pieces 18K in that order.

The system

Built at the Robotekin LeRobot Hackathon 2026 (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs a small quality-control cell for a metal bracket that may or may not carry a pin. Project page: MikelRubi/robotekin-so101-pin-sorting.

flowchart LR
    order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"]
    tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"]
    pick --> cls{"2 · pin-classifier<br/>front + wrist images"}
    cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"]
    cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"]

Stage	Repository	Model
1. Picking	MikelRubi/act-pick	ACT policy
2. Inspection	MikelRubi/pin-classifier	EfficientNet-B0 image classifier
3a. Sorting, GOOD	MikelRubi/act-up-down-good	ACT policy
3b. Sorting, BAD	MikelRubi/act-up-down-bad	ACT policy

Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two sorting policies start where the pick policy ends (part held in front of the camera).

Orchestration. The team's integrated demo, 30_full_demo_app.py, runs the loop on the real follower: an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model IkerAzurmendi/bin-intent-gliner2; the arm waits until a part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (policy_routes.json, run through run_policy.py). In the usual order GOOD parts go to the green box and BAD parts to the white box. During data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose contract between stages is in act-up-down-bad/HANDOFF.md.

Naming note: good / bad describe the part, not the quality of a demonstration. GOOD = part with a pin → green box. BAD = part without a pin (defective) → white box. up-down-bad is a policy trained on successful demonstrations of carrying a defective part to the white box. No failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility: so101-pick-cube_* datasets contain the bracket, not a cube, and up-down_* datasets are the GOOD-box task (except the 23 October warm-up sessions).

Hardware

Component	What is known	Source
Arm	SO-101 follower, 6 joints: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper (degrees; gripper 0–100)	dataset robot_type: so_follower, recording scripts
Teleoperation	SO-101 leader arm driving the follower (leader/follower); action = leader position	recording scripts
Cameras	Two USB RGB cameras: front (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and wrist (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps	dataset meta/info.json
Work cell	Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box	recordings, tray_detect.py, demo_orders.py
Recording PC	Linux, arms on USB serial ports	team notes

Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models, the part's dimensions, and the calibration procedure beyond LeRobot's standard lerobot-calibrate.

Repository layout

Folder	Original repository (IkerAzurmendi/…)	Saved steps
pick_v0_mixed/	act_so101_all_30fps	30K
pick_v1/	act-pick-*	6K · 12K · 18K · 24K · 30K
pick_v2/	act-pick-v2-*	6K · 12K · 18K · 24K · 30K
pick_v3_super/	act-pick-v3-6000, act-pick-v3-super-*	6K · 12K · 18K · 24K · 30K
pick_v3_strict/	act-pick-v3-strict-*	2.4K · 4.8K · 7.2K · 9.6K · 12K
pick_v3_1-2-pieces/	act-pick-v3-1-2-pieces-*	6K · 12K · 18K · 24K · 30K
pick_v3_chunk60/	act-pick-v3-chunk60-*	2.4K · 4.8K · 7.2K · 9.6K · 12K
pick_v3_1piece/	act-pick-v3-1piece-*	2.4K · 4.8K · 7.2K · 9.6K · 12K
pick_v3_recent/	act-pick-v3-recent-*	2.4K · 4.8K · 7.2K · 9.6K · 12K

Each checkpoint holds pretrained_model/ (weights and normalisers, loaded with ACTPolicy.from_pretrained), dataset_features.json (names, order and shapes of inputs and outputs) and model_card_original.md (the author's original card with its metrics). The originals published only the policy, without training_state/, so training cannot be resumed from these checkpoints.

Training data

Real recordings MikelRubi/so101-pick-cube_* (66 sessions on 23 and 24 October, leader/follower teleoperation). From 24 October the task is "Pick a bracket from the crate and hold it up to the camera" (41 sessions, 350 episodes, 30 fps, 640×480). The 25 sessions of 23 October were made with the first recorder (stored task "Pick up the cube and place it in the bowl", 10 fps at 1280×720 or 30 fps at 640×480); most of their episodes end with the part dropped in the bin. pick_v0_mixed used them together with the 23 October up-down_* return trips; later versions keep a filtered subset (12 episodes at 30 fps in v2 and v3; v3_super also the 10 fps sessions, up-sampled to 30 fps and centre-cropped to 4:3).

Each version applies a different filter (table below). All v3 runs are evaluated on the same 16 held-out episodes: 12 from so101-pick-cube_20261024_181939 (one part, varied positions) and 4 from so101-pick-cube_20261024_144537 (two parts).

Inputs: front and wrist cameras (RGB 480×640) and the 6 joint positions. Output: chunks of 90 absolute joint targets at 30 fps (60 for v3_chunk60), replanning every 30.

Versions

Common base: LeRobot ACT, batch 8, chunk 90, replanning every 30 actions, learning rate 1e-5, image augmentation, seed 1000.

Version	Training episodes	What changes
v0_mixed	145	First experiment: pick and return trips mixed in one policy
v1	165	First pick-only run
v2	278	Only 30 fps recordings with 1–2 parts, 5–15 s episodes, quality filters, bad takes removed
v3_super	398	Everything that passes the quality filter: 30 fps and 10 fps (up-sampled), any number of parts
v3_strict	205	Stricter filter: 6–12 s, at most 5° start gap and 1 s idle
v3_1-2-pieces	331	The v2 recipe plus the two new evening sessions of 24 October
v3_chunk60	331	Same data as v3_1-2-pieces with a chunk of 60 (2 s plan instead of 3 s)
v3_1piece	292	Only episodes with exactly one part, loose filters
v3_recent	100	Only the recent sessions, with parts in corners and along the edges

Results (offline validation)

Mean error in degrees over the predicted chunk (↓ better), and in brackets the improvement over staying still. The stay-still baseline is 20.71 on the 16 v3 episodes, 21.10 on the 15 v2 episodes and 16.23 with chunk 60.

Version	Best checkpoint	Error at best	All checkpoints
v0_mixed	30K	21.80 (1 %)	pick-cube only; up-down 19.41 (32 %)
v1	6K	16.87 (19 %)	—
v2	30K	15.24 (28 %)	15.36 · 15.57 · 15.57 · 15.34 · 15.24
v3_super	6K	15.16 (27 %)	15.16 · 16.08 · 16.04 · 16.36 · 16.27
v3_strict	2.4K	15.14 (27 %)	15.14 · 15.44 · 15.52 · 15.53 · 15.96
v3_1-2-pieces	18K	15.35 (26 %)	15.86 · 15.89 · 15.35 · 15.73 · 16.38
v3_chunk60	4.8K	11.94 (26 %) *	13.13 · 11.94 · 12.16 · 12.43 · 13.04
v3_1piece	9.6K	15.47 (25 %)	16.00 · 15.82 · 15.70 · 15.47 · 15.86
v3_recent	12K	16.87 (19 %)	17.19 · 17.10 · 17.00 · 17.10 · 16.87

* With chunk 60 the raw error is not comparable, because it looks a shorter time ahead; compare its percentage.

Reading:

• Data: going from 165 to ≥ 278 filtered episodes (v1 → v2/v3) lowers the error from 16.9 to about 15.2. Beyond that, the different data mixes are tied within noise.
• Training length: the best point comes early (2.4K–6K steps); training longer makes it worse.
• Joints: in every model, wrist_roll does not beat staying still.
• Limits of these metrics: they are offline errors against the demonstrations and do not say whether the arm grasps the part. The real test is the robot.

Usage

from lerobot.policies.act.modeling_act import ACTPolicy
policy = ACTPolicy.from_pretrained("<folder>/pick_v3_super/checkpoints/006000/pretrained_model")

Observations with the same names and units as the recordings: front and wrist cameras at 640×480 and 30 fps, and the joint state. Call policy.reset() before each part.

Limitations

• Offline evaluation only; no pick model has been run on the physical robot yet.
• The offline error does not measure grasp success.

Credits and acknowledgements

Models trained by Iker Azurmendi (originally published as IkerAzurmendi/act-pick-*). Created by Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez during the Robotekin LeRobot Hackathon 2026 (23–25 October 2026, Vitoria-Gasteiz). Thanks to Bravo Robots and to Robotekin for the hackathon, to Hugging Face for LeRobot, and to everyone who took part in it.