MikelRubi/act-up-down-bad

Up-down BAD — ACT sorting policy for defective parts (SO-101)

Carries a part WITHOUT pin (BAD / defective) from the inspection camera to the white box and releases it. Stage 3b of the pipeline below. bad refers to the part, not to the demonstrations: every training episode is a successful demonstration. The sister policy for parts with a pin is MikelRubi/act-up-down-good.

Recommended checkpoint

Checkpoint	up-down-bad_v4/checkpoints/015000/
Why	First action error 1.87 vs 3.32 for "stay still"; 14/14 correct releases; most accurate release pose (5.3°)
Requirements	Needs the joint velocity as an extra input and runs with blending at replanning (tools/run_policy.py --blend 12)
Status	Intermediate: meets every offline criterion except the gripper F1@30 (0.66, target 0.70). Not yet tested on the robot

If there is time for only a few robot trials, try v4 15K → v4 25K → v4 30K in that order.

The system

Built at the Robotekin LeRobot Hackathon 2026 (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs a small quality-control cell for a metal bracket that may or may not carry a pin. Project page: MikelRubi/robotekin-so101-pin-sorting.

flowchart LR
    order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"]
    tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"]
    pick --> cls{"2 · pin-classifier<br/>front + wrist images"}
    cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"]
    cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"]

Stage	Repository	Model
1. Picking	MikelRubi/act-pick	ACT policy
2. Inspection	MikelRubi/pin-classifier	EfficientNet-B0 image classifier
3a. Sorting, GOOD	MikelRubi/act-up-down-good	ACT policy
3b. Sorting, BAD	MikelRubi/act-up-down-bad	ACT policy

Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two sorting policies start where the pick policy ends (part held in front of the camera).

Orchestration. The team's integrated demo, 30_full_demo_app.py, runs the loop on the real follower: an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model IkerAzurmendi/bin-intent-gliner2; the arm waits until a part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (policy_routes.json, run through run_policy.py). In the usual order GOOD parts go to the green box and BAD parts to the white box. During data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose contract between stages is in act-up-down-bad/HANDOFF.md.

Naming note: good / bad describe the part, not the quality of a demonstration. GOOD = part with a pin → green box. BAD = part without a pin (defective) → white box. up-down-bad is a policy trained on successful demonstrations of carrying a defective part to the white box. No failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility: so101-pick-cube_* datasets contain the bracket, not a cube, and up-down_* datasets are the GOOD-box task (except the 23 October warm-up sessions).

Hardware

Component	What is known	Source
Arm	SO-101 follower, 6 joints: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper (degrees; gripper 0–100)	dataset robot_type: so_follower, recording scripts
Teleoperation	SO-101 leader arm driving the follower (leader/follower); action = leader position	recording scripts
Cameras	Two USB RGB cameras: front (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and wrist (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps	dataset meta/info.json
Work cell	Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box	recordings, tray_detect.py, demo_orders.py
Recording PC	Linux, arms on USB serial ports	team notes

Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models, the part's dimensions, and the calibration procedure beyond LeRobot's standard lerobot-calibrate.

Repository layout

Folder	Saved steps
up-down-bad_v1/	10K · 15K · 20K · 25K · 30K
up-down-bad_v2/	5K
up-down-bad_v3/	5K · 10K · 15K · 20K
up-down-bad_v4/	5K · 10K · 15K · 20K · 25K · 30K · 35K · 40K

Each checkpoint holds pretrained_model/ (weights and normalisers, loaded with ACTPolicy.from_pretrained), dataset_features.json (names, order and shapes of inputs and outputs, read by run_policy.py) and training_state/ (optimizer, scheduler and RNG, to resume training).

At the root: HANDOFF.md (robot contract: start and end pose, cameras, units, stop condition), TRAINING_LOG.md (every training run, why each change was made, and the ranking) and tools/ (run_policy.py, the control loop for the SO-101 with a hardware-free --dry-run-dataset mode; robust_eval.py and eval_bad.py for evaluation).

Training data

The 27 MikelRubi/up-down-bad_* datasets (task "Carry the bracket from the camera to the bad box and release it"), combined into one dataset of 153 episodes at 30 fps. Ten faulty recordings were excluded: 28, 32, 41, 101–103, 110–112 and 131. 129 episodes for training (v1 and v2 used only the first 20 datasets, 79 episodes) and 14 whole episodes held out for evaluation: 4, 14, 24, 35, 45, 55, 65, 75, 85, 95, 115, 125, 140 and 150.

Inputs: front and wrist cameras (RGB 480×640) and the 6 joint positions. From v4 also observation.environment_state, the joint velocity state(t) − state(t−3). Output: 6 absolute joint targets in the order above. Trained with LeRobot 0.6.1 (lerobot-train) on one NVIDIA RTX 4060 Laptop GPU (8 GB).

Versions

Common base: ACT with an ImageNet ResNet18 backbone, VAE, learning rate 1e-5, batch 8, mixed precision and image augmentation.

Version	Data	Velocity input	Chunk / actions per prediction	Steps	Why
v1	79 episodes	no	100 / 100	30K	Standard ACT configuration
v2	79 episodes	no	50 / 25	5K (stopped)	v1 plateaued; try a shorter chunk
v3	129 episodes	no	100 / 100	20K	Use the new recordings
v4	129 episodes	yes	100 / 100	40K	Without velocity, every model's first action was worse than staying still

Results (14 held-out episodes, executed actions, replanning every 24 frames)

Checkpoint	First action (still = 3.32)	Improvement over copying the state	Gripper F1@30 (≥ 0.70)	Correct releases	Release timing	Release pose
v1 25K	4.68	50 %	0.71	14/14	0.31 s	8.2°
v3 20K	4.55	54.5 %	0.59	13/14	0.36 s	9.8°
v4 15K + blend	1.87	54 %	0.66	14/14	0.34 s	5.3°
v4 25K + blend	1.87	56.6 %	0.64	14/14	0.39 s	6.1°
v4 30K + blend	1.87	57 %	0.63	14/14	0.40 s	6.3°

Metrics carry 95 % bootstrap confidence intervals over episodes (tools/robust_eval.py); every checkpoint is detailed in TRAINING_LOG.md. These are offline comparisons against the demonstrations; the real test is counting how many parts end up in the box on the robot.

Running it on the robot

python tools/run_policy.py --policy up-down-bad_v4/checkpoints/015000/pretrained_model --port <follower_port> \
    --front-camera <index_or_path> --wrist-camera <index_or_path> --robot-id <calibration_id>

run_policy.py reads the checkpoint's dataset_features.json, computes the velocity live when the model needs it (v4) and blends by default. Use --blend 0 with v1–v3, which were evaluated without blending. It calls policy.reset() and clears the history before each part. HANDOFF.md describes how to plug it into an orchestrator.

Limitations

• Offline evaluation only; not yet run on the physical robot.
• Recorded in one room, with one lighting setup, on 24 October 2026.
• Gripper timing (F1@30) is just below the target for the recommended checkpoint.

Acknowledgements

Created by Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez during the Robotekin LeRobot Hackathon 2026 (23–25 October 2026, Vitoria-Gasteiz). Thanks to Bravo Robots and to Robotekin for the hackathon, to Hugging Face for LeRobot, and to everyone who took part in it.