MikelRubi/pin-classifier

Pin classifier — part with pin / without pin (SO-101)

Looks at the part held by the arm, through the front and wrist cameras, and decides whether it has a pin (pin, GOOD → green box) or not (nopin, BAD → white box). Stage 2 of the pipeline below: its verdict selects the sorting policy.

Model

File	model.pt
Architecture	ImageNet-pretrained EfficientNet-B0 (4.0 M parameters), shared by both cameras. For each camera the feature map is mean- and max-pooled; the two cameras are concatenated (front, then wrist; 5120 values) and one linear layer gives the 2 classes
Input	Two RGB images, front and wrist, centre-cropped to 4:3 (never stretched) and resized to 448×336. ImageNet normalisation is inside the model
Output	nopin / pin and the probability of pin
Accuracy	≈ 85 % on unseen recording sessions (cross-validation grouped by session). Majority-class baseline: 52 %
Caveat	Measured in the same room, with the same operator and lighting as the training data. Not tested on another day, under other lighting or with another batch of parts

The system

Built at the Robotekin LeRobot Hackathon 2026 (23–25 October 2026, Robotory, Vitoria-Gasteiz). One SO-101 arm runs a small quality-control cell for a metal bracket that may or may not carry a pin. Project page: MikelRubi/robotekin-so101-pin-sorting.

flowchart LR
    order["Order (voice / text)<br/>intent model → sorting rule"] --> tray["Part lies in the tray"]
    tray --> pick["1 · act-pick<br/>grasp the part according to its orientation,<br/>lift it, hold it up to the camera"]
    pick --> cls{"2 · pin-classifier<br/>front + wrist images"}
    cls -- "rule: usually pin (GOOD)" --> good["3a · act-up-down-good<br/>carry it to the green box, release"]
    cls -- "rule: usually no pin (BAD)" --> bad["3b · act-up-down-bad<br/>carry it to the white box, release"]

Stage	Repository	Model
1. Picking	MikelRubi/act-pick	ACT policy
2. Inspection	MikelRubi/pin-classifier	EfficientNet-B0 image classifier
3a. Sorting, GOOD	MikelRubi/act-up-down-good	ACT policy
3b. Sorting, BAD	MikelRubi/act-up-down-bad	ACT policy

Picking is a separate policy because the bracket's geometry makes the right grasp depend on its orientation; the two sorting policies start where the pick policy ends (part held in front of the camera).

Orchestration. The team's integrated demo, 30_full_demo_app.py, runs the loop on the real follower: an order (voice in Spanish, English or Basque, text, or Telegram) is read by the intent model IkerAzurmendi/bin-intent-gliner2; the arm waits until a part lies still in the tray; the pick policy lifts it to the camera; the classifier averages 1.5 s of frames; the order maps PIN / NO PIN to the green or the white box, and that box's policy carries the part there (policy_routes.json, run through run_policy.py). In the usual order GOOD parts go to the green box and BAD parts to the white box. During data collection the operator looked at the part and chose which sorting demonstration to record. The start/end-pose contract between stages is in act-up-down-bad/HANDOFF.md.

Naming note: good / bad describe the part, not the quality of a demonstration. GOOD = part with a pin → green box. BAD = part without a pin (defective) → white box. up-down-bad is a policy trained on successful demonstrations of carrying a defective part to the white box. No failed or negative demonstrations were intentionally used for training. Other informal names kept for compatibility: so101-pick-cube_* datasets contain the bracket, not a cube, and up-down_* datasets are the GOOD-box task (except the 23 October warm-up sessions).

Hardware

Component	What is known	Source
Arm	SO-101 follower, 6 joints: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper (degrees; gripper 0–100)	dataset robot_type: so_follower, recording scripts
Teleoperation	SO-101 leader arm driving the follower (leader/follower); action = leader position	recording scripts
Cameras	Two USB RGB cameras: front (fixed overhead camera looking down at the tray; the part is lifted towards it for inspection) and wrist (on the gripper). 640×480 at 30 fps; the first sessions of 23 October used 1280×720 at 10 fps	dataset meta/info.json
Work cell	Orange tray with the parts, inspection position in front of the camera, green (big) box and white (small) box	recordings, tray_detect.py, demo_orders.py
Recording PC	Linux, arms on USB serial ports	team notes

Not documented yet: camera models and exact mounting positions, workspace dimensions and lighting, servo models, the part's dimensions, and the calibration procedure beyond LeRobot's standard lerobot-calibrate.

Usage

from pin_classifier import load_model, predict      # pin_classifier.py is in this repository
model = load_model("model.pt")                      # needs torch, torchvision, numpy and Pillow
label, p_pin = predict(model, front_rgb, wrist_rgb) # HxWx3 uint8 RGB arrays, any size

pin_classifier.py rebuilds the architecture and loads the weights in strict mode. It was checked against the 110 labelled photo pairs of so101-pick-cube_20261024_115532/pin_snapshots: it classifies all 110 correctly, as expected, because those photos are part of the training data. With the cameras swapped the accuracy drops to 56–66 %: the order matters.

Training data

1004 front + wrist pairs from the MikelRubi/so101-pick-cube_* recordings (23 and 24 October 2026):

• 654 pairs from the last 3 frames of 218 pick episodes (112 pin, 106 nopin), labelled by eye. 16 of the pin and 72 of the nopin are "moderate" (blurred or partial view). 39 ambiguous episodes and 5 without a part were discarded.
• 350 pairs of photos taken on purpose (186 pin, 164 nopin), stored in the pin_snapshots/ folder of each dataset.

Training: 25 epochs, class-weighted cross-entropy, AdamW and OneCycle (backbone at a tenth of the head's learning rate), and strong augmentation: rotation, zoom, shift, flip, saturation and white balance, blur and noise.

Results (5-fold cross-validation; each fold holds out whole recording sessions)

Subset	n	Accuracy	Majority class
All pairs	1004	0.85	0.52
Episode end frames	654	0.87	0.51
Photos taken on purpose	350	0.83	0.53
Per episode	568	0.84	0.52
Per episode, clear labels only	480	0.84	0.59

Per fold: 0.76 · 0.85 · 1.00 · 0.88 · 0.88. The model.pt in this repository is trained on all the data, so these figures come from the five cross-validation models, not from it (metrics.json: training accuracy 0.998). The labels are a visual judgement, not ground truth.

Other architectures tried:

Architecture	Accuracy
ResNet18 at 448×336	0.85 on all pairs, 0.82 per episode
Small CNN trained from scratch	0.55 (chance)
ResNet18 with less data	0.66 (627 pairs) and 0.81 (905 pairs)

More data, pretrained weights and higher resolution made the biggest difference.

Files

• model.pt: dictionary with state_dict, classes, height, width, backbone, cameras and channels.
• pin_classifier.py: loading and inference.
• metrics.json: report of the final training run.
• model_card_original.md: the author's original card.

Credits and acknowledgements

Model trained by Iker Azurmendi (originally published as IkerAzurmendi/pin_classifier_so101; IkerAzurmendi/pin-classifier holds the same model.pt). Created by Iker Azurmendi, Mikel Zulueta, Ander Autillo and Diego Suarez during the Robotekin LeRobot Hackathon 2026 (23–25 October 2026, Vitoria-Gasteiz). Thanks to Bravo Robots and to Robotekin for the hackathon, to Hugging Face for LeRobot, and to everyone who took part in it.