ZebTaylor/panthera-assembly-cnns

Panthera assembly CNN checkpoints

Final selected CNN checkpoints from RobotArmLearning for pin perception and gear height alignment. The original PyTorch checkpoint files are preserved, including embedded model configuration, landmark metadata, and saved training state where present. Each checkpoint has a JSON sidecar with its configuration and metadata. manifest.json records source paths, file sizes, and SHA-256 hashes.

Checkpoint	Purpose	Selection
pin_perception/carrier_v4/adapted.pt	Pin endpoints and four large carrier insertion bores, adapted to real recordings	400 fixed adaptation steps after synthetic epoch 6
pin_perception/carrier_v4/synthetic.pt	Corrected v4 synthetic pin model	Best synthetic validation error, epoch 6
gear_alignment/real_v4/adapted.pt	Gear and side-gap entrance, adapted to real wrist recordings	Selected checkpoint from the real-adaptation run; 400 steps
gear_alignment/native_v3/synthetic.pt	Gear and side-gap entrance, trained with synthetic augmentation	Best synthetic validation error, epoch 6

Load and predict

Download the repository, install its requirements, and run Python from the downloaded directory so the bundled tools package is importable:

hf download ZebTaylor/panthera-assembly-cnns --local-dir panthera-assembly-cnns
cd panthera-assembly-cnns
python -m pip install -r requirements.txt

Both predictors require three RGB arrays in overhead, left_wrist, right_wrist order, with shape (3, 480, 640, 3) and dtype uint8. The pin model also requires six measured tool coordinates: left XYZ in the left arm base frame, then right XYZ in the right arm base frame, in metres.

import numpy as np
from tools.pin_perception_model import PinPerceptionPredictor
from tools.gear_alignment_model import GearAlignmentPredictor

pin = PinPerceptionPredictor("pin_perception/carrier_v4/adapted.pt", device="cpu")
gear = GearAlignmentPredictor("gear_alignment/real_v4/adapted.pt", device="cpu")

# Replace these arrays with synchronized native RGB frames and measured tool XYZ.
images = np.zeros((3, 480, 640, 3), dtype=np.uint8)
tool_xyz = np.zeros(6, dtype=np.float32)
pin_result = pin.predict(images, end_effector_positions=tool_xyz)
gear_result = gear.predict(images)
print(pin_result["keypoints_2d"].shape)   # (3, 6, 2)
print(gear_result["keypoints_2d"].shape)  # (3, 2, 2)

Use device="cuda" for a compatible GPU. Image coordinates are native pixels, with X rightward and Y downward. The pin landmarks are pin_tip, pin_top, and four unordered large insertion-bore mouths. Hole slot numbers do not identify a particular physical bore. The gear landmarks are gear_center and slot_center; the slot is the peripheral side-gap entrance, separate from the upper carrier pin bores. The gear model uses RGB without tool-coordinate input.

Evaluation and use limits

The corrected adapted pin model averaged 2.55 pixels across 15 approach-bore references and 2.82 pixels across 10 held-pin endpoint references from two real episodes. These coarse manual references have approximately 5–7 pixel uncertainty and reuse previously evaluated RGB frames. They are not a new blind test or a measurement of insertion precision. One separate pre-pickup pin-top prediction had 211 pixels of error. Earlier v2/v3 real hole annotations targeted small threaded holes and are incompatible with the corrected v4 bore targets.

The real-adapted gear checkpoint averaged 7.68 pixels on eight development references; approximate entrance labels have 10-pixel uncertainty. Qualitative reserved-frame review found plausible approach localization and false gear predictions after release. Gate use to the held-gear alignment phase. Real overhead landmarks were too small to annotate and are excluded from real metric geometry, while overhead RGB remains required network context.

These models predict image landmarks. Metric triangulation requires verified camera calibration, current wrist poses, distortion correction, and a shared metric frame. Real metric accuracy and hardware alignment remain unmeasured. Confidence and heatmap spread are uncalibrated. Synthetic alignment results use kinematic IK with ideal rigid grasps and do not establish physical insertion success.