RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home

ACT: Piper Hexagonal Bolt Insertion (zero-home, 80 demos)

An ACT policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras and joint positions. It is trained on 80 demonstrations that all start from the zero home pose. The main branch holds the final 100k-step checkpoint. The 80k-step checkpoint is kept on the step-80000 branch (revision="step-80000").

Status: final 100k-step checkpoint, not yet evaluated on the robot. It has seen each training frame about 94 times. Compare it on the robot with the 80k checkpoint on the step-80000 branch before relying on it.

---

Model Details

• Method: ACT (Action Chunking with Transformers), trained from scratch with LeRobot
• Robot: AgileX Piper, single arm (piper_follower), joint-space control
• Task: "Pick up the hexagonal bolt and place it in the matching opening."
• Control rate: 30 Hz
• Developed by: RoboticsCollectiveAachen
• License: Apache 2.0

Architecture

Setting	Value
Vision backbone	ResNet-18, initialized from ImageNet (IMAGENET1K_V1)
Transformer	4 encoder layers, 1 decoder layer, dim_model 512, 8 heads, FFN 3200
CVAE	enabled, latent dim 32, 4 encoder layers, KL weight 10
Chunk size / actions executed per chunk	100 / 100
Dropout	0.1
Parameters	51.6 M

Inputs & Outputs

Feature	Shape	Description
observation.images.overhead	3 × 480 × 640	Overhead RGB camera
observation.images.angled	3 × 480 × 640	Angled RGB camera
observation.state	7	Measured positions: joints 1–6 (degrees), then gripper opening (mm)
action (output)	7	Absolute follower targets in the same order and units as the state

The policy predicts a chunk of 100 future actions (3.3 s at 30 FPS) per inference call. All inputs and outputs are normalized with mean/std statistics from the training dataset; the bundled pre/post-processors apply this automatically.

Training Data

Samisaliveagain/piper-hexagonal-bolt-insertion-80-demos-zero-home-v1

• Episodes: 80 teleoperated demonstrations (Piper follower driven by a 6-joint leader with a gripper trigger)
• Frames: 50,921 at 30 FPS (≈ 28 min). Episodes are 14–34 s long (mean 21 s)
• Recording limits: firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
• Start pose: every episode starts within 1 °/mm of all-zero joints with the gripper closed

Training Procedure

Setting	Value
Hardware	3 × NVIDIA TITAN Xp (12 GB), DDP
Batch size	16 per GPU (48 effective)
Optimizer	AdamW, lr 2e-5 (backbone 2e-5), weight decay 1e-4, constant schedule
Gradient clipping	10.0
Precision	fp32
Steps (this checkpoint)	100,000 (≈ 4.8 M samples, ≈ 94 epochs)
Seed	1000

Image augmentation (applied on the GPU, up to 3 per frame, randomly chosen): brightness 0.8–1.2, contrast 0.8–1.2, saturation 0.5–1.5, hue ±0.05, sharpness 0.5–1.5, and a random affine transform (±5° rotation, ±5 % translation).

Training loss (action L1 in normalized units; the KL term falls to ≈0 within the first 10k steps, which is typical for ACT):

Step	5k	10k	20k	30k	40k	50k	60k	70k	80k	90k	100k
L1 loss	0.103	0.074	0.055	0.046	0.038	0.035	0.032	0.030	0.028	0.026	0.025

These are training-set losses. The dataset has no held-out split, so they do not measure overfitting or task success.

How to Use

LeRobot has no built-in Piper robot driver, so lerobot-rollout cannot drive this arm directly. Load the policy inside your own 30 Hz control loop:

import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.act import ACTPolicy

repo = "RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home"
policy = ACTPolicy.from_pretrained(repo).to("cuda").eval()
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, pretrained_path=repo
)

policy.reset()  # at the start of every episode
while running:  # 30 Hz
    obs = {
        "observation.state": state,             # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
        "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
        "observation.images.angled": img_ang,   # same format
        "task": "Pick up the hexagonal bolt and place it in the matching opening.",
    }
    with torch.inference_mode():
        action = postprocessor(policy.select_action(preprocessor(obs)))
    send_absolute_joint_targets(action.squeeze(0))  # 7 values, same order/units as the state

Deployment notes

• Start every rollout from the zero home pose: all joint targets at 0 ° and the gripper closed. Every training episode begins there, so other start poses are out of distribution.
• Apply the same safety limits used during recording: firmware speed 40 % and a maximum target offset of 4 °/mm from the measured position. The training actions are the targets after this limiting, so the policy has never seen large jumps.
• Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
• Camera placement, resolution (640 × 480), and the overhead/angled key names must match the recording setup.
• By default the policy executes each 100-step chunk open-loop. For smoother, more reactive motion near the insertion, enable temporal ensembling: policy.config.temporal_ensemble_coeff = 0.01 and policy.config.n_action_steps = 1 (one inference per control step; costs more GPU time).

Evaluation

No real-robot evaluation has been run on this checkpoint yet.

Limitations

• Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under different lighting, camera poses, bolt or fixture positions outside the demonstrated range, or a different Piper unit.
• The training demonstrations carry no success labels and were not exhaustively reviewed, so the policy imitates whatever the operator did, including imperfect attempts.
• Single-task policy. The task string is not used by ACT, so the policy cannot be steered with language.
• Run it with a hand on the emergency stop. Nothing in the model enforces joint limits, collision avoidance, or force limits.
• 94 epochs over 80 demonstrations is a lot of training. The policy may reproduce the demonstrated trajectories closely but generalize less to new bolt or fixture placements than an earlier checkpoint would.

Citation

@inproceedings{zhao2023learning,
    title     = {Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware},
    author    = {Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea},
    booktitle = {Robotics: Science and Systems (RSS)},
    year      = {2023},
    eprint    = {2304.13705},
    archivePrefix = {arXiv}
}

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}