RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion

ACT: Piper Hexagonal Bolt Insertion (100 demos, 45k steps)

An ACT policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras and joint positions. This is the 45k-step checkpoint trained on the first, 100-demonstration dataset.

Status: superseded. A newer policy trained on a cleaner re-recording of the task, with every episode starting from the zero home pose, is available at RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home. Prefer that model. This one has not been evaluated on the robot.

---

Model Details

• Method: ACT (Action Chunking with Transformers), trained from scratch with LeRobot
• Robot: AgileX Piper, single arm (piper_follower), joint-space control
• Task: "Pick up the hexagonal bolt and place it in the matching opening."
• Control rate: 30 Hz
• Developed by: RoboticsCollectiveAachen
• License: Apache 2.0

Architecture

Setting	Value
Vision backbone	ResNet-18, initialized from ImageNet (IMAGENET1K_V1)
Transformer	4 encoder layers, 1 decoder layer, dim_model 512, 8 heads, FFN 3200
CVAE	enabled, latent dim 32, 4 encoder layers, KL weight 10
Chunk size / actions executed per chunk	100 / 100
Dropout	0.1
Parameters	51.6 M

Inputs & Outputs

Feature	Shape	Description
observation.images.overhead	3 × 480 × 640	Overhead RGB camera
observation.images.angled	3 × 480 × 640	Angled RGB camera
observation.state	7	Measured positions: joints 1–6 (degrees), then gripper opening (mm)
action (output)	7	Absolute follower targets in the same order and units as the state

The policy predicts a chunk of 100 future actions (3.3 s at 30 FPS) per inference call. All inputs and outputs are normalized with mean/std statistics from the training dataset; the bundled pre/post-processors apply this automatically.

Training Data

Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1

• Episodes: 100 teleoperated demonstrations (Piper follower driven by a 7-servo Piper leader)
• Frames: 104,172 at 30 FPS (≈ 58 min). Episodes are 26–53 s long (mean 35 s)
• Recording limits: firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
• Start pose: not standardized across episodes

Training Procedure

Setting	Steps 0 – 25k	Steps 25k – 45k
Hardware	2 × NVIDIA TITAN Xp (12 GB), DDP	3 × NVIDIA TITAN Xp (12 GB), DDP
Batch size	16 per GPU (32 effective)	16 per GPU (48 effective)
Optimizer	AdamW, lr 1.5e-5 (backbone 1.5e-5), weight decay 1e-4, constant schedule	same
Gradient clipping	10.0	same
Precision	fp32	same

• Total: 45,000 steps, ≈ 1.76 M samples, ≈ 17 epochs (resumed from the 25k checkpoint; the run was stopped at 45k when the dataset was superseded)
• Seed: 1000

Image augmentation (applied on the GPU, up to 3 per frame, randomly chosen): brightness 0.8–1.2, contrast 0.8–1.2, saturation 0.5–1.5, hue ±0.05, sharpness 0.5–1.5, and a random affine transform (±5° rotation, ±5 % translation).

Training loss (action L1 in normalized units; the KL term falls to ≈0 within the first 10k steps, which is typical for ACT):

Step	5k	10k	15k	20k	25k	30k	35k	40k	45k
L1 loss	0.140	0.107	0.089	0.078	0.071	0.062	0.056	0.053	0.050

These are training-set losses. The dataset has no held-out split, so they do not measure overfitting or task success.

How to Use

LeRobot has no built-in Piper robot driver, so lerobot-rollout cannot drive this arm directly. Load the policy inside your own 30 Hz control loop:

import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.act import ACTPolicy

repo = "RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion"
policy = ACTPolicy.from_pretrained(repo).to("cuda").eval()
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, pretrained_path=repo
)

policy.reset()  # at the start of every episode
while running:  # 30 Hz
    obs = {
        "observation.state": state,             # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
        "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
        "observation.images.angled": img_ang,   # same format
        "task": "Pick up the hexagonal bolt and place it in the matching opening.",
    }
    with torch.inference_mode():
        action = postprocessor(policy.select_action(preprocessor(obs)))
    send_absolute_joint_targets(action.squeeze(0))  # 7 values, same order/units as the state

Deployment notes

• Episodes in the training data do not share a common start pose. Start each rollout from a pose similar to those in the dataset.
• Apply the same safety limits used during recording: firmware speed 40 % and a maximum target offset of 4 °/mm from the measured position. The training actions are the targets after this limiting, so the policy has never seen large jumps.
• Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
• Camera placement, resolution (640 × 480), and the overhead/angled key names must match the recording setup.
• By default the policy executes each 100-step chunk open-loop. For smoother, more reactive motion near the insertion, enable temporal ensembling: policy.config.temporal_ensemble_coeff = 0.01 and policy.config.n_action_steps = 1 (one inference per control step; costs more GPU time).

Evaluation

No real-robot evaluation has been run on this checkpoint yet.

Limitations

• Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under different lighting, camera poses, bolt or fixture positions outside the demonstrated range, or a different Piper unit.
• The training demonstrations carry no success labels and were not exhaustively reviewed, so the policy imitates whatever the operator did, including imperfect attempts.
• Single-task policy. The task string is not used by ACT, so the policy cannot be steered with language.
• Run it with a hand on the emergency stop. Nothing in the model enforces joint limits, collision avoidance, or force limits.

Citation

@inproceedings{zhao2023learning,
    title     = {Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware},
    author    = {Zhao, Tony Z. and Kumar, Vikash and Levine, Sergey and Finn, Chelsea},
    booktitle = {Robotics: Science and Systems (RSS)},
    year      = {2023},
    eprint    = {2304.13705},
    archivePrefix = {arXiv}
}

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}