RoboticsCollectiveAachen/diffusion_piper_hex_bolt_insertion

Diffusion Policy: Piper Hexagonal Bolt Insertion (100 demos, 100k steps)

A Diffusion Policy for an AgileX Piper arm that picks up a hexagonal bolt and places it in the matching opening, using two RGB cameras and joint positions. This is the final 100k-step checkpoint, trained on the first, 100-demonstration dataset.

Status: superseded. Newer policies were trained on a cleaner re-recording of the task, in which every episode starts from the zero home pose: RoboticsCollectiveAachen/act_piper_hexagonal_bolt_insertion_zero_home and RoboticsCollectiveAachen/smolvla_piper_hex_bolt_insertion_80. Prefer those. This model has not been evaluated on the robot.

---

Model Details

• Method: Diffusion Policy, trained from scratch with LeRobot (vision backbone initialized from ImageNet)
• Robot: AgileX Piper, single arm (piper_follower), joint-space control
• Task: "Pick up the hexagonal bolt and place it in the matching opening."
• Control rate: 30 Hz
• Developed by: RoboticsCollectiveAachen
• License: Apache 2.0

Architecture

Setting	Value
Vision backbone	ResNet-18, initialized from ImageNet (IMAGENET1K_V1), separate encoder per camera, spatial softmax (32 keypoints)
Image preprocessing	480 × 640 resized to 240 × 320, no crop
Denoising network	1-D conditional U-Net, channels 512 / 1024 / 2048, kernel 5, 8 groups, FiLM conditioning
Noise schedule (training)	100 DDPM timesteps, squaredcos_cap_v2, ε-prediction, samples clipped to ±1
Sampler (inference)	DDIM, 10 steps
Observation history / horizon / actions executed	2 / 64 / 32
Parameters	277.9 M

Inputs & Outputs

Feature	Shape	Description
observation.images.overhead	3 × 480 × 640	Overhead RGB camera
observation.images.angled	3 × 480 × 640	Angled RGB camera
observation.state	7	Measured positions: joints 1–6 (degrees), then gripper opening (mm)
action (output)	7	Absolute follower targets in the same order and units as the state

From the last 2 observations, the policy plans a 64-step action trajectory and executes the first 32 (1.07 s at 30 FPS) before replanning. Images are normalized with mean/std statistics. State and action are min/max-scaled to [−1, 1] using the training-dataset range. The bundled pre/post-processors apply this automatically. Because samples are clipped to ±1, the policy cannot command targets outside the range covered by the training data.

Training Data

Samisaliveagain/piper-hexagonal-bolt-insertion-100-demos-v1

• Episodes: 100 teleoperated demonstrations (Piper follower driven by a 7-servo Piper leader)
• Frames: 104,172 at 30 FPS (≈ 58 min). Episodes are 26–53 s long (mean 35 s)
• Recording limits: firmware speed 40 %, max target offset 4 °/mm, joint 4 locked at 0 °
• Start pose: not standardized across episodes

Training Procedure

Setting	Value
Hardware	1 × NVIDIA Quadro RTX 6000 (24 GB)
Batch size	32
Optimizer	Adam, lr 1e-4, betas (0.95, 0.999), weight decay 1e-6
LR schedule	cosine, 500 warmup steps
Gradient clipping	10.0
Precision	fp32
Steps (this checkpoint)	100,000 (≈ 3.2 M samples, ≈ 31 epochs), 10 h 54 min
Image augmentation	none
Seed	1000

Training loss (MSE of the predicted noise, in normalized units):

Step	10k	20k	30k	40k	50k	60k	70k	80k	90k	100k
Loss	0.010	0.007	0.007	0.005	0.005	0.004	0.003	0.003	0.003	0.002

These are training-set losses. The dataset has no held-out split, so they do not measure overfitting or task success. The loss was still falling when training stopped. Intermediate checkpoints (every 10k steps) are not published.

How to Use

LeRobot has no built-in Piper robot driver, so lerobot-rollout cannot drive this arm directly. Load the policy inside your own 30 Hz control loop:

import torch
from lerobot.policies import make_pre_post_processors
from lerobot.policies.diffusion.modeling_diffusion import DiffusionPolicy

repo = "RoboticsCollectiveAachen/diffusion_piper_hex_bolt_insertion"
policy = DiffusionPolicy.from_pretrained(repo).to("cuda").eval()
preprocessor, postprocessor = make_pre_post_processors(
    policy.config, pretrained_path=repo
)

policy.reset()  # at the start of every episode (clears the observation history and action queue)
while running:  # 30 Hz
    obs = {
        "observation.state": state,             # torch.float32, shape (7,): joints 1-6 [deg], gripper [mm]
        "observation.images.overhead": img_ovh, # torch.float32, shape (3, 480, 640), RGB in [0, 1]
        "observation.images.angled": img_ang,   # same format
    }
    with torch.inference_mode():
        action = postprocessor(policy.select_action(preprocessor(obs)))
    send_absolute_joint_targets(action.squeeze(0))  # 7 values, same order/units as the state

Inference cost (measured on a Quadro RTX 6000): about 1.2 GB of VRAM and about 110 ms per replanning step. Replanning happens once every 32 control steps; the other select_action calls only pop the next queued action. Laptop GPUs with 8 GB are enough.

Deployment notes

• Episodes in the training data do not share a common start pose. Start each rollout from a pose similar to those in the dataset.
• Apply the same safety limits used during recording: firmware speed 40 % and a maximum target offset of 4 °/mm from the measured position. The training actions are the targets after this limiting, so the policy has never seen large jumps.
• Joint 4 was locked to a 0 ° target in the training data. Keep it locked, or force its target to 0.
• Camera placement, resolution (640 × 480), and the overhead/angled key names must match the recording setup. Pass full-resolution frames; the policy resizes them itself.
• The ~110 ms replanning step happens inside one control tick. Either let that tick run long, or run inference in a separate thread so the 30 Hz loop keeps sending the queued actions.

Evaluation

No real-robot evaluation has been run on this checkpoint yet.

Limitations

• Trained on a single robot, cell, camera setup, and object. Expect degraded behavior under different lighting, camera poses, bolt or fixture positions outside the demonstrated range, or a different Piper unit.
• The training demonstrations carry no success labels and were not exhaustively reviewed, so the policy imitates whatever the operator did, including imperfect attempts.
• Single-task policy. It takes no language input.
• Trained without image augmentation, so it may be more sensitive to lighting and camera changes than the ACT models.
• Run it with a hand on the emergency stop. Nothing in the model enforces joint limits, collision avoidance, or force limits.

Citation

@article{chi2024diffusionpolicy,
    author  = {Cheng Chi and Zhenjia Xu and Siyuan Feng and Eric Cousineau and Yilun Du and Benjamin Burchfiel and Russ Tedrake and Shuran Song},
    title   = {Diffusion Policy: Visuomotor Policy Learning via Action Diffusion},
    journal = {The International Journal of Robotics Research},
    year    = {2024}
}

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}