angkul07/pi05-piper-dual-ea

π₀.₅ dual-arm Piper — 50/50 retargeted ego + teleop (mf_pi05_dual_ea)

A π₀.₅ (flow-matching) fine-tune of pi05_base for the AgileX Piper bimanual rig: half of every training batch drawn from real teleoperation of the plate-handover task, half from egocentric human video retargeted onto the same robot. Trained 2026-08-29/30.

What is in this repo

path	step	size	what
29999/	30,000 (final, fully annealed)	~6.7 GB	params/ + assets/ (norm stats)
15000/	15,000 (mid-run ladder pin)	~6.7 GB	params/ + assets/ (norm stats)

Optimizer state (train_state/, ~5.9 GB per step) is not uploaded — these checkpoints serve inference and evaluation, not resumption. Four more ladder pins (5k, 10k, 20k, 25k) exist off-Hub if the holdout scoring points between these two.

The assets/ folder holds the quantile norm stats computed over the sampled 50/50 mixture (asset id mf_dual_ea). They describe the blend, not either pool, and must be used with these weights — substituting stats from any other run silently normalizes against the wrong distribution.

Data

Two LeRobot v2.1 datasets, schema-aligned at build time (one 14-D joints-major layout [right_j1-6, left_j1-6, right_gripper, left_gripper] in degrees, grippers normalized [0,1], three cameras top / left-arm / right-arm, 480×640 h264, 20 Hz):

pool	episodes	frames	tasks	source
teleop_v21	86	138,120 (~1.92 h)	1 (right_pick_handover_left_place)	real dual-arm rig, 3 physical cameras
ego_v21	377	141,302 (~1.96 h)	241	human video retargeted to Piper; 3 views synthesized from one egocentric camera

The draw is a hard 64/64 per batch, so each half contributes exactly 50% of every gradient. Teleop episodes 78–85 (a contiguous tail, ~9.3%) were held out of training as the evaluation split; nothing was withheld from the ego half.

Recipe

• Model: π₀.₅ — LoRA gemma_2b_lora trunk, full-rank action expert, EMA off, action_dim=32 (padded; real width 14), action_horizon=50 = 2.5 s of future at 20 Hz, max_token_len=200, image augmentation on (both stacks).
• Schedule: 30,000 steps at batch 128, cosine 3.5e-5 → 3.5e-6, warmup 1,000. That is 3.84 M samples: ~15.3 passes over the teleop train split, ~13.6 over ego. Batch 128 @ 30k was chosen to match the exposure of the originally planned 60k @ 64 at roughly half the wall-clock; peak LR was deliberately not rescaled for the larger batch (first batch-128 run of this recipe — one variable at a time).
• Hardware: 4× H100 SXM 80 GB, pure data parallel (32 samples/GPU), 1.4–1.5 s/step, 12 h 33 m wall-clock (~$148 at $11.79/h).
• Prompt: prompt_from_task=True. The teleop half is conditioned on the one task string above; the ego half carries its 241 distinct strings. Evals of the handover task should prompt with right_pick_handover_left_place.

Training curve

Per-step metrics (100-step window means; train_metrics.log, un-averaged, is the source — also mirrored in W&B run 212iran7):

step	loss	chunk_first	chunk_last	grad_norm
200	0.0503	0.0216	0.0864	0.072
1,000	0.0387	0.0104	0.0672	0.055
5,000	0.0225	0.0078	0.0373	0.053
10,000	0.0147	0.0065	0.0238	0.043
15,000	0.0113	0.0055	0.0182	0.040
20,000	0.0087	0.0048	0.0138	0.039
25,000	0.0071	0.0045	0.0110	0.040
29,900	0.0062	0.0039	0.0095	0.042

Monotone descent end to end; grad norm flat at ~0.04 after warmup (peak 0.58 at step 1, inside warmup, as on every run of this recipe); the early-stop stall guard never fired. chunk_last (the hardest, furthest-future slice of the 50-step chunk) tracks ~2.4× chunk_first throughout — the honest column to watch, since chunk_first benefits most from the delta-action shortcut.

How to read these numbers — and how not to

The training loss cannot support capability claims. Under cosine decay the loss falls all the way to the last step because the LR does, and at ~14–15 epochs of a ~267k-frame corpus it cannot distinguish fitting from memorising. That is exactly why two checkpoints are published and the ladder was kept: the ship decision belongs to offline evaluation on the held-out teleop episodes (78–85), not to this table. The final checkpoint has now been scored on that holdout (see the evaluation section below); the other ladder pins remain unscored, so "29999 beats its own knee" is still an open question — the eval gives it an absolute score, not a within-run rank.

Known data caveats that no training recipe removes: the ego half's three views are crops of one egocentric frame (no real parallax, unlike teleop's three physical cameras), and gripper resting-aperture statistics were reconciled across the halves at the data layer before this run.

Holdout evaluation — checkpoint 29999

Offline eval (fd-studio suite) of the final checkpoint only, on the 8 held-out teleop episodes (78–85, ~10.7 min, never seen in training), dense at every frame: open-loop 50-step chunk prediction, errors in degrees (the robot's native units — RobotOutputs un-normalises before comparison). Critical intervals were annotated by gemini-3.7-flash (Vertex) from the upright top-camera video and verified to bracket the ground-truth gripper transitions. CI-MSE is DTW-aligned within each interval.

episode	frames	action MSE (deg²)	action MAE (deg)	grip acc R	grip acc L	CI-MSE mean	grasp	handover	place
78	1,375	5.45	1.088	90.9%	90.7%	3.55	7.15	1.31	3.38
79	1,502	6.86	1.167	89.1%	93.6%	5.36	8.71	4.41	0.28
80	1,697	4.31	0.969	92.1%	94.1%	1.61	2.65	1.40	0.97
81	1,728	8.71	1.277	85.0%	94.0%	3.18	3.50	3.14	— *
82	1,325	6.02	1.172	89.4%	92.6%	4.46	7.72	2.84	4.09
83	1,295	10.29	1.422	86.9%	92.0%	4.65	1.91	3.52	11.43
84	1,711	10.16	1.226	89.9%	92.2%	2.15	1.88	1.31	4.47
85	1,578	6.66	1.200	80.6%	94.5%	2.58	4.89	2.63	0.82
all	12,211	7.32	1.186	88.0%	93.0%	3.40	4.80	2.57	3.63

* ep 81's place interval was dropped: the VLM's timestamps for that episode overrun its length (place labelled at frames 2370–2470 of 1,728), so its handover interval is also partially clipped. One defective annotation of eight; the aggregate row's per-interval means are computed over the valid intervals.

How to read this:

• Per-dim MAE ~1.2° is tight and uniform — episode means span only 0.97–1.42°, so no holdout episode is out-of-distribution for the policy; there is no collapse, freezing, or thrashing anywhere (the failure modes that sank earlier π₀.₅ runs).
• The grasp phase is the hardest on aggregate (CI-MSE 4.80 vs 2.57 handover / 3.63 place) — precision-approach is where error concentrates, consistent with grasp intervals being the discriminative segment in prior FD evals. Per episode it varies: ep 83's error lives almost entirely in the place phase (11.43), and its grasp is actually the cleanest (1.91).
• The right arm is consistently worse than the left — higher joint MAE (its j2 is the worst dim at 2.20°) and lower gripper accuracy (88.0% vs 93.0%, dropping to 80.6% on ep 85). The right arm performs the pick, the harder role in this task.
• Best episode: 80 (MAE 0.969°, CI-MSE 1.61) — the side-by-side video below shows this episode; it was chosen as the middle holdout episode before scoring and happens to be the policy's best, so read it as the upper bound, not the average.
• Caveats: degrees-native numbers are not comparable to any YAM/radians eval; and with a single checkpoint scored there is no within-run reference — these are baseline absolutes for the family, not a comparison.

Artifacts in this repo (eval/): per_episode.json · offline_proxies.json · ci_mse_results.txt · annotations_gemini-3.7-flash.json · side-by-side joint video, ep 80 (upright top camera + all 14 dims GT vs prediction) · val_episodes/ — the 8 holdout episodes' top-camera videos, rotated upright, for reference alongside the numbers.

Usage

from openpi.training import config as _config
from openpi.policies import policy_config

cfg = _config.get_config("mf_pi05_dual_ea")   # needs the configs/ tree from the repo below
policy = policy_config.create_trained_policy(cfg, "<download>/29999")

Config, robot spec (PIPER_DUAL, including the grippers_trailing joints-major layout handling) and the full run documentation live in angkul07/openpi, branch mf-piper-ea (configs/mf/piper/dual_ego_vs_teleop.py, configs/mf/datasets_dual.py).

Reproduce

# on a box with /workspace/final_data/{teleop_v21,ego_v21} and 4 GPUs
export WANDB_PROJECT=mf-piper
./vast_run/run_yam.sh mf_pi05_dual_ea dual_ea