SeonghoonYu/Race2_realworld

RACE2 on real-world Piper data (pi0.5 fine-tunes, H = 10/15/20/25/30)

Fine-tunes of pi0.5 (gs://openpi-assets/checkpoints/pi05_base, PyTorch port) on a real-world AgileX Piper 6-DoF arm + gripper, and RACE2 post-training of those models, for three table-top manipulation tasks. Trained with openpi (scripts/real_world/, scripts/real_world_race2/) on 4x RTX 6000 Ada.

Data

90 teleoperated demonstrations (30 per task), 24,576 frames at 30 fps, LeRobot v3.0 format.

task id	instruction	episodes	frames
0	pick up the gray bowl next to the plate and place it on the plate	30	9,346
1	pick up the gray bowl on the plastic cabinet and place it on the plate	30	8,264
2	pick up the gray bowl on the gift box and place it on the plate	30	6,966

Observations: cam0 wrist + cam1 fixed third-person, both 224x224, plus the 7-D state (joint_1..6 in rad, gripper 0 = closed / 1 = open). Actions are absolute leader-arm joint targets; training normalizes the 6 joints as deltas from the chunk's first state (gripper stays absolute) and un-normalizes back to absolute targets at serving time. All 90 episodes are used for training — there is no held-out validation split, so the metrics below are training metrics.

Runs

Every run: 20,000 steps, global batch 32, AdamW (0.9/0.95, wd 1e-10, clip 1.0), warmup 1,000 + cosine 2.5e-5 -> 2.5e-6, bf16, no EMA. Checkpoints at 5k/10k/15k/20k; the 20k one also carries optimizer.pt. Norm stats (assets/piper_t123/norm_stats.json) are shared by every run.

folder	init	trainable	final metrics (step 19,900)
plain_ft_h10	pi05_base	VLM LoRA r=16 (28.3M) + action expert (428M) + projections	loss 0.0059
plain_ft_h15	pi05_base	same	loss 0.0072
plain_ft_h20	pi05_base	same	loss 0.0078
plain_ft_h25	pi05_base	same	loss 0.0086
plain_ft_h30	pi05_base	same	loss 0.0094
race2_post_ft_h10	plain_ft_h10 @20k	expert + RACE modules (VLM frozen)	fm 0.0049 / base_fm 0.0025 / BCE 0.134 / peak±1 0.85 / pres 0.93
race2_post_ft_h15	plain_ft_h10 @20k	same	fm 0.0059 / base_fm 0.0035 / BCE 0.148 / peak±1 0.78 / pres 0.89
race2_post_ft_h20	plain_ft_h10 @20k	same	fm 0.0065 / base_fm 0.0047 / BCE 0.159 / peak±1 0.67 / pres 0.85
race2_post_ft_h25	plain_ft_h10 @20k	same	fm 0.0068 / base_fm 0.0063 / BCE 0.168 / peak±1 0.62 / pres 0.81
race2_post_ft_h30	plain_ft_h10 @20k	same	fm 0.0071 / base_fm 0.0078 / BCE 0.176 / peak±1 0.52 / pres 0.76
race2_from_base_h10	pi05_base	VLM LoRA r=16 + expert + RACE modules	fm 0.0058 / base_fm 0.0032 / BCE 0.133 / peak±1 0.82 / pres 0.93

H is the action horizon (chunk length in frames; 30 fps). RACE2 adds a timing head that predicts per-slot proximity to a transition (PELT changepoints on the action signal, Gaussian target, sigma 1 / support 3 frames) and applies a localized zero-init adaRMS modulation, so b = 0 reproduces the base policy exactly. fm = flow-matching loss with modulation, base_fm = the unmodulated read pass, BCE = timing-head loss, peak±1 = fraction of transition-bearing chunks whose predicted peak lands within one frame of the label, pres = transition-present accuracy. LoRA adapters are merged into every published model.safetensors, so all files load into the plain model; trainable.safetensors (LoRA A/B + expert) is kept only for exact resume.

Files per checkpoint

<run>/<step>/model.safetensors        merged weights (7.2-7.9 GB)
<run>/<step>/optimizer.pt             AdamW state (20k step only)
<run>/<step>/trainable.safetensors    LoRA + expert params, for resume (LoRA runs only)
<run>/<step>/config.json, metadata.pt
<run>/<step>/assets/piper_t123/norm_stats.json
logs/                                 full training logs, chain logs, data-prep log, PLAN.md

Serving

# plain FT
python scripts/serve_policy.py policy:checkpoint \
  --policy.config pi05_real_world_t123_h15 --policy.dir <download>/plain_ft_h15/20000
# RACE2 (config name must match the horizon)
python scripts/serve_policy.py policy:checkpoint \
  --policy.config pi05_real_world_race2_t123_h15 --policy.dir <download>/race2_post_ft_h15/20000

The client sends {"observation/image": cam1 HWC uint8, "observation/wrist_image": cam0, "observation/state": 7 floats, "prompt": str} and receives (H, 7) absolute joint targets. Measured inference latency on one RTX 6000 Ada after torch.compile warm-up: ~0.07 s.

Caveats

• No held-out validation set, so there is no over-fitting signal in these numbers; pick a checkpoint by real-robot evaluation.
• Every demonstration ends with the bowl on the plate but the gripper still closed (no release), so a trained policy stops while holding the bowl.
• The follower arm reaches a commanded target about 3 frames (0.1 s) later; actions are the leader arm's commands, not the achieved state.