abdulrahimmirani/pi05-piperx-pick-cube-ft01-simdata

pi0.5 fine-tune — PiPER-X pick_cube, SIM data, ft01-simdata (expert-only)

π0.5 fine-tuned from pi05_base on the simulated replay of 49 PiPER-X teleop demonstrations of pick up the cube and put it in the cup.

Sim counterpart of ft01 — identical recipe and identical trajectories; only the domain differs.

Orbax checkpoints, not HF safetensors. No from_pretrained — download and point openpi at the directory.

Recipe — expert-only

VLM frozen (SigLIP + Gemma-2B); only the action expert trains. 430.1 M of 3.353 B trainable (12.83%) = 427.93 M action expert + 2.17 M projections.

Batch 16, 10k steps (8.7 epochs), action_horizon=30, warmup 400 → 2.5e-5 → cosine → 2.5e-6 @10k, EMA off, arm deltas / absolute gripper. Norm stats computed fresh (required: deltas are action − state and the states differ from real).

Trained in 4 h 14 m on 1× L40S (g6e.2xlarge), GPU 98.9% mean.

Checkpoints

step	epochs	train loss
2000	1.7	0.0077
4000	3.5	0.0067
6000	5.2	0.0046
8000	6.9	0.0048
9999	8.7	0.0040

Each holds params/ (weights), train_state/ (optimizer state — resume only) and assets/ (normalization statistics, required for inference).

Loss flattens after ~6000, so compare 6000 and 9999 first.

Serving

Needs branch rahim-trc of The-Robotics-Company/openpi at commit f055402 or later — this config is not on main.

uv run scripts/serve_policy.py policy:checkpoint \
  --policy.config=pi05_piperx_sim_expert \
  --policy.dir=<downloaded>/9999

Fetch one checkpoint instead of all 44 GB:

hf download abdulrahimmirani/pi05-piperx-pick-cube-ft01-simdata \
  --include "9999/params/*" "9999/assets/*" "9999/_CHECKPOINT_METADATA" \
  --local-dir ./ft01sim

Gotcha: the config carries absolute paths from the training box (assets_base_dir=/home/ubuntu/training/assets). Norm stats ship inside each checkpoint's assets/, but override those fields if config resolution fails.

Note on evaluation: because this policy was trained on renders from scene1.usd, sim rollouts in that scene are in-distribution and a legitimate performance measure — unlike the real-data policies, for which sim eval would be uninterpretable.

Data — sim-data-1 (GT scene 1)

The 49 real teleop trajectories replayed in simulation through GT-scenes/usd/scene1.usd — 49/49 successes (meta/sim_report.json). Same 49 episodes, same per-episode lengths, same 18,431 frames, same 30 Hz, same 7-dim schema as the real set.

Measured differences vs the real dataset:

	finding
action	bit-identical (max diff 0.0) — same commanded joint targets replayed
observation.state	differs 0.018–0.043 rad/joint — sim physics tracking those commands
delta-action spread	35–40% wider per joint — sim tracks the commands less tightly
gripper	normalized correlation 0.943; sim reports the TRUE jaw gap, so it closes ~3 mm further than the real encoder, which deflects when the jaw stalls on the cube
images	PSNR 10.4–13.9 dB vs real, sim notably brighter (external mean 0.52 vs 0.31)

gripper_open_value is kept at 0.07, byte-identical to the real runs, so any difference in results comes from the data rather than preprocessing.

Verified before training: all 49 episodes pass episodes.jsonl length == parquet rows == decoded frame counts for both cameras; frame_index 0..N−1; timestamp == frame_index/30; loads through openpi's pinned LeRobot with exact index→(episode, frame) mapping and working mid-episode video seeking.

Action space — read before evaluating

State and action are 7-dim: joint1..joint6 (rad) + gripper.

1. Gripper is normalized; the robot wants metres. Training used gripper = clip(1 - aperture_m / 0.07) → 0.0 = fully open, 1.0 = fully closed.

aperture_m = (1.0 - gripper_norm) * 0.07

Invert this and the policy looks completely broken for reasons unrelated to training.

2. Binarization threshold needs re-tuning. The ~0.22 with hysteresis value in circulation was fitted on cuRobo planner data, not this dataset.

3. Delta actions are internal. The model predicts joint deltas (gripper absolute); AbsoluteActions converts back, so the served interface is absolute joint targets in rad. Nothing to integrate yourself.

4. action_horizon=30, i.e. 1 s of actions at 30 Hz — not the 15 that the RoboLab client assumes. Executing fewer than 30 per query is fine and more reactive; you cannot get more than 30 without retraining.

Cameras

dataset key	model slot
observation.images.external (static)	base_0_rgb
observation.images.wrist (on the arm)	left_wrist_0_rgb
—	right_wrist_0_rgb — zeroed and masked

640×480 letterboxed to 224×224 (168×224 of real content, black bars top and bottom). Aspect ratio preserved, not stretched.

Prompt: pick up the cube and put it in the cup

Caveats

• No validation split. All 49 episodes are training data, so the loss curve cannot separate learning from memorising.
• Training loss carries no information about sim-vs-real transfer. The expert-only recipe reached an identical 0.0040 on both real and sim data despite the large image gap. Only rollouts distinguish these policies.
• Flow-matching loss has no absolute scale.
• Single seed per cell; small loss differences are within plausible seed variance.

The 2×2

	real data	sim data
expert-only	ft01 — 0.0040	ft01-simdata — 0.0040
+ Gemma LoRA	ft02 — 0.0032	ft02-simdata — 0.0027

Same 49 trajectories throughout; only the domain and the freeze pattern vary.