abdulrahimmirani/pi05-piperx-pick-cube-ft02-simdata

pi0.5 fine-tune — PiPER-X pick_cube, SIM data, ft02-simdata (+LoRA on Gemma-2B)

π0.5 fine-tuned from pi05_base on the simulated replay of 49 PiPER-X teleop demonstrations of pick up the cube and put it in the cup.

Sim counterpart of ft02, and ft01-simdata's recipe plus LoRA adapters on Gemma-2B. Completes the 2×2.

Orbax checkpoints, not HF safetensors.

Recipe — expert-only + LoRA on Gemma-2B

SigLIP frozen; Gemma-2B frozen except LoRA adapters; action expert trains in full. 458.0 M of 3.381 B trainable (13.54%) = 427.93 M action expert + 27.87 M Gemma LoRA + 2.17 M projections.

Single-variable change from ft01-simdata: the 27.87 M of LoRA. Everything else identical — batch 16, 10k steps, action_horizon=30, warmup 400 → 2.5e-5 → cosine → 2.5e-6 @10k, EMA off, arm deltas / absolute gripper, same norm stats.

Trained in 5 h 14 m on 1× L40S (g6e.2xlarge).

Checkpoints

step	epochs	train loss
2000	1.7	0.0075
4000	3.5	0.0055
6000	5.2	0.0036
8000	6.9	0.0034
9999	8.7	0.0027

Loss flattens after ~6000, so compare 6000 and 9999 first.

This run has the lowest training loss of all four (0.0027) and that is not evidence it is the best policy. It also has the most trainable parameters, and on 49 episodes a tighter fit is as consistent with memorisation as with better learning.

Serving

Needs branch rahim-trc at commit a3021ff or later.

uv run scripts/serve_policy.py policy:checkpoint \
  --policy.config=pi05_piperx_sim_expert_lora \
  --policy.dir=<downloaded>/9999

hf download abdulrahimmirani/pi05-piperx-pick-cube-ft02-simdata \
  --include "9999/params/*" "9999/assets/*" "9999/_CHECKPOINT_METADATA" \
  --local-dir ./ft02sim

Why LoRA on Gemma-2B

Gemma-2B is the backbone fusing vision tokens, text and state into what the action expert reads, so adapting it can help even with a single constant prompt.

Counterpoint: π0.5's headline feature is knowledge insulation — training actions without disturbing the VLM. This relaxes that. Expect the failure mode to be brittleness on cube positions outside the training distribution, not worse loss. Worth evaluating against ft01-simdata specifically on held-out placements.

Data — sim-data-1 (GT scene 1)

The 49 real teleop trajectories replayed in simulation through GT-scenes/usd/scene1.usd — 49/49 successes (meta/sim_report.json). Same 49 episodes, same per-episode lengths, same 18,431 frames, same 30 Hz, same 7-dim schema as the real set.

Measured differences vs the real dataset:

	finding
action	bit-identical (max diff 0.0) — same commanded joint targets replayed
observation.state	differs 0.018–0.043 rad/joint — sim physics tracking those commands
delta-action spread	35–40% wider per joint — sim tracks the commands less tightly
gripper	normalized correlation 0.943; sim reports the TRUE jaw gap, so it closes ~3 mm further than the real encoder, which deflects when the jaw stalls on the cube
images	PSNR 10.4–13.9 dB vs real, sim notably brighter (external mean 0.52 vs 0.31)

gripper_open_value is kept at 0.07, byte-identical to the real runs, so any difference in results comes from the data rather than preprocessing.

Verified before training: all 49 episodes pass episodes.jsonl length == parquet rows == decoded frame counts for both cameras; frame_index 0..N−1; timestamp == frame_index/30; loads through openpi's pinned LeRobot with exact index→(episode, frame) mapping and working mid-episode video seeking.

Action space — read before evaluating

State and action are 7-dim: joint1..joint6 (rad) + gripper.

1. Gripper is normalized; the robot wants metres. Training used gripper = clip(1 - aperture_m / 0.07) → 0.0 = fully open, 1.0 = fully closed.

aperture_m = (1.0 - gripper_norm) * 0.07

Invert this and the policy looks completely broken for reasons unrelated to training.

2. Binarization threshold needs re-tuning. The ~0.22 with hysteresis value in circulation was fitted on cuRobo planner data, not this dataset.

3. Delta actions are internal. The model predicts joint deltas (gripper absolute); AbsoluteActions converts back, so the served interface is absolute joint targets in rad. Nothing to integrate yourself.

4. action_horizon=30, i.e. 1 s of actions at 30 Hz — not the 15 that the RoboLab client assumes. Executing fewer than 30 per query is fine and more reactive; you cannot get more than 30 without retraining.

Cameras

dataset key	model slot
observation.images.external (static)	base_0_rgb
observation.images.wrist (on the arm)	left_wrist_0_rgb
—	right_wrist_0_rgb — zeroed and masked

640×480 letterboxed to 224×224 (168×224 of real content, black bars top and bottom). Aspect ratio preserved, not stretched.

Prompt: pick up the cube and put it in the cup

Caveats

• No validation split. All 49 episodes are training data, so the loss curve cannot separate learning from memorising.
• Training loss carries no information about sim-vs-real transfer. The expert-only recipe reached an identical 0.0040 on both real and sim data despite the large image gap. Only rollouts distinguish these policies.
• Flow-matching loss has no absolute scale.
• Single seed per cell; small loss differences are within plausible seed variance.

The 2×2

	real data	sim data
expert-only	ft01 — 0.0040	ft01-simdata — 0.0040
+ Gemma LoRA	ft02 — 0.0032	ft02-simdata — 0.0027

Same 49 trajectories throughout; only the domain and the freeze pattern vary.