abdulrahimmirani/pi05-piperx-pick-cube-ft02

pi0.5 fine-tune — PiPER-X pick_cube (real teleop), ft02 — +LoRA on Gemma-2B

π0.5 fine-tuned from pi05_base on 49 real-hardware teleop demonstrations of pick up the cube and put it in the cup. PiPER-X (6-DOF + gripper), XR-controller teleop at 30 Hz.

This is ft01's recipe plus LoRA adapters on Gemma-2B — a deliberate single-variable change. Companion run: pi05-piperx-pick-cube-ft01.

Orbax checkpoints, not HF safetensors. No from_pretrained — download and point openpi at the directory.

ft01 vs ft02

	ft01	ft02 (this)
SigLIP vision	frozen	frozen
Gemma-2B	frozen	frozen + LoRA adapters
action expert	full (427.93M)	full (427.93M)
trainable	430.1M (12.83%)	458.0M (13.54%)
runtime	4 h 19 m	5 h 12 m
final train loss	0.0040	0.0032

Everything else is identical: same data, batch 16, 10k steps, action_horizon=30, warmup 400 → 2.5e-5 → cosine → 2.5e-6 @10k, EMA off, delta arm actions.

ft02's lower training loss is NOT evidence of a better policy. It has 28M more trainable parameters on 49 episodes, so fitting the training set more tightly is the expected outcome and is equally consistent with memorisation. Only hardware eval distinguishes them.

Checkpoints

step	epochs	train loss
2000	1.7	0.0070
4000	3.5	0.0048
6000	5.2	0.0036
8000	6.9	0.0030
9999	8.7	0.0032

Each holds params/ (weights), train_state/ (optimizer state — resume only) and assets/ (normalization statistics, required for inference).

Loss flattens after ~6000, so compare 6000 and 9999 first.

Serving

Needs branch rahim-trc of The-Robotics-Company/openpi at commit aabe501 or later — this config is not on main.

uv run scripts/serve_policy.py policy:checkpoint \
  --policy.config=pi05_piperx_teleop_expert_lora \
  --policy.dir=<downloaded>/9999

Fetch one checkpoint rather than all 45 GB:

hf download abdulrahimmirani/pi05-piperx-pick-cube-ft02 \
  --include "9999/params/*" "9999/assets/*" "9999/_CHECKPOINT_METADATA" \
  --local-dir ./ft02

Gotcha: the config carries absolute paths from the training box (assets_base_dir=/home/ubuntu/training/assets). Norm stats ship inside each checkpoint's assets/, but override those fields if config resolution fails.

Action space — read before evaluating

State and action are 7-dim: joint1..joint6 (rad) + gripper.

1. Gripper is normalized; your robot wants metres. Training used gripper = clip(1 - aperture_m / 0.07) → 0.0 = fully open, 1.0 = fully closed.

aperture_m = (1.0 - gripper_norm) * 0.07

Invert this and the policy looks completely broken for reasons unrelated to training.

2. Binarization threshold needs re-tuning. The existing ~0.22 with hysteresis was fitted on cuRobo planner sim data, not human teleop.

3. Delta actions are internal. The model predicts joint deltas (gripper absolute); AbsoluteActions converts back, so the served interface is absolute joint targets in rad. Nothing to integrate yourself.

Cameras

dataset key	model slot
observation.images.external (static)	base_0_rgb
observation.images.wrist (on arm)	left_wrist_0_rgb
—	right_wrist_0_rgb — zeroed and masked

640×480 letterboxed to 224×224 (168×224 content, black bars). Aspect preserved.

Prompt: pick up the cube and put it in the cup

Why LoRA on Gemma-2B

Gemma-2B is the backbone fusing vision tokens, text and state into what the action expert reads, so adapting it can help even though this dataset has a single constant prompt.

Counterpoint worth knowing: π0.5's headline feature is knowledge insulation — training action capabilities without disturbing the VLM. This run deliberately relaxes that. The risk is not worse loss but brittleness: failures on cube positions outside the training distribution. Worth testing explicitly against ft01.

Note get_freeze_filter() cannot express this recipe — with a LoRA paligemma variant it leaves the SigLIP tower fully trainable (467.0M). The freeze filter is hand-written to keep vision frozen.

Caveats

• No validation split. All 49 episodes are training data; the loss curve cannot separate learning from memorising.
• A previous π0.5 attempt on this task scored 0/24 with absolute action targets, with a similarly clean loss curve. Delta actions are the identified fix and are used here, but remain unvalidated on hardware as of this upload.
• Flow-matching loss has no absolute scale; 0.0032 is not meaningful alone.

Training environment

1× NVIDIA L40S (46 GB) on g6e.2xlarge (8 vCPU / 61 GB). 1.8–1.9 s/it, GPU util 99.3% mean, peak host RAM 33.3 GB. EMA is disabled because its full shadow param copy OOM-killed a 30 GB host during development.