Bohany/pi05-isaac-packing

π0.5 LoRA skills for simulated order packing (Isaac Sim)

Language-conditioned manipulation skills for a Franka Panda at an order-packing table, fine-tuned from Physical Intelligence's π0.5 (openpi). One folder per skill; inside it the openpi checkpoint layout <config>/<exp>/<step>/.

Skill	Prompt	Best checkpoint	Held-out success
pick	"pick up the X"	pick/pi05_isaac_pick3_droid/pick3_droid/23999 (v3)	64% (72% on partly cleared tables)
place / set-aside	"put the X in box A | box B | the tray"	place/pi05_isaac_place3_droid/place3_droid/7999	98%
scan	"scan the X"	scan/pi05_isaac_scan2_droid/scan2_droid/10000	83%
declutter	"separate the X"	declutter/pi05_isaac_declutter_droid/declutter_droid/5999	55% separated; pick on blocked items 35% → 60%
cube (early)	"pick up the cube"	cube/pi05_isaac_pick_lora/demos50/4999	80%

Download one checkpoint:

huggingface-cli download Bohany/pi05-isaac-packing --include 'scan/pi05_isaac_scan2_droid/scan2_droid/10000/*' --local-dir .

π0.5-DROID LoRA — "pick up the X" (Isaac Sim order packing)

Language-conditioned pick: lift the named item (one of 5 on the table) ≥ 8 cm and hold it. Success = the item's grasp point ≥ 8 cm above its start, held in the fingers, for 5 consecutive control steps within 12 s.

Recommended checkpoint: pi05_isaac_pack2c_droid/pack2c_droid/16000 (63%).

Run	Config / exp	Data	Steps
2b	pi05_isaac_pack2b_droid/pack2b_droid	1,375 demos, 100 layouts (resumed from a 5k run on 681 demos)	5k → 10k
2c	pi05_isaac_pack2c_droid/pack2c_droid	same as 2b	10k → 20k

Norm stats: from the 681-demo run (local/isaac_pack_pick2), shipped in each assets/.

Step	Held-out success (100 targets)
6000 (2b)	24%
7000 (2b)	35%
8000 (2b)	32%
9000 (2b)	44%
9999 (2b)	42%
12000 (2c)	32%
14000 (2c)	47%
16000 (2c)	63% (53–72%)
18000 (2c)	60%
19999 (2c)	60%

Pick v3 (pi05_isaac_pick3_droid/pick3_droid, resumed from 2c's 16000 to 24k): the scripted expert chooses the grasp for clearance (hand flip, ±20° tilt, any yaw for the round cans), 0–3 items start already packed (partly cleared tables), 30% recovery demos (unrecorded deliberate miss, recorded re-grasp); 1,573 new demos + the 1,375 above. Evaluated the same day as a fresh 16k baseline (repeat evaluations of one checkpoint vary by ~±9 points):

Checkpoint	Standard start (100)	Partly cleared (67)	Both
2c 16000 (re-run)	54%	55%	54%
v3 20000	51%	49%	50%
v3 23999	64% (54–73%)	72% (60–81%)	67%

Recommended pick checkpoint: pi05_isaac_pick3_droid/pick3_droid/23999.

Steps 11000, 13000, 15000, 17000 and 19000 were saved but not evaluated. The scripted expert lifts 90/100 of the evaluation targets. Weakest items at 20k: banana, potted meat can.

π0.5-DROID LoRA — "put the X in box A | box B | the tray" (Isaac Sim order packing)

Language-conditioned place: the item is already held (a scripted pick sets up the grip and is not scored); carry it to the named destination and let go. Success = the item inside the named destination's interior with the gripper open, for 5 consecutive control steps. Episodes whose scripted pick fails are excluded.

Run	Config / exp	Data	Steps
place (A/B)	pi05_isaac_place_droid/place_droid	260 demos, 30 layouts, both boxes from each grip	0 → 5k
place3 (A/B/tray)	pi05_isaac_place3_droid/place3_droid	the 260 + 135 tray demos from the same grips	5k → 8k (resumed)

The tray is order_packing's exception tray (its set_aside action); for the tray the item is lowered to 2 cm above the tray floor before release.

Checkpoint	Box A	Box B	Tray	Overall
place 3000	85/90	88/88	—	97%
place 4000	89/90	83/88	—	97%
place 4999	88/90	86/88	—	98% (94–99%)
place3 6000	87/91	85/88	89/91	97% (94–98%)
place3 7999	88/91	87/88	88/90	98% (95–99%)

Recommended: place3_droid/7999 — one model for all three destinations, as good on the boxes as the A/B-only model. place 1000 and 2000 were saved but not evaluated.

π0.5-DROID LoRA — "scan the X" (Isaac Sim order packing)

Language-conditioned barcode scan: hold the hand-mounted 1024 px scanner camera 15 cm above the named item's barcode sticker, looking down. Success = after 5 steps inside a geometric gate (scanner within the gate distance of the label in xy, ±2.5 cm in height, tilt < 10°), an OpenCV EAN-13 decode of the scanner image returns this item's code (up to 3 tries). No evaluated episode ever decoded another item's code.

Recommended checkpoint: pi05_isaac_scan2_droid/scan2_droid/10000 (83%).

Run	Config / exp	Data	Steps
1	pi05_isaac_scan_droid/scan_droid	279 demos, 60 layouts	0 → 5k
2	pi05_isaac_scan2_droid/scan2_droid	1,219 demos, 260 layouts (norm stats of run 1)	5k → 12k (resumed from run 1's 4999)

Checkpoint	Gate	Held-out success (100 scans)
run 1, 3000	1.5 cm	1%
run 1, 4000	1.5 cm	6%
run 1, 4999	1.5 cm / 2.5 cm	6% / 12%
run 2, 8000	2.5 cm	71%
run 2, 10000	2.5 cm	83% (75–89%)
run 2, 11999	2.5 cm	82%

Run 1 mostly ignored which item was named (e.g. scanned the same item for 4 of 5 prompts on a layout); 4× more data fixed it. Run 1 steps 1000/2000 and run 2 step 6000 were not evaluated. The scripted expert decodes 97/100 evaluation targets (misses: items beside the robot base, out of reach at scan height).

π0.5 LoRA — "separate the X" (declutter / singulation, Isaac Sim order packing)

Push the named item out of a cluster of touching items so that it can be grasped. Success = the pick-v3 grasp scorer finds a grasp with ≥ 15 mm clearance, the item is upright and on the table, and no other item tipped over.

pi05_isaac_declutter_droid/declutter_droid/{4000,5999}: initialised from the pick v3 24k weights (fresh optimizer and schedule), 6k steps on 915 scripted singulation pushes (gripper closed, finger axis along the push, push at ~1/3 item height; push direction/distance chosen for clearance over 16 directions × 4–14 cm) on herded clusters of 4–5 items. Pick's norm stats.

Checkpoint	Blocked items separated (held-out clusters)
scripted push (ceiling)	69%
4000	39%
5999	55%

Pick v3 on blocked items, same start: directly 35% → after this declutter policy 60% (paired McNemar p = 0.004). Recommended: declutter/pi05_isaac_declutter_droid/declutter_droid/5999.

π0.5-LIBERO LoRA — pick up a cube (Isaac Sim, early experiments)

The first fine-tuning experiments: a single cube on a table, Franka Panda, interface A (15 Hz TCP deltas, see below). Initialised from Physical Intelligence's pi05_libero (not pi05_droid), robot state input on, own norm stats.

Checkpoint	Data	Steps	Held-out success
pi05_isaac_pick_lora/demos50/4999	50 scripted demos (local/isaac_pick_cube_50)	5k	80% (40/50, 95% CI 67–89%), held-out cube poses (seeds 10000+)
pi05_isaac_pick_lora/trial/19	—	20	pipeline smoke test only, not a policy

Success = cube held ≥ 8 cm up for 5 consecutive control steps within 8 s. Failures were all grasps that did not hold, mostly at the far edge of the cube range.

Setup common to these checkpoints (unless the card above says otherwise)

• Model: π0.5 (openpi), LoRA fine-tune (gemma_2b_lora VLM + gemma_300m_lora action expert, frozen base), batch 32, cosine LR (warmup 100, peak 2.5e-5, floor 2.5e-6), no EMA. Initialised from Physical Intelligence's pi05_droid unless stated otherwise.
• Simulator: Isaac Sim, Franka Panda on an order-packing table (5 YCB items: sugar box, tomato soup can, mustard bottle, tuna can, pudding box, gelatin box, potted meat can, banana; boxes A and B; an exception tray).
• Observation: 224 px scene camera (front-side diagonal) + 224 px wrist camera + the language prompt. Robot state input is off (discrete_state_input=False).
• Action (interface A): 15 Hz, 7-D [dx, dy, dz, rx, ry, rz, gripper], world-frame TCP deltas, a = 1 ↔ 0.02 m / 0.15 rad; gripper > 0 = close.
• Data: scripted-expert demos, contrastive — every target of a layout starts from the identical scene, so the prompt is the only thing that differs.
• Evaluation: 20 held-out layouts (seeds 1000–1019), never seen in training; 95% Wilson intervals. Success definitions are per skill below.
• Checkpoint layout: <config>/<exp>/<step>/ as written by openpi (orbax): params/ (weights, needed to run), assets/ (norm stats, needed to run), train_state/ (optimizer state, only to resume training).
• Serving: the configs live in a local openpi fork (src/openpi/training/config.py, helper _isaac_pick_1k); e.g. python scripts/serve_policy.py policy:checkpoint --policy.config=<config> --policy.dir=<step dir>.

License

Derived from pi05_droid (openpi), which builds on Google's PaliGemma / Gemma. Use and redistribution are subject to the Gemma Terms of Use and its Prohibited Use Policy, and to the openpi license. Research checkpoints trained in simulation only — not validated on hardware.