JingwuLuo/pick_and_place_cube_quick_ur3

π0.5 policies for UR3 cube pick-and-place

π0.5 policies fine-tuned on teleoperated demonstrations of a dual-arm UR3 (only the right arm moves, 30 Hz, three cameras) putting an orange, blue or pink cube on a white box, a plate or another cube. The checkpoints come in three generations, one folder per generation; each generation was trained on a larger cleaned set of recordings. Only the 107-episode set is published as a dataset (JingwuLuo/pick_and_place_cube_quick_ur3, tag v2.1); the 174- and 223-episode sets are not uploaded.

folder	training episodes	held out	steps	weights	what differs between its policies
107TrialsTrainData/	97	10	5000, 9999	raw	action-chunk label (3 policies)
174TrialsTrainData/	158	16	5000, 9999	raw	action-chunk label (3 policies)
223TrialsTrainData/	223	5	10000, 19999	EMA 0.999	joint representation and state input (3 policies, same label)

223TrialsTrainData/ is the newest generation.

The policies

223TrialsTrainData: same label, three input/output designs

All three use the curvature-resampled label (span 48, below), the same data and the same recipe; they differ in what the model predicts and what it sees. The serve flag column is required at serving time: the flag changes the model configuration that the checkpoint is loaded into.

folder	joints predicted as	state input to the model	run.py serve flags	--name
pi05_ur3_cube_quick_v4h5_curvature_s48_pp/	deltas from the observed state (grippers absolute)	none	none	pi05_ur3_cube_quick_v4h5_curvature_s48_pp
pi05_ur3_cube_quick_v4h5_curvature_s48_abs_pp/	absolute joint positions (grippers absolute)	none	--absolute-joints	pi05_ur3_cube_quick_v4h5_curvature_s48_abs_pp
pi05_ur3_cube_quick_v4h5_curvature_s48_state_pp/	deltas from the observed state (grippers absolute)	the 14-dim state as discrete tokens in the prompt	--state-input	pi05_ur3_cube_quick_v4h5_curvature_s48_state_pp

--repo-id for all three: pick-and-place-cube-quick-v4h5-curvature-s48-d0-train (the folder name under assets/ in each checkpoint).

• Delta policy (..._curvature_s48_pp): the network outputs, for each of the 10 chunk points, the joint change relative to the state in the observation; the server adds the state back. Quantile statistics are per chunk point; the q01–q99 ranges of the right-arm joint deltas are 0.10–0.36 rad.
• Absolute policy (..._abs_pp): no delta transform; the 10 points are absolute joint angles, normalised with per-point quantile statistics (q01–q99 ranges of the right-arm joints 0.36–2.6 rad). The network has no view of the state, so the joint positions come from the images alone. The server still needs observation/state: the spline that re-times the points starts there.
• State-input policy (..._state_pp): the state is normalised with its quantile statistics (right-arm joint q01–q99 ranges 0.37–2.6 rad; the gripper entries have range 1) and written into the prompt as 256-bin discrete tokens, the π0.5 convention. Left-arm state dimensions, which never change, have their normalisation range floored to 0.05 so that a near-constant dimension does not span the whole token range. Outputs are deltas as in the delta policy. The client must send an accurate observation/state with every request; a stale or offset state changes the tokens the model sees as well as the integration of the deltas.

In all three, left-arm action dimensions have a normalisation range floored to 0.05 rad (the left arm stays parked), and the gripper dimensions are absolute (0 open, 1 closed).

107 and 174 folders: three labels

The three policies of a generation differ only in how the action chunk is labelled; all predict deltas, take no state input and need no extra flag.

policy folder suffix	label	points per chunk	reach of one chunk
curvature_s48_pp	curvature-resampled, span 48 frames	10	48 recorded frames (1.6 s)
two_phase_s48_pp	two-phase: evenly spaced in time over 48 frames, one point per frame around grasp, lift and release	10	29 frames median
time_s10_pp	per-frame	10	10 frames (0.33 s)

Folder names: pi05_ur3_cube_quick_all_<suffix> (107) and pi05_ur3_cube_quick_v3_<suffix> (174). --repo-id is the assets/ folder name inside each checkpoint:

generation	curvature	two-phase	per-frame
107	pick-and-place-cube-quick-all-curvature-s48-d0-train	pick-and-place-cube-quick-all-two-phase-s48-train	pick-and-place-cube-quick-all-time-s10-train
174	pick-and-place-cube-quick-v3-curvature-s48-d0-train	pick-and-place-cube-quick-v3-two-phase-s48-train	pick-and-place-cube-quick-v3-time-s10-train

Contents of a checkpoint

<generation>TrialsTrainData/<policy folder>/<step>/
  params/                                   model weights (orbax)
  assets/<repo_id>/norm_stats.json          quantile statistics: state (14), actions per chunk point (10, 14)
  assets/<repo_id>/arc_resample.json        label config and robot block (fps 30, joint velocity limits, gripper holds)
                                            (its `keyframes` field names the keyframes file; the path was shortened to the file name)
  _CHECKPOINT_METADATA

The optimizer state is not included. In 223TrialsTrainData the params are the exponential moving average of the weights.

Serving host

• NVIDIA GPU with CUDA 12. JAX preallocates 75% of GPU memory by default; set XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 to share the GPU (tested at that setting on a 46 GB L40S).
• openpi with the patches in luo-jingw/VLA_AGENT integrations/openpi/patches/ applied, and VLA_AGENT installed in the same environment (pip install -e <VLA_AGENT>). The 107 and 174 checkpoints need commit 75f7f11 or later; the --absolute-joints and --state-input flags of the 223 policies need commit 627357e or later.
• openpi fetches the PaliGemma tokenizer (gs://big_vision/paligemma_tokenizer.model) on first use; without internet access, copy ~/.cache/openpi/big_vision/ from a machine that has it (or set OPENPI_DATA_HOME).

Download one policy:

python -c "from huggingface_hub import snapshot_download; snapshot_download('JingwuLuo/pick_and_place_cube_quick_ur3', \
  allow_patterns=['223TrialsTrainData/pi05_ur3_cube_quick_v4h5_curvature_s48_pp/19999/**'], local_dir='ckpt')"

Serve it (from the openpi root). Delta policy:

export PYTHONPATH=scripts:<VLA_AGENT>/src:<VLA_AGENT>/examples/ur3_dual_arm
python <VLA_AGENT>/examples/ur3_dual_arm/run.py serve \
  --name pi05_ur3_cube_quick_v4h5_curvature_s48_pp \
  --repo-id pick-and-place-cube-quick-v4h5-curvature-s48-d0-train \
  --data-root /unused \
  --checkpoint ckpt/223TrialsTrainData/pi05_ur3_cube_quick_v4h5_curvature_s48_pp/19999 \
  --port 8000 --cap 4 --velocity-fraction 0.5

Absolute policy: the same command with --name pi05_ur3_cube_quick_v4h5_curvature_s48_abs_pp, the matching --checkpoint folder, and --absolute-joints. State-input policy: --name pi05_ur3_cube_quick_v4h5_curvature_s48_state_pp, the matching --checkpoint folder, and --state-input. Serving a policy without the flag it was trained with processes its input and output wrongly: an absolute policy served without --absolute-joints has its absolute angles added to the state as if they were deltas.

• --name and --repo-id must be the folder name and the assets/<repo_id> name of the checkpoint (tables above); --data-root is not read when serving.
• Velocity timing: every chunk is timed so that no joint exceeds velocity_fraction × the UR3 limits (3.14 rad/s base, shoulder, elbow; 6.28 rad/s wrists) and it never runs faster than cap × the recording's pace; the arm holds still 0.50 s after a gripper close and 0.52 s after an open (measured actuation time). The teleoperation recordings never exceeded 2.5 rad/s: start with --velocity-fraction 0.5 and raise it after the first trials. --uniform --cap 1 (points at fixed frame spacing, the recording's pace) is meaningful for the per-frame policy only; see the next section.

Resampled (curvature) and two-phase policies: execute the server's targets, never the raw points

The per-frame policy's 10 points are the next 10 control frames. The curvature (every 223 policy) and two-phase policies are not:

• curvature, span 48: the 10 points lie along the next 1.6 s of path, placed by path curvature (denser per unit of joint path in turns, for example the reversal at a grasp or release, and sparser on straight moves; there is no extra density in time or at gripper events); their times are not predicted;
• two-phase, span 48: one point every 4.8 frames during gross motion and one per frame around grasp, lift and release, so a chunk covers anywhere from 10 to 48 frames.

So for these:

1. Use the run.py serve server above (vla_openpi.launch.serve). It fits a cubic spline from the observed joints through the 10 points and re-times it with velocity timing, returning actions at 30 Hz. A plain openpi serve_policy.py would return the raw points, and sending each point as one control frame jumps up to 48 frames of motion per step.
2. Keep velocity timing (the default); do not pass --uniform. Uniform timing assumes evenly spaced points, which holds for neither label.
3. The number of targets per response varies with the path (29 in our test of the curvature policy, several dozen for a long move); read len(actions) and execute at most that many.
4. Around a gripper switch, the response repeats the same target for the hold time (0.50 s close, 0.52 s open): keep sending those repeated targets; they are what makes the arm wait for the gripper.
5. Send the measured joints as observation/state; the spline starts there, so a stale state makes the first targets jump. With asynchronous execution (the arm keeps moving while the next request is in flight), drop the first d targets of the new response, d = latency × 30 frames; when the arm waits for each response, start at the first target.
6. arc_points and arc_point_frames are for inspection: the points and the frames the server placed them at.

Request and response

Client: openpi_client.websocket_client_policy.WebsocketClientPolicy(host, port).infer(obs).

key	value
observation/image	top camera, uint8 H×W×3 RGB (recorded at 240×320)
observation/left_wrist_image	left wrist camera, uint8 H×W×3 (240×424)
observation/right_wrist_image	right wrist camera, uint8 H×W×3 (240×424)
observation/state	float32 (14): left joints 0–5 (rad), left gripper 6, right joints 7–12 (rad), right gripper 13; a gripper entry is the last gripper command sent (0 open, 1 closed), not the measured opening
prompt	English instruction, e.g. Put the pink cube on the white box.

Images of any size are resized with padding to 224×224 by the server; use the same camera placement as the recording. How the state is used depends on the policy: the server of the delta and state-input policies adds it to the predicted deltas to get absolute targets; the absolute policy uses it only as the start of the spline; the state-input policy also feeds it to the model as discrete tokens. In every case the spline starts from it.

Response:

key	value
actions	float32 (N, 14): one absolute target per 30 Hz control frame; actions[j] is the target for frame t + j + 1 after the observation
arc_points	float32 (10, 14): the model's points (absolute)
arc_point_frames	the frames the points are placed at

Executing: send actions as joint position targets at 30 Hz (e.g. UR servoJ with t = 1/30 s), execute the first 4–8 targets, then query again with a fresh observation. Right gripper: close when actions[:, 13] > 0.5. Keep the left arm where it is (dims 0–6 only reproduce its parked pose). Measured on one L40S, client on the same host: 0.17 s for the first request, 0.08 s after it.

Prompts

Trained with five phrasings per episode drawn at random (Put the orange cube on the white box., Pick up the orange cube from the table and place it on top of the white box., Move the orange block onto the white box., Could you stack the orange cube on the white box?, orange cube onto white box). Name the cube's colour (orange, blue, pink) and the target (white box, plate, or another cube).

Single-move training episodes per task (107 generation):

cube → target	episodes		cube → target	episodes
orange → white box	10		orange → blue cube	8
blue → white box	8		orange → pink cube	7
pink → white box	11		blue → pink cube	4
orange → plate	8		pink → orange cube	4
blue → plate	9		blue → orange cube	3
pink → plate	8		pink → blue cube	3

Tasks onto the white box or the plate have the most data; stacking onto another cube, especially pink → blue and blue → orange, has little. The 15 multi-move episodes (stack or unstack two or three cubes) are too few to expect multi-step instructions to work.

Training

item	107 and 174	223
base	pi05_base, action horizon 10	same
trained	action expert (Gemma 300M) and the action and time projections, 430 M parameters; SigLIP and Gemma 2B frozen	same
actions	joints as deltas from the state, grippers absolute; quantile normalisation per chunk point, left-arm joints floored to a 0.05 rad range	delta policy and state-input policy as before; the absolute policy predicts joints as absolute positions
state input	none	none, except the state-input policy (256-bin discrete tokens)
optimisation	batch 32, 10,000 steps, AdamW (β2 0.95, weight decay 1e-4, clip 1.0), cosine LR 2.5e-5 → 2.5e-6 after 500 warm-up steps	same, 20,000 steps (about 10 passes over 62,410 training rows)
weight averaging	none	EMA, decay 0.999; saved params are the averaged weights
prompt	one of the episode's five phrasings, drawn at random per sample	same
hardware	one L40S per policy, about 3 s per step	one GPU per policy, 3.1 s per step

The 223 policies were trained from the same 223 episodes, with 5 held-out episodes (raw recording indices 54, 102, 122, 180, 185). Logged batch loss at the end of training (normalised space; the three designs predict different quantities and their losses are not comparable): delta 0.009, absolute 0.006, state input 0.008.