pi05-base-p300x2

Physical Intelligence's pi-0.5 vision-language-action policy (lerobot/pi05_base: SigLIP + Gemma-2B VLM + Gemma-300M flow-matching action expert) running on two Tenstorrent Blackhole p300 boards (4 chips) via tt-nn: two camera images + a task prompt in, a 50-step chunk of normalised actions out. The SigLIP tower and the VLM prefill are tensor-parallel over the four chips, the action expert is replicated, and the whole graph runs as one Metal trace: 76 ms per chunk, 1.6× the single-chip package, PCC 0.999 vs the fp32 reference. Weights: lerobot/pi05_base · Paper: arXiv:2504.16054 · Upstream code: Physical-Intelligence/openpi · Single-chip package: changh95/pi05-base-p150

Runs on p300x2 (mesh P300x2).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/pi05-base-p300x2 --with-weights
tt-model serve changh95/pi05-base-p300x2

pull --with-weights downloads the Docker image and the lerobot/pi05_base weights at b211f3d44c36b6acfcf7ae94a64e8e96f75a64ba (into your HF cache; they are not in the image). serve starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

Run with tt-cli

tt serve changh95/pi05-base-p300x2
printf '{"images":["%s","%s"],"prompt":"pick up the cube","state":[0.1,-0.2,0.3,0,0,0,0.5,-0.5]}' \
  "$(base64 -w0 media/sample_base.png)" "$(base64 -w0 media/sample_wrist.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/pi05-base-p300x2

• POST /predict: images (1-2 base64 PNG/JPEG, [base/exterior, wrist]), prompt (task text) or tokens (pre-tokenised PaliGemma ids, ≤224); optional state (≤32 floats already normalised to [-1, 1], default zeros), seed (initial flow-matching noise; default fixed → deterministic).
• GET /health, GET /info (reports the mesh shape and the resolved tensor-parallel knobs).

Response

{"actions": [[-0.0203, -0.0197, -0.0349, 0.0669, -0.0635, 0.0442, 0.5430, 0.0625, ...], ...],
 "action_horizon": 50, "action_dim": 32, "normalized": true, "denoising_steps": 10,
 "prompt": "Task: pick up the cube, State: 140 102 166 128 ...;\nAction: ", "num_tokens": 142, "token_len": 224, "prompt_truncated": false,
 "images_used": 2, "images_padded": 0, "original_sizes": [[224, 224], [224, 224]], "image_size": [224, 224], "seed": null,
 "timing_ms": {"preprocess": 1.1, "inference": 78.0, "total": 79.1}}

• actions is 50 × 32 in lerobot's normalised QUANTILES action space, zero-padded to 32 dims: denormalise with (a+1)*(q99-q01)/2+q01 from your own dataset and slice to your action dim (e.g. the first 7 for LIBERO).
• Images are squash-resized to 224×224; state is discretised into 256 bins inside the prompt (Task: <prompt>, State: b0 … b31;\nAction: ), right-padded to 224 tokens.

Demo

Base camera (media/sample_base.png, synthetic)	Wrist camera (media/sample_wrist.png, synthetic)

How the four chips are used

part	on the 1x4 ring (FABRIC_1D_RING, ttnn.all_reduce)
SigLIP-so400m (27 blocks, 16 heads)	4 heads per chip, fc1 columns / fc2 rows sharded, 2 all-reduces per block
Gemma-2B VLM prefill (18 blocks, 8 q heads, 1 K/V head)	2 q heads + the full K/V head per chip (every chip fills a complete KV cache), o_proj / down rows and gate / up columns sharded, 2 bf16 all-reduces per block
Gemma-300M action expert (18 blocks × 10 denoising steps on the 64-row suffix)	replicated on every chip (its 64-row ops are launch-bound, so sharding would cost more in all-reduces than it saves); the host reads chip 0
whole graph	host im2col → SigLIP → VLM prefill → 10 expert steps captured once into ONE Metal trace and replayed per request; the expert's K path uses the fused ttnn.experimental.rotary_embedding_to_cache shipped in this tree

Accuracy and speed

Metric	Value
PCC vs torch reference, served shape (2×224×224, 224 tokens, 10 steps; device pass 2026-09-17)	0.9988 / 0.9986 on two random observations; 16 further random observations min 0.9201, mean 0.9846, median 0.9899 (the single-chip p150a package on its 16: min 0.8865, mean 0.9840)
Inference, in process (tests/perf/test_perf_pi05_mesh.py, warm, batch 1, 2×224×224, 224 tokens, 10 steps; traced graph)	76.1 ms per 50-step chunk (median of 100, min 75.6; 657 actions/s); traced halves: prefix + VLM 25.4 ms, expert ×10 49.5 ms
Inference, served over HTTP from this image (warm, same shape, median of 100 requests)	78.9 ms end-to-end (78.1 min, 80.4 p90); server-side inference 78.0 ms
Same graph on ONE p300 chip (this code, PI05_MESH=1x1; the p150a package: 125.9 ms)	122.7 ms · PCC 0.8995 / 0.9987 on the same two observations (the 4-way split also removes the single chip's bf16 accumulation over the 16384-wide MLP)
Determinism	traced repeats bit-exact; all four chips hold the same output
LIBERO closed loop	not run for this package: needs the lerobot/pi05_libero fine-tune, not these base weights (the p150a card's 4/5 row used that fine-tune)

Caveats

• Fixed geometry: 1-2 images squashed to 224×224 (a missing wrist camera is padded with a black image and still attended), prompt ≤224 tokens, state ≤32 floats, batch 1; num_steps is baked in at 10 and rejected per request.
• The prompt tokenizer google/paligemma-3b-pt-224 is gated (Gemma terms): accept them on huggingface.co and hf auth login on the host before tt serve, or send pre-tokenised tokens; the weights themselves are ungated but under the same Gemma terms.
• Outputs are normalised actions of the base (not task-fine-tuned) checkpoint; pi05_base ships no per-feature stats, so they are not usable on an arbitrary robot without fine-tuning (the PCC rows use random inputs).
• Needs all four chips of a 2× p300 box (hardware: p300x2, mesh P300x2); the single-chip package is changh95/pi05-base-p150. No GPU comparison was made on this box.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
• Built from tenstorrent/tt-metal main @ 975015c2f03 (v0.79.0-dev20260917) plus this port and its rotary_embedding_to_cache op, committed on the local branch changh95/pi05-p300x2 (d44292fd10e); code/ is the exact model code inside the image (code/models/experimental/pi0_5/README.md documents the design, the sweep and the measurements).

Licensing

• Weights: lerobot/pi05_base, Gemma Terms of Use (not redistributed here; fetched into your HF cache); the tokenizer google/paligemma-3b-pt-224 is gated under the same terms.
• Port and serving code (code/): Apache-2.0 headers, distributed under the same Gemma terms; derived from the single-chip port in changh95/pi05-base-p150.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published
code/ digest	2aa15d2b144ae527 (sha256, first 16 hex digits)
built	2026-09-17T04:12:04+00:00 by tt-model 0.1.0