xvla-base-blackhole

X-VLA (lerobot/xvla-base), the cross-embodiment vision-language-action foundation model (Florence-2 encoder + 24-layer SoftPromptedTransformer, flow matching), on a single Tenstorrent Blackhole p150a via tt-nn: 3 camera views + a language instruction + proprio state in, a 30-step x 20-D end-effector action chunk out. The transformer blocks, the Florence-2 BART encoder and the DaViT FFN / channel- / window-attention run on the chip; 181 fps (action steps/s) vs 10.75 fps torch-CPU fp32 (16.9x), PCC 0.99998 vs fp32 at 10 denoising steps, open-loop MAE delta +0.00% on lerobot/pusht_image. This is the BASE checkpoint (meant for fine-tuning): outputs are raw model-space actions. Weights are lerobot/xvla-base (Apache-2.0); the port code is Apache-2.0 from github.com/changh95/tt-XVLA @ a96e1f451d3dc520cf351f733bd7300572be8851 and depends on lerobot 0.5.0 (Apache-2.0); the facebook/bart-large tokenizer files (Apache-2.0) are vendored under code/tt/assets/.

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/xvla-base-blackhole --with-weights
tt-model serve changh95/xvla-base-blackhole

pull --with-weights downloads the Docker image and the lerobot/xvla-base weights at cdb7964e4fe842935d671bfab5a5ebe00a96648c (into your HF cache; they are not in the image). serve starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

With tt-cli

tt serve changh95/xvla-base-blackhole          # pulls the image + 3.5 GB weights on first use
tt model stop changh95/xvla-base-blackhole

The server listens on the port serve printed (20000, or the next free one). It is not an OpenAI-style API: tt-model curl and the /v1/models hint on the ready card do not apply (GET /v1/models answers with a one-entry list only so they do not 404). The real routes are below.

Routes

curl -s localhost:20000/health      # {"status": "ok" | "starting", "model": "xvla-base", "device": ...}
curl -s localhost:20000/info        # weights repo + revision, source commit, I/O contract, limits
curl -s -X POST localhost:20000/reset   # clears the policy's action queue (stateless /predict does not need it)

POST /predict takes 1 or 3 base64 PNG/JPEG camera views, an instruction and the proprio state, and returns one action chunk:

IMG=$(base64 -w0 media/pusht_synthetic.png)
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d "{
  \"images\": [\"$IMG\", \"$IMG\", \"$IMG\"],
  \"instruction\": \"push the T\",
  \"state\": [0, 0, 0, 0, 0, 0, 0, 0],
  \"num_denoising_steps\": 1,
  \"seed\": 42
}"

Request fields: images (list of 1 or 3 base64 PNG/JPEG; slots image, image2 are the third-person cameras the base checkpoint saw at 256x256, image3 the 224x224 wrist camera; one image is copied into all three slots; any resolution -- each view is ImageNet-normalised and resize_with_pad-ed to 224x224 exactly as lerobot does), image (legacy single view), instruction (or task; tokenised with facebook/bart-large to a fixed 32 tokens, right-padded), state (1..20 floats, default 8 zeros; zero-padded to 20), domain_id (0..29, default 0), num_denoising_steps (1..50, default 1 -- the validated speed setting; upstream uses 10), seed (optional; seeds the flow-matching noise so a repeated request reproduces the chunk).

Response:

{
  "actions": [[20 floats], ... 30 rows],
  "chunk_size": 30, "n_action_steps": 30, "action_dim": 20, "action_space": "ee6d",
  "normalized": false,
  "note": "raw model-space actions from the base checkpoint ...",
  "num_denoising_steps": 1, "seed": 42, "domain_id": 0, "instruction": "push the T",
  "language_tokens": 32, "state_dim": 8,
  "views_used": ["image", "image2", "image3"], "single_view_replicated": false,
  "input_size": [224, 224], "image_sizes": [{"width": 256, "height": 256}, ...],
  "timing_ms": {"preprocess": 12.3, "inference": 170.5, "total": 182.8}
}

actions[t] = [x, y, z, r1..r6 (6-D rotation), gripper, 0 x 10] in the model's own units: no dataset statistics ship with the base checkpoint, so the MEAN_STD un-normaliser is the identity and the values are not directly usable on a robot (fine-tune first). Errors: 400 for bad input (2 views, >20 state dims, undecodable image, missing instruction), 503 while starting, 500 with the exception text otherwise.

A stdlib-only end-to-end check lives at code/tt/server/smoke_test.py --url http://127.0.0.1:<port> (3 copies of media/pusht_synthetic.png + "push the T" + zero state -> finite 30x20 chunk, seed determinism, single-view path, /reset).

First boot

serve downloads lerobot/xvla-base @ cdb7964e (3.5 GB) into your HF cache, then the container loads the fp32 checkpoint, casts it to bf16, uploads 500 M parameters to the chip (bfp8_b for the MLP weights) and runs one warm-up chunk, which JIT-compiles the kernels into `/.cache/tt-model/xvla-base-blackhole/cache -- expect several minutes the first time and much less afterwards. READY (Application startup complete) means the warm-up is done. The tokenizer needs no network (vendored). Stop with tt-model stop changh95/xvla-base-blackhole` (SIGTERM closes the device cleanly).

Environment switches (all optional, set via the manifest's serve.env): XVLA_NUM_DENOISING_STEPS (default 1), XVLA_LANG_TOKENS (32; the port validated 16 and 32 -- longer merged sequences must stay under the model's 512), XVLA_WARMUP (1), TT_DEVICE_ID (0), XVLA_WEIGHTS_DIR (a local snapshot dir instead of the Hub), XVLA_TOKENIZER_PATH.

What runs where

Component	Params	Location
SoftPromptedTransformer (24 layers, hidden=1024)	~308 M	Blackhole p150a
Florence-2 BART encoder (12 layers)	~208 M	Blackhole p150a
Florence-2 DaViT FFN modules (24)	--	Blackhole p150a
Florence-2 DaViT ChannelAttention (12)	--	Blackhole p150a
Florence-2 DaViT WindowAttention (12)	--	Blackhole p150a
DaViT ConvEmbed + DepthWiseConv2d	~30 M	torch CPU
Token / positional embedding lookups, window partition/reverse, flow-matching bookkeeping	--	torch CPU

Numerics on-chip: bf16 activations; bfp8_b weights on the transformer MLPs and DaViT FFN MLPs; attention weights stay bf16. Manual SDPA (matmul + softmax) is used because the fused kernel needs sequence lengths padded to 32.

Results (from the port's README; Zen 5 host, single p150a)

Benchmark (benchmark/run_benchmark.py, synthetic inputs, 1 denoising step): 181.42 fps vs 10.75 fps torch CPU fp32 (16.87x); PCC vs the cached fp32 reference 99.98%. Throughput depends on the host CPU too: DaViT ConvEmbed, embeddings, window partition and every host<->device transfer run in torch on the CPU.

Relative PCC vs torch fp32 (eval/eval_relative_pcc.py, 5 seeds, 10 steps):

backend	mean PCC	min PCC	mean |err|/std(ref)	max abs err
torch_cpu	1.000000	1.000000	0.00e+00	0.00e+00
ttnn	0.999983	0.999983	4.66e-03	4.57e-03

Open-loop dataset eval (eval/eval_gt_dataset.py, lerobot/pusht_image, 10 samples, 10 steps, single camera copied into all 3 views, --skip-postprocess):

backend	MAE	delta vs fp32
torch_cpu	2.5378e+02	--
ttnn	2.5378e+02	+4.27e-04 (+0.00 %)

The absolute MAE is large because the base checkpoint was never fine-tuned on pusht; the signal is the backend delta, which is ~0.

Optimisation trajectory (20 autoresearch iterations, keep if faster and PCC >= 99%): bf16 dtype 10.75 -> 36.65 fps; denoising steps 10 -> 1: 100.57 fps; 24 transformer blocks on chip: 116.75; bfp8_b MLP weights: 118.40; BART encoder on chip: 118.74; DaViT FFNs: 137.72; ChannelAttention (4-D ttnn ops only): 144.27; WindowAttention: 181.42. Discarded: torch.compile on CPU, bfp8 on attention projections, bfp4 MLPs, LoFi, explicit core grids, single-op offloads that do not amortise PCIe.

Caveats

• Single chip only (TT_MESH_SHAPE must be 1x1); one request at a time (a lock serialises the device).
• num_denoising_steps=1 is a real accuracy trade-off (<0.05% open-loop PCC, but closed-loop flow-matching policies usually want 4-10 steps); override per request.
• Fixed shapes: 3 views at 224x224, 32 language tokens. A different XVLA_LANG_TOKENS re-JITs kernels on the next boot.
• lerobot 0.5.0 is required (0.6 removed the in-repo Florence-2 the on-chip swaps target); it drags in datasets, torchvision, av, pyarrow, opencv as hard imports. numpy is 2.2.x inside the image (see runtime comments in the manifest).
• Open items from the port: DaViT ConvEmbed/depthwise conv still on CPU; fused SDPA blocked on 32-padding; no closed-loop simulator evaluation yet.

Licensing

Weights: lerobot/xvla-base, Apache-2.0, fetched from the Hub at serve time (not redistributed here). Port code: Apache-2.0, written by Hyunggi Chang (github.com/changh95/tt-XVLA @ a96e1f45), which monkey-patches the lerobot 0.5.0 X-VLA policy (Apache-2.0; Florence-2 code by Microsoft, MIT, as vendored in lerobot). code/tt/assets/bart-large-tokenizer/ is copied from facebook/bart-large (Apache-2.0) at cb48c1365bd826bd521f650dc2e0940aee54720c.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	2a6ddd8e572bb09b236a2adbd3afab1153e0a17e
code/ digest	f46cae534c35e55b (sha256, first 16 hex digits)
built	2026-09-12T05:59:02+00:00 by tt-model 0.1.0