RLDX-1-PT-p150

RLWRLD RLDX-1 PT (vision-language-action policy: Qwen3-VL-8B backbone truncated to 18 layers with 64 cognition tokens, FLUX-style MM-DiT "MSAT" action head, 4 Euler steps) running on one Tenstorrent Blackhole p150a via tt-nn, in the DROID layout: three cameras × four frames (t−6, t−4, t−2, t) at 320×180 with the layer-4 visual-token compression (VTC: the 9 older images pooled into one token), a 7-dim end-effector / gripper state and a task instruction in, a 16-step chunk of DROID end-effector deltas and gripper targets out. The ViT and the LLM are TTNN ops in Metal traces; the MSAT denoise runs as one persistent megakernel streaming bfp8 weights from a DRAM arena. tt-metal's fast-dispatch kernels run on the chip's idle Ethernet cores, which frees the full 12×10 worker grid for the model. Weights: RLWRLD/RLDX-1-PT (RLWRLD Model License v1.0, non-commercial) · Paper: arXiv:2605.03269 · Upstream code: RLWRLD/RLDX-1 (v1.0.2) · Port: code/models/experimental/rldx1 and the server code/rldx1_p150 in this repo

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/RLDX-1-PT-p150 --with-weights
tt-model serve changh95/RLDX-1-PT-p150

• Weights RLWRLD/RLDX-1-PT at 1592013ab955 (three bf16 safetensors shards + the processor json files) go to your Hugging Face cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete (the first boot converts the weights and JIT-compiles the kernels, see Caveats).

Run with tt-cli

tt serve changh95/RLDX-1-PT-p150
curl -s localhost:20000/demo | python3 -c 'import json,sys; json.dump(json.load(sys.stdin)["request"], open("req.json","w"))'
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/RLDX-1-PT-p150

• POST /predict takes the RLDX RLDXPolicy observation: video ({"primary": [<base64 PNG at t−6>, <t−4>, <t−2>, <t>], "secondary": [...], "wrist": [...]}, four frames (t−6, t−4, t−2, t) per camera as base64 PNG/JPEG or nested uint8 lists (1, 4, H, W, 3)), state (end_effector_position 3, end_effector_rotation 3, gripper_position 1, raw DROID units), language (≤ 60 BPE tokens after RLDX's formalize_language); optional seed (default 1000) or noise ([16][64]), return_normalized.
• GET /health, GET /info (the full observation contract, the policy actually served, warm-up fidelity), GET /demo (the shipped demo request with its reference noise and the fp32 reference actions), GET /v1/models (stub). RLDX's own ZeroMQ clients (PolicyClient, open_loop_eval.py, rollout_policy.py) connect through the bridge: python -m rldx1_p150.server.zmq_bridge --url http://localhost:20000 --port 5555 (same msgpack codec; ping, get_action, reset, get_modality_config, kill).

Response

{"actions": {"end_effector_position": [[<3 floats>], "... 16 steps"],
             "end_effector_rotation": [[<3 floats>], "... 16 steps"],
             "gripper_close":         [[<1 float>],  "... 16 steps"]},
 "action_horizon": 16, "action_keys": ["end_effector_position", "end_effector_rotation", "gripper_close"],
 "embodiment": "droid", "embodiment_id": 27, "variant": "pt", "n_text": 8, "seq_len": 581, "s_pad": 640,
 "noise_source": "client", "seed": null,
 "timing_ms": {"decode": 9.7, "encode": 2.8, "device": 71.4, "decode_actions": 0.1, "total": 84.1}}

• actions are physical, un-normalised DROID targets exactly as RLDXPolicy.get_action returns them: end_effector_position / end_effector_rotation are delta end-effector actions (the DROID RLDS cartesian_velocity space, in [−1, 1]), gripper_close is the absolute gripper target in [0, 1]; 16 steps × [3, 3, 1]; (clip(a, −1, 1) + 1) / 2 · (q99 − q01) + q01 with the checkpoint's processor/statistics.json, no relative → absolute composition.
• The example is the demo request with the reference noise, as GET /demo returns it (noise_source: "client", seed: null); timing_ms are the medians of the warm container bench (publish/logs/container-pt-20260921-232046.json, 50 requests). device covers the input writes, the three trace replays (V, L, D) and the readback.

Demo

Input (media/demo_frames.png: the three cameras × four frames (t−6 … t; secondary := primary); DROID demo episode 1, step 40, 320×180)	Instruction	p150a actions vs the fp32 reference (media/demo_actions_vs_golden.png)
	Put the blue block in the green bowl

What runs where

Stage	Where	What	dtype (plan D18; admission JSON reference/out/bfp_emulation_pt.json)
Host encode	CPU, outside the device lock	AspectAreaResizeAndCrop (320×180 → 160×284 → centre crop 160×256), Qwen2-VL patching (16 px patches, 2×2 merge → 40 tokens per image), tokenizer, static slot layout, mRoPE tables, additive mask, state q01/q99 normalisation, the ≤ 60 instruction embedding rows gathered from the safetensors shards	fp32 → bf16 uploads
Trace V	device	Qwen3-VL vision tower as the port's tuned fork: 27 blocks on 12 × 160 patches, 3 deepstack mergers + final merger → 40 × 4096 tokens per image	ViT bf16 HiFi2 fp32-accumulate (admitted: bfp8 buys nothing on the compute-bound ViT)
Trace L	device	token splice (constant skeleton + text rows + ViT rows), 18 Qwen3 layers with deepstack adds after layers 0/1/2 (layers 0–3 on the 640-slot layout, the layer-4 selection-matmul compression to 256 slots, layers 4–17), final RMSNorm, the 64 cognition tokens, the DROID state encoder	LLM matmul weights bfp8_b (D19 admission: 0.94539 min per-dim-gated action_pred_valid PCC vs fp32 on 5 emulated samples, admitted = yes); activations bf16; norm statistics fp32
Trace D	device	the MSAT flow-matching head — 4 double blocks + 8 single blocks + output head, all 4 Euler steps — as one persistent ttnn.generic_op megakernel streaming the block weights from a DRAM arena; 5 wrapper ops (noise copy, resident buffers, descriptor rebind, action readback)	arena bfp8_b (admission 0.97261, admitted = yes); tails (proj_out_*, encoders / decoder slot 27, biases, norm vectors, cog_emb, tables) bf16 HiFi4, never bfp8; x_t fp32 with bf16 rounding per step (matches the reference's bf16 Euler update)
Host decode	CPU	action_pred[:16, :7] → q01/q99 de-normalisation → the three groups	fp32

Release policy (GET /info -> policy): the release LLM+MSAT bfp8 combination was emulated on the GPU first (0.94811 min action_pred_valid per-dim PCC, 0.99379 min pred_velocity PCC, worst margin 0.0007 above the gate on pt_droid_real/0:canonical_ep1_s40; admitted = yes) and then confirmed on the device on every golden sample (table below). RLDX1_MSAT_BACKEND=ttnn serves the bring-up MSAT (the same head as a sequence of TTNN ops; the device oracle of the megakernel).

Dispatch on the Ethernet cores. tt-metal's fast-dispatch kernels (prefetcher and dispatcher) normally occupy one column of Tensix cores, which leaves 11×10 worker cores on the p150a. This image opens the device with DispatchCoreType.ETH instead: the dispatch kernels run on the idle Ethernet cores and that column returns to compute, so every op of traces V / L / D is laid out on a 12×10 worker grid (program-config tables tt/tables/grid_12x10; the megakernel keeps its proven 11×10 core map and leaves the twelfth column idle during trace D). It needs two upstream changes on top of tt-metal main 58648d44443 that are built into the image: tenstorrent/tt-metal#57142 (Ethernet-core dispatch on Blackhole boards with harvested Ethernet channels: the core-descriptor map and the idle-Ethernet kernel budget) and tenstorrent/tt-metal#53988 (the program-config command sequence split at fetch-entry boundaries, which the megakernel's launch needs). Measured on that runtime with the port benchmark, same policy, same observation: 76.3 ms end-to-end under Ethernet dispatch vs 80.7 ms with the previous Tensix dispatch (trace V 27.4 vs 30.6 ms, trace L 27.4 vs 29.1 ms, trace D 16.0 vs 16.0 ms; tt-metal/models/experimental/rldx1/benchmarks/results/bench_e2e_pt_20260921-214141.json); the actions are checked against the same fp32 goldens under both. GET /info -> device reports the grid and dispatch type actually opened (compute_grid [12, 10], dispatch_core eth); RLDX1_DISPATCH_CORE=tensix restores the previous 11×10 configuration.

Layout lock

One embodiment, one static layout, batch 1 — everything else is refused with 400 before the device is touched:

item	served value
embodiment	DROID (EmbodimentTag.OXE_DROID, projector slot 27); statistics.json["droid"] q01/q99 for state and actions
cameras / video_length	primary, secondary, wrist — four frames (t−6, t−4, t−2, t) each (video delta_indices [-6, -4, -2, 0], oldest first)
frame size	any frame whose AspectAreaResizeAndCrop (area budget 65,536 px, multiples of 32) lands on 160×256 — the native DROID 320×180 frames do; one 10×16 patch grid = 40 merged tokens per image
instruction	≤ 60 BPE tokens after formalize_language (60 right-padded text slots; longer → 400); the empty instruction is accepted
LLM sequence	S_pad 640 (layers 0–3) → 256 (layers 4–17, after the visual-token compression): prefix 3 + 60 text + 12 × 42 image slots + <|im_end|>, \n, 64 cognition tokens + 7 tail pads = 640; after the layer-4 compression 3 + 60 + 1 pooled + 3 × 42 + 2 + 64 = 256
state / actions	state 7 (end_effector_position 3, end_effector_rotation 3, gripper_position 1) → padded 64; actions 7 of 64, horizon 16
noise	[1, 16, 64], torch.randn on a CPU generator seeded with seed (default 1000, which reproduces the goldens' noise_x0), rounded to bf16; or sent explicitly

Accuracy and speed

Metric	Value
Action chunk vs the fp32 reference (host serve, demo observation, same initial noise; publish/logs/smoke-host-pt.json)	PCC(actions) 0.99977 · max|d| 0.0244 · end_effector_position 0.99954 · end_effector_rotation 0.99602 · gripper_close max|d| 0.0017 (near-constant group) · normalised valid slice 0.99967
Every golden sample, every tap (WP-A test_tt_multisample, tt-metal/models/experimental/rldx1/tests/tt/results/test_multisample_pt_20260921-213332.json; acceptance document tt-metal/models/experimental/rldx1/tests/tt/results/acceptance_pt.md)	674 rows over 6 DROID samples (real + budget-edge synthetic), 51 below gate; tightest full-tensor margin -0.000067 on vit_block_12; the per-class minima are in the table below
Official bf16 GPU path vs the same fp32 reference (the floor any bf16/bfp8 port is measured against; tt-metal/models/experimental/rldx1/tests/tt/gates/gates_bf16_pt.json)	min action_pred_valid PCC 0.99779 over the golden samples
Traced vs untraced, determinism, stale state	traced replays bit-identical (tt-metal/models/experimental/rldx1/tests/tt/results/test_determinism_pt_20260921-213600.json); A→B→A→B alternation FAILS (tt-metal/models/experimental/rldx1/tests/tt/results/test_stale_state_abab_pair0_pt_20260921-212953.json)
Inference on p150a (port benchmark bench_e2e, warm, batch 1, median of 50 traced replays after 10 warm-ups; tt-metal/models/experimental/rldx1/benchmarks/results/bench_e2e_pt_20260921-214033.json)	76.3 ms end-to-end (p90 76.8) = host encode 3.5 + upload 2.1 + traces 70.4 (V 27.4 · L 27.4 · D 16.0 = 4 × 4.00) + readback and decode 0.25; device (upload → readback) 72.6 (p90 73.0). Plan D27 release gate 85 ms (target 80): meets
Served over HTTP by this image (container, warm boot, batch 1, 50 requests after 5 warm-ups; publish/logs/container-pt-20260921-232046.json)	71.4 ms device (p90 71.6) · 84.1 ms server-side incl. base64 + 12 PNG decodes (p90 84.5) · 88.6 ms client wall on localhost
Same policy on an RTX 5090 (same host, RLWRLD's unmodified RLDXPolicy, eager PyTorch bf16 + sdpa, stock flags, same observation; reference/out/gpu/gpu_baseline_pt.json)	RLDXPolicy.get_action 96.7 ms e2e (p90 99.5), model 75.9, backbone 48.4, action head 23.5 → p150a 1.27× end-to-end, 1.04× model-only, 1.47× on the action head. RLWRLD's published RTX 5090 numbers (2 cameras, their stack): eager 191.2 ms, CUDA-graph 33.7, custom Triton chain 25.2 — different environment and input, see GPU_COMPARISON.md
Boot (container; publish/logs/container-pt-20260921-232046.json) and clock	cold (empty weight and kernel caches) 65 s, warm 22 s from tt-model serve to READY; AICLK 800 MHz before / 1325 after the benchmark, 1225 after 400 back-to-back replays of trace V (sustained-clock proxy)

Fidelity per tap class, TT vs the fp32 reference over every golden sample (min full-tensor PCC; gates frozen from the official bf16 path's own spread per plan D23, gate = min(0.99999, 1 − 3·(1 − min PCC_bf16)); PT gates are per sample (D23), because the canonical PT sample's own bf16 floor is visibly lower than the block-level samples' (GOLDENS §5.2)):

tap class (golden_tap_map.json)	min PCC TT vs fp32 (tap, sample)	its gate	official bf16 vs fp32, min over samples
vit — 27 ViT blocks, deepstack mergers, final merger	0.99858 (vit_merger (canonical_ep1_s40))	0.99403	0.99801
llm — 18 LLM layers, final norm, cog_features	0.99901 (llm_layer_0 (short1))	0.99545	0.99849
feature — state / action encoders, temb, x_t, decoder rows	0.99994 (llm_pooled_token (short1))	0.99995	0.99595
msat — double / single block outputs per Euler step	1.00000 (proj_out_1_3 (canonical_ep1_s40))	0.99999	0.99495
action — pred_velocity_k	0.99970 (pred_velocity_3 (short1))	0.98137	0.99587
output — action_pred_normalized, action_pred_valid, unnormalized_actions.end_effector_*	0.99283 (unnormalized_actions.end_effector_rotation (at_budget))	0.99900	0.96726
output_abs — unnormalized_actions.gripper_close (near-constant: abs-only)	max|d| 0.0130	tol 0.0110	max|d| 0.0051

Per-dimension action_pred_valid PCC over the horizon, min over samples (pos x, y, z · rot x, y, z · gripper; gated at 0.9190 where the fp32 range on that sample is ≥ 0.25, the gripper dimension abs-only): 0.9736 · 0.9902 · 0.9815 · 0.9701 · 0.9940 · 0.9797 · 0.9764. Physical max|d| over samples: end_effector_position 0.0249 (tol 0.1200), end_effector_rotation 0.0254 (tol 0.1500).

What differs from the reference

The device model is a static re-implementation of RLWRLD's RLDXPolicy (RLDX-1 v1.0.2) for one layout; these are the deliberate differences (plan decision ids in brackets; each is gated against the fp32 reference above):

• [D3] Static slot table instead of dynamic tokenisation. Every call uses the same token slots: <|im_start|> user \n, 60 right-padded text slots (real tokens first, pad id 151643 after, masked as keys), 12 × [<|vision_start|> + 40 <|image_pad|> + <|vision_end|>], <|im_end|>, \n, 64 cognition slots, S_pad 640 (layers 0–3) → 256 (layers 4–17, after the visual-token compression). Real tokens keep their reference mRoPE positions; pad query rows take the last real text position (finite, never read). Anything the table cannot hold is a 400, never a re-trace.
• [D5] No 1.26 GB embedding table on the device. A constant device skeleton (special-token rows and the cog_emb rows at their slots, zeros elsewhere) is completed per call by ≤ 60 instruction rows (text_rows, gathered on the host from embed_tokens through safetensors slices with an LRU per token id) and by the ViT rows placed into the image slots; deepstack features are added after layers 0/1/2 the same way. The 35 unused embodiment slots and lm_head are never converted.
• [D7] Visual-token compression as one selection matmul. The reference averages the 9 older images' tokens (378 slots incl. their markers) into one token at the input of layer 4 (hidden_states 581 → 204). Here h_B = Sel @ h_A with a constant [256, 640] selection matmul under HiFi4 with fp32 accumulation and fp32 output; the pooled row is multiplied by fp32 1/378 before the typecast to bf16, the kept rows are exact copies (bf16 × 1.0 in fp32 accumulation); the regime-B mask and RoPE tables are built on the host with the same index map (the pooled token takes slot 63's position).
• [D10] Tuned fork of the ViT instead of the verbatim tower. The Qwen3-VL tower runs as a generalised fork of the GR00T port's tuned tower: images as the batch dim, one batched SDPA over the 12 × 160 patches with an explicit 72**-0.5 scale, fused GELU_TANH in fc1, RoPE on the 96-padded heads with an identity tail (the golden tower is 16 × 72 wide), mergers around a ROW_MAJOR reshape. The verbatim tt-nn Qwen3VlVisionModel is kept as the oracle only (erf GELU in its blocks; RLDX1_VIT_PATH=verbatim), never traced for serving.
• [D18] Per-class dtype policy. ViT bf16 HiFi2 with fp32 accumulation; LLM matmul weights bfp8_b and the MSAT arena bfp8_b because their D19 admission passed (emulated on the GPU, confirmed on the device — both JSONs cited above; bf16 is the automatic fallback of a rejected class); every tail (proj_out_1/2, encoders / decoder slot 27, biases, norm vectors, cog_emb, RoPE tables, the selection matrix) bf16 HiFi4, never bfp8; activations bf16; norm statistics fp32; every matmul with fp32_dest_acc_en; the additive attention mask uses -1e9, never −inf; x_t is kept in fp32 with the reference's bf16 rounding per Euler step.

Caveats

• One model per process, one layout, batch 1. The traces are shape-locked to the DROID layout above (12 images × 160 patches, S_pad 640 (layers 0–3) → 256 (layers 4–17, after the visual-token compression), embodiment slot 27, video_length 4); the 35 other embodiment slots of the checkpoint have no traces here. The server builds exactly one model per process (a second megakernel model on the same open device is not supported by design).
• No memory, no real-time chunking. RLDX's options (memory / RTC) are accepted and ignored; reset returns {"cleared_sessions": []}. Actions are a deterministic function of (observation, seed | noise): two identical requests return bit-identical chunks.
• secondary := primary in the demo. Every local DROID sample has only exterior_1_left and wrist_left; the shipped demo (and every golden, benchmark and GPU row) feeds the primary frame as secondary too. A real three-camera DROID rig will differ from the demo numbers; the layout itself is unchanged.
• Host encode p90 above p50 on unseen vocabulary. The instruction's embedding rows are gathered from the safetensors shards the first time a token id is seen (then cached): p50 3.5 ms vs p90 4.0 ms in the port benchmark (host_encode, tt-metal/models/experimental/rldx1/benchmarks/results/bench_e2e_pt_20260921-214033.json; 0 cold ids during the timed calls). A new instruction costs a few extra milliseconds on its first call only.
• First boot is cold: the checkpoint is converted to the per-class device layout into /weight-cache and every kernel including the megakernel is JIT-compiled into /cache — 65 s measured for this image, 22 s with warm caches (both under ~/.cache/tt-model/rldx-1-pt-p150/).
• Megakernel path. Trace D replays one generic_op for all four Euler steps; the device is opened with a 96 KiB worker-L1 cut (worker_l1_size 1,362,944 B) for the kernel-config ring the megakernel binaries need, and the MSAT block internals are not observable from the served model (the ttnn-op MSAT, RLDX1_MSAT_BACKEND=ttnn, is the A/B path and exposes them).
• Not an OpenAI-compatible API; GET /v1/models is a stub. Validated on tt-metal main 58648d44443 + tenstorrent/tt-metal#57142 + #53988 (b8e73f0755c9) with the fast-dispatch kernels on the Ethernet cores, single p150a only. Details: SERVING.md, GPU_COMPARISON.md.

Licensing

• Weights: RLWRLD/RLDX-1-PT @ 1592013ab955, Licensed under the RLWRLD Model License v1.0 — non-commercial use only (research, education, personal or evaluation), share-alike (derivative models only under this licence), attribution to RLWRLD, complete licence copy required with any redistribution; the copy is LICENSE in this repo. The weights are not redistributed here: tt-model serve fetches exactly the files in tt-model.yaml into your Hugging Face cache and converts them on your machine. Changes made: tt-nn re-implementation of the network for one Blackhole p150a; bf16 → bfp8 device layout for the admitted weight classes (LLM matmuls, MSAT arena); static DROID layout (fixed slot table, 60-token instruction budget, one embodiment); MSAT denoise as one megakernel.
• Vendored RLDX-1-VLM configuration / tokenizer files (code/rldx1_p150/assets/vlm/, RLWRLD Model License v1.0; derived from Qwen/Qwen3-VL-8B-Instruct, Apache-2.0) and the demo's reference actions (code/rldx1_p150/demo/pt/expected.json, outputs of the model) are covered by the same licence; hashes in the PROVENANCE.md next to them.
• Port and serving code (code/models/experimental/rldx1, code/rldx1_p150, code/scripts): Apache-2.0, © 2026 Tenstorrent; copied GR00T-port modules keep their Apache-2.0 headers and are listed with sha256 in code/models/experimental/rldx1/PROVENANCE.md. Because the code only runs the RLWRLD-licensed weights, the package as a whole is offered for non-commercial use. Details and the third-party list: LICENSE-NOTICE.md.
• Citation: RLDX-1 Technical Report, arXiv:2605.03269; upstream code github.com/RLWRLD/RLDX-1 v1.0.2.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	b8e73f0755c962013485ceb66fa90e682cbbec92 (v0.79.0-dev20260914-301-gb8e73f0755) = main 58648d44443 + tenstorrent/tt-metal#57142 (head 0d037bf72f18) + tenstorrent/tt-metal#53988 (head 9c315f43e3a1, merged --no-ff); the merge commit lives on a local branch (changh95/rldx1-eth-release) — the main commit and the two PR heads are on GitHub; built from source inside the image
dispatch	fast-dispatch kernels on the idle Ethernet cores (DispatchCoreType.ETH), 12×10 worker grid — GET /info -> device reports compute_grid / dispatch_core; RLDX1_DISPATCH_CORE=tensix restores the previous 11×10 Tensix-dispatch configuration
port snapshot	code/models/experimental/rldx1 = the frozen snapshot of the port (tt-metal ref rldx1-p150-snapshot-r2, committed on top of the development checkout 668c2907575 the package is versioned in — not the runtime above); package sources sha256 faf88bdcf276 (the commit.rldx1_files_sha every cited JSON carries)
code/ digest	5671d6c62dc604f7 (sha256, first 16 hex digits)
image	tt-model/rldx-1-pt-p150:43757607585a (sha256:437576075)
built	2026-09-21T14:24:48+00:00 by tt-model 0.1.0
weights	RLWRLD/RLDX-1-PT @ 1592013ab955b4919facd91fb37449f33a292a70 (tt-model.yaml weights.revision = serve.env.TT_WEIGHTS_REVISION); VLM assets RLWRLD/RLDX-1-VLM @ 4b9f870d1287e0d38d7eb1445e6d8c60afe66dd7
reference	RLWRLD/RLDX-1 v1.0.2 (cf67c31f), fp32 goldens and RTX 5090 rows from that code on this host