RLDX-1-PT-IMG-p150

RLWRLD RLDX-1 PT-IMG (vision-language-action policy: Qwen3-VL-8B backbone truncated to 18 layers with 64 cognition tokens, FLUX-style MM-DiT "MSAT" action head, 4 Euler steps) running on one Tenstorrent Blackhole p150a via tt-nn, in the DROID layout: three cameras × one frame at 320×180, a 7-dim end-effector / gripper state and a task instruction in, a 16-step chunk of DROID end-effector deltas and gripper targets out. The ViT and the LLM are TTNN ops in Metal traces; the MSAT denoise runs as one persistent megakernel streaming bfp8 weights from a DRAM arena. Weights: RLWRLD/RLDX-1-PT-IMG (RLWRLD Model License v1.0, non-commercial) · Paper: arXiv:2605.03269 · Upstream code: RLWRLD/RLDX-1 (v1.0.2) · Port: code/models/experimental/rldx1 and the server code/rldx1_p150 in this repo

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/RLDX-1-PT-IMG-p150 --with-weights
tt-model serve changh95/RLDX-1-PT-IMG-p150

• Weights RLWRLD/RLDX-1-PT-IMG at d67fc642a7e7 (three bf16 safetensors shards + the processor json files) go to your Hugging Face cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete (the first boot converts the weights and JIT-compiles the kernels, see Caveats).

Run with tt-cli

tt serve changh95/RLDX-1-PT-IMG-p150
curl -s localhost:20000/demo | python3 -c 'import json,sys; json.dump(json.load(sys.stdin)["request"], open("req.json","w"))'
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/RLDX-1-PT-IMG-p150

• POST /predict takes the RLDX RLDXPolicy observation: video ({"primary": [<base64 PNG at t>], "secondary": [...], "wrist": [...]}, one frame (the current step) per camera as base64 PNG/JPEG or nested uint8 lists (1, 1, H, W, 3)), state (end_effector_position 3, end_effector_rotation 3, gripper_position 1, raw DROID units), language (≤ 61 BPE tokens after RLDX's formalize_language); optional seed (default 1000) or noise ([16][64]), return_normalized.
• GET /health, GET /info (the full observation contract, the policy actually served, warm-up fidelity), GET /demo (the shipped demo request with its reference noise and the fp32 reference actions), GET /v1/models (stub). RLDX's own ZeroMQ clients (PolicyClient, open_loop_eval.py, rollout_policy.py) connect through the bridge: python -m rldx1_p150.server.zmq_bridge --url http://localhost:20000 --port 5555 (same msgpack codec; ping, get_action, reset, get_modality_config, kill).

Response

{"actions": {"end_effector_position": [[<3 floats>], "... 16 steps"],
             "end_effector_rotation": [[<3 floats>], "... 16 steps"],
             "gripper_close":         [[<1 float>],  "... 16 steps"]},
 "action_horizon": 16, "action_keys": ["end_effector_position", "end_effector_rotation", "gripper_close"],
 "embodiment": "droid", "embodiment_id": 27, "variant": "pt_img", "n_text": 8, "seq_len": 203, "s_pad": 256,
 "noise_source": "client", "seed": null,
 "timing_ms": {"decode": 2.6, "encode": 1.2, "device": 54.4, "decode_actions": 0.1, "total": 58.4}}

• actions are physical, un-normalised DROID targets exactly as RLDXPolicy.get_action returns them: end_effector_position / end_effector_rotation are delta end-effector actions (the DROID RLDS cartesian_velocity space, in [−1, 1]), gripper_close is the absolute gripper target in [0, 1]; 16 steps × [3, 3, 1]; (clip(a, −1, 1) + 1) / 2 · (q99 − q01) + q01 with the checkpoint's processor/statistics.json, no relative → absolute composition.
• The example is the demo request with the reference noise, as GET /demo returns it (noise_source: "client", seed: null); timing_ms are the medians of the warm container bench (publish/logs/container-pt_img-20260920-182501.json, 50 requests). device covers the input writes, the three trace replays (V, L, D) and the readback.

Demo

Input (media/demo_frames.png: the three cameras at t (secondary := primary); DROID demo episode 1, step 40, 320×180)	Instruction	p150a actions vs the fp32 reference (media/demo_actions_vs_golden.png)
	Put the blue block in the green bowl

What runs where

Stage	Where	What	dtype (plan D18; admission JSON reference/out/bfp_emulation_pt_img.json)
Host encode	CPU, outside the device lock	AspectAreaResizeAndCrop (320×180 → 160×284 → centre crop 160×256), Qwen2-VL patching (16 px patches, 2×2 merge → 40 tokens per image), tokenizer, static slot layout, mRoPE tables, additive mask, state q01/q99 normalisation, the ≤ 61 instruction embedding rows gathered from the safetensors shards	fp32 → bf16 uploads
Trace V	device	Qwen3-VL vision tower as the port's tuned fork: 27 blocks on 3 × 160 patches, 3 deepstack mergers + final merger → 40 × 4096 tokens per image	ViT bf16 HiFi2 fp32-accumulate (admitted: bfp8 buys nothing on the compute-bound ViT)
Trace L	device	token splice (constant skeleton + text rows + ViT rows), 18 Qwen3 layers with deepstack adds after layers 0/1/2, final RMSNorm, the 64 cognition tokens, the DROID state encoder	LLM matmul weights bfp8_b (D19 admission: 0.99734 min per-dim-gated action_pred_valid PCC vs fp32 on 9 emulated samples, admitted = yes); activations bf16; norm statistics fp32
Trace D	device	the MSAT flow-matching head — 4 double blocks + 8 single blocks + output head, all 4 Euler steps — as one persistent ttnn.generic_op megakernel streaming the block weights from a DRAM arena; 5 wrapper ops (noise copy, resident buffers, descriptor rebind, action readback)	arena bfp8_b (admission 0.99080, admitted = yes); tails (proj_out_*, encoders / decoder slot 27, biases, norm vectors, cog_emb, tables) bf16 HiFi4, never bfp8; x_t fp32 with bf16 rounding per step (matches the reference's bf16 Euler update)
Host decode	CPU	action_pred[:16, :7] → q01/q99 de-normalisation → the three groups	fp32

Release policy (GET /info -> policy): the release LLM+MSAT bfp8 combination was emulated on the GPU first (0.99329 min action_pred_valid per-dim PCC, 0.99961 min pred_velocity PCC, worst margin 0.0009 above the gate on ptimg_droid_real/5:ep2_s120; admitted = yes) and then confirmed on the device on every golden sample (table below). RLDX1_MSAT_BACKEND=ttnn serves the bring-up MSAT (the same head as a sequence of TTNN ops; the device oracle of the megakernel).

Layout lock

One embodiment, one static layout, batch 1 — everything else is refused with 400 before the device is touched:

item	served value
embodiment	DROID (EmbodimentTag.OXE_DROID, projector slot 27); statistics.json["droid"] q01/q99 for state and actions
cameras / video_length	primary, secondary, wrist — one frame (the current step) each (video delta_indices [0], oldest first)
frame size	any frame whose AspectAreaResizeAndCrop (area budget 65,536 px, multiples of 32) lands on 160×256 — the native DROID 320×180 frames do; one 10×16 patch grid = 40 merged tokens per image
instruction	≤ 61 BPE tokens after formalize_language (61 right-padded text slots; longer → 400); the empty instruction is accepted
LLM sequence	S_pad 256: prefix 3 + 61 text + 3 × 42 image slots + <|im_end|>, \n, 64 cognition tokens = 256 exactly (no tail pads)
state / actions	state 7 (end_effector_position 3, end_effector_rotation 3, gripper_position 1) → padded 64; actions 7 of 64, horizon 16
noise	[1, 16, 64], torch.randn on a CPU generator seeded with seed (default 1000, which reproduces the goldens' noise_x0), rounded to bf16; or sent explicitly

Accuracy and speed

Metric	Value
Action chunk vs the fp32 reference (host serve, demo observation, same initial noise; publish/logs/smoke-host-pt_img.json)	PCC(actions) 0.99990 · max|d| 0.0225 · end_effector_position 0.99347 · end_effector_rotation 0.99951 · gripper_close max|d| 0.0026 (near-constant group) · normalised valid slice 0.99985
Every golden sample, every tap (WP-A test_tt_multisample, tt-metal/models/experimental/rldx1/tests/tt/results/test_multisample_pt_img_20260920-173018.json; acceptance document tt-metal/models/experimental/rldx1/tests/tt/results/acceptance_pt_img.md)	1145 rows over 10 DROID samples (real + budget-edge synthetic), 17 below gate; tightest full-tensor margin -0.001396 on pred_velocity_0; the per-class minima are in the table below
Official bf16 GPU path vs the same fp32 reference (the floor any bf16/bfp8 port is measured against; tt-metal/models/experimental/rldx1/tests/tt/gates/gates_bf16_pt_img.json)	min action_pred_valid PCC 0.99984 over the golden samples
Traced vs untraced, determinism, stale state	traced replays bit-identical (tt-metal/models/experimental/rldx1/tests/tt/results/test_determinism_pt_img_20260920-171910.json); A→B→A→B alternation reproduces A and B exactly (tt-metal/models/experimental/rldx1/tests/tt/results/test_stale_state_abab_pair0_pt_img_20260920-171939.json)
Inference on p150a (port benchmark bench_e2e, warm, batch 1, median of 50 traced replays after 10 warm-ups; tt-metal/models/experimental/rldx1/benchmarks/results/bench_e2e_pt_img_20260920-172259.json)	56.3 ms end-to-end (p90 56.6) = host encode 1.9 + upload 0.7 + traces 53.5 (V 13.5 · L 24.7 · D 15.5 = 4 × 3.87) + readback and decode 0.21; device (upload → readback) 54.3 (p90 54.4). Plan D27 release gate 60 ms (target 55): meets
Served over HTTP by this image (container, warm boot, batch 1, 50 requests after 5 warm-ups; publish/logs/container-pt_img-20260920-182501.json)	54.4 ms device (p90 54.5) · 58.4 ms server-side incl. base64 + 3 PNG decodes (p90 58.9) · 60.5 ms client wall on localhost
Same policy on an RTX 5090 (same host, RLWRLD's unmodified RLDXPolicy, eager PyTorch bf16 + sdpa, stock flags, same observation; reference/out/gpu/gpu_baseline_pt_img.json)	RLDXPolicy.get_action 62.6 ms e2e (p90 64.4), model 56.9, backbone 30.7, action head 25.1 → p150a 1.11× end-to-end, 1.05× model-only, 1.62× on the action head. RLWRLD's published RTX 5090 numbers (2 cameras, their stack): eager 191.2 ms, CUDA-graph 33.7, custom Triton chain 25.2 — different environment and input, see GPU_COMPARISON.md
Boot (container; publish/logs/container-pt_img-20260920-182501.json) and clock	cold (empty weight and kernel caches) 121 s, warm 23 s from tt-model serve to READY; AICLK 800 MHz before / 1350 after the benchmark, 1350 after 400 back-to-back replays of trace V (sustained-clock proxy)

Fidelity per tap class, TT vs the fp32 reference over every golden sample (min full-tensor PCC; gates frozen from the official bf16 path's own spread per plan D23, gate = min(0.99999, 1 − 3·(1 − min PCC_bf16))):

tap class (golden_tap_map.json)	min PCC TT vs fp32 (tap, sample)	its gate	official bf16 vs fp32, min over samples
vit — 27 ViT blocks, deepstack mergers, final merger	0.99874 (vit_merger (canonical_ep1_s40))	0.99833	0.99911
llm — 18 LLM layers, final norm, cog_features	0.99921 (llm_layer_0 (ep1_s120))	0.99877	0.99960
feature — state / action encoders, temb, x_t, decoder rows	0.99994 (actions_out_3 (ep2_s120))	0.99989	0.99984
msat — double / single block outputs per Euler step	1.00000 (proj_out_1_3 (canonical_ep1_s40))	0.99999	0.99977
action — pred_velocity_k	0.99836 (pred_velocity_0 (ep1_s120))	0.99976	0.99985
output — action_pred_normalized, action_pred_valid, unnormalized_actions.end_effector_*	0.97975 (unnormalized_actions.end_effector_rotation (at_budget))	0.99900	0.99487
output_abs — unnormalized_actions.gripper_close (near-constant: abs-only)	max|d| 0.0082	tol 0.0096	max|d| 0.0048

Per-dimension action_pred_valid PCC over the horizon, min over samples (pos x, y, z · rot x, y, z · gripper; gated at 0.9817 where the fp32 range on that sample is ≥ 0.25, the gripper dimension abs-only): 0.9432 · 0.9771 · 0.9744 · 0.9674 · 0.9818 · 0.9767 · 0.9146. Physical max|d| over samples: end_effector_position 0.0768 (tol 0.0460), end_effector_rotation 0.0918 (tol 0.0430).

What differs from the reference

The device model is a static re-implementation of RLWRLD's RLDXPolicy (RLDX-1 v1.0.2) for one layout; these are the deliberate differences (plan decision ids in brackets; each is gated against the fp32 reference above):

• [D3] Static slot table instead of dynamic tokenisation. Every call uses the same token slots: <|im_start|> user \n, 61 right-padded text slots (real tokens first, pad id 151643 after, masked as keys), 3 × [<|vision_start|> + 40 <|image_pad|> + <|vision_end|>], <|im_end|>, \n, 64 cognition slots, S_pad 256. Real tokens keep their reference mRoPE positions; pad query rows take the last real text position (finite, never read). Anything the table cannot hold is a 400, never a re-trace.
• [D5] No 1.26 GB embedding table on the device. A constant device skeleton (special-token rows and the cog_emb rows at their slots, zeros elsewhere) is completed per call by ≤ 61 instruction rows (text_rows, gathered on the host from embed_tokens through safetensors slices with an LRU per token id) and by the ViT rows placed into the image slots; deepstack features are added after layers 0/1/2 the same way. The 35 unused embodiment slots and lm_head are never converted.
• [D7] Visual-token compression stays out of this package (PT-IMG never compresses). The 4-frame PT model pools the 9 older images' 378 slots into one token at the input of layer 4; the sibling PT package does it as one Sel @ h selection matmul under HiFi4 with fp32 accumulation and fp32 output, the pooled row scaled by fp32 1/378 before the bf16 cast so the kept rows stay bit-exact. This 1-frame model has no compression step; its 18 layers run at S_pad 256 throughout.
• [D10] Tuned fork of the ViT instead of the verbatim tower. The Qwen3-VL tower runs as a generalised fork of the GR00T port's tuned tower: images as the batch dim, one batched SDPA over the 3 × 160 patches with an explicit 72**-0.5 scale, fused GELU_TANH in fc1, RoPE on the 96-padded heads with an identity tail (the golden tower is 16 × 72 wide), mergers around a ROW_MAJOR reshape. The verbatim tt-nn Qwen3VlVisionModel is kept as the oracle only (erf GELU in its blocks; RLDX1_VIT_PATH=verbatim), never traced for serving.
• [D18] Per-class dtype policy. ViT bf16 HiFi2 with fp32 accumulation; LLM matmul weights bfp8_b and the MSAT arena bfp8_b because their D19 admission passed (emulated on the GPU, confirmed on the device — both JSONs cited above; bf16 is the automatic fallback of a rejected class); every tail (proj_out_1/2, encoders / decoder slot 27, biases, norm vectors, cog_emb, RoPE tables, the selection matrix) bf16 HiFi4, never bfp8; activations bf16; norm statistics fp32; every matmul with fp32_dest_acc_en; the additive attention mask uses -1e9, never −inf; x_t is kept in fp32 with the reference's bf16 rounding per Euler step.

Caveats

• One model per process, one layout, batch 1. The traces are shape-locked to the DROID layout above (3 images × 160 patches, S_pad 256, embodiment slot 27, video_length 1); the 35 other embodiment slots of the checkpoint have no traces here. The server builds exactly one model per process (a second megakernel model on the same open device is not supported by design).
• No memory, no real-time chunking. RLDX's options (memory / RTC) are accepted and ignored; reset returns {"cleared_sessions": []}. Actions are a deterministic function of (observation, seed | noise): two identical requests return bit-identical chunks.
• secondary := primary in the demo. Every local DROID sample has only exterior_1_left and wrist_left; the shipped demo (and every golden, benchmark and GPU row) feeds the primary frame as secondary too. A real three-camera DROID rig will differ from the demo numbers; the layout itself is unchanged.
• Host encode p90 above p50 on unseen vocabulary. The instruction's embedding rows are gathered from the safetensors shards the first time a token id is seen (then cached): p50 1.9 ms vs p90 2.1 ms in the port benchmark (host_encode, tt-metal/models/experimental/rldx1/benchmarks/results/bench_e2e_pt_img_20260920-172259.json; 0 cold ids during the timed calls). A new instruction costs a few extra milliseconds on its first call only.
• First boot is cold: the checkpoint is converted to the per-class device layout into /weight-cache and every kernel including the megakernel is JIT-compiled into /cache — 121 s measured for this image, 23 s with warm caches (both under ~/.cache/tt-model/rldx-1-pt-img-p150/).
• Megakernel path. Trace D replays one generic_op for all four Euler steps; the device is opened with a 96 KiB worker-L1 cut (worker_l1_size 1,362,944 B) for the kernel-config ring the megakernel binaries need, and the MSAT block internals are not observable from the served model (the ttnn-op MSAT, RLDX1_MSAT_BACKEND=ttnn, is the A/B path and exposes them).
• Not an OpenAI-compatible API; GET /v1/models is a stub. Validated on tt-metal 668c2907575, single p150a only. Details: SERVING.md, GPU_COMPARISON.md.

Licensing

• Weights: RLWRLD/RLDX-1-PT-IMG @ d67fc642a7e7, Licensed under the RLWRLD Model License v1.0 — non-commercial use only (research, education, personal or evaluation), share-alike (derivative models only under this licence), attribution to RLWRLD, complete licence copy required with any redistribution; the copy is LICENSE in this repo. The weights are not redistributed here: tt-model serve fetches exactly the files in tt-model.yaml into your Hugging Face cache and converts them on your machine. Changes made: tt-nn re-implementation of the network for one Blackhole p150a; bf16 → bfp8 device layout for the admitted weight classes (LLM matmuls, MSAT arena); static DROID layout (fixed slot table, 61-token instruction budget, one embodiment); MSAT denoise as one megakernel.
• Vendored RLDX-1-VLM configuration / tokenizer files (code/rldx1_p150/assets/vlm/, RLWRLD Model License v1.0; derived from Qwen/Qwen3-VL-8B-Instruct, Apache-2.0) and the demo's reference actions (code/rldx1_p150/demo/pt_img/expected.json, outputs of the model) are covered by the same licence; hashes in the PROVENANCE.md next to them.
• Port and serving code (code/models/experimental/rldx1, code/rldx1_p150, code/scripts): Apache-2.0, © 2026 Tenstorrent; copied GR00T-port modules keep their Apache-2.0 headers and are listed with sha256 in code/models/experimental/rldx1/PROVENANCE.md. Because the code only runs the RLWRLD-licensed weights, the package as a whole is offered for non-commercial use. Details and the third-party list: LICENSE-NOTICE.md.
• Citation: RLDX-1 Technical Report, arXiv:2605.03269; upstream code github.com/RLWRLD/RLDX-1 v1.0.2.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	668c290757550588d0ce46b180c344a462a2aaf5 (v0.79.0-dev20260914-1-g668c290757), built from source inside the image
port snapshot	code/models/experimental/rldx1 = the frozen snapshot of the port (tt-metal ref rldx1-p150-snapshot, parent 668c2907575); package sources sha256 80c4c80d81f4 (the commit.rldx1_files_sha every cited JSON carries)
code/ digest	431b1f20ead84db1 (sha256, first 16 hex digits)
image	tt-model/rldx-1-pt-img-p150:ec7f8a8c6369 (sha256:ec7f8a8c6)
built	2026-09-20T09:28:43+00:00 by tt-model 0.1.0
weights	RLWRLD/RLDX-1-PT-IMG @ d67fc642a7e768049ba41ff8cf0dfe0ff17f556b (tt-model.yaml weights.revision = serve.env.TT_WEIGHTS_REVISION); VLM assets RLWRLD/RLDX-1-VLM @ 4b9f870d1287e0d38d7eb1445e6d8c60afe66dd7
reference	RLWRLD/RLDX-1 v1.0.2 (cf67c31f), fp32 goldens and RTX 5090 rows from that code on this host