GR00T-N1.7-3B-p150

NVIDIA Isaac GR00T N1.7 (3B vision-language-action policy) running on one Tenstorrent Blackhole p150a via tt-nn, in the DROID layout: two cameras × two frames (t−15 and t) at 16:9, a 17-dim Franka state and a task instruction in, a 40-step chunk of end-effector, gripper and joint actions out. Stage-2 release: the whole DiT denoise runs as one persistent megakernel streaming bfp8 weights from a DRAM arena; vision (tuned Qwen3-VL tower), LLM and adapter are TTNN ops in Metal traces; GR00T_DIT_BACKEND=ttnn selects the Stage-1 denoise. Weights: nvidia/GR00T-N1.7-3B · Paper: arXiv:2503.14734 · Upstream code: NVIDIA/Isaac-GR00T (main, the n1.7-release line) · Port: code/models/experimental/gr00t and the server code/gr00t_p150 in this repo

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/GR00T-N1.7-3B-p150 --with-weights
tt-model serve changh95/GR00T-N1.7-3B-p150

• Weights nvidia/GR00T-N1.7-3B at 2fc962b973bc (6.91 GB) go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/GR00T-N1.7-3B-p150
python3 - <<'EOF'
import base64, json
b64 = lambda f: base64.b64encode(open(f"media/{f}.png", "rb").read()).decode()
obs = json.load(open("media/demo_observation.json"))   # DROID demo step: raw state + instruction
obs["images"] = {cam: [b64(f"{cam}_tm15"), b64(f"{cam}_t0")]          # per camera: the frame at t-15, then the frame at t
                 for cam in ("exterior_image_1_left", "wrist_image_left")}
json.dump(obs, open("req.json", "w"))
EOF
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/GR00T-N1.7-3B-p150

• POST /predict: images ({"exterior_image_1_left": [<base64 PNG/JPEG at t−15>, <at t>], "wrist_image_left": [...]}, exactly two frames per camera, 16:9 frames), state ({group: [floats]}: eef_9d 9 = xyz + rot6d, gripper_position 1, joint_position 7, raw values), instruction (≤ 51 tokens); optional seed (default 42) or noise ([40, 132]), return_normalized.
• GET /health, GET /info (stage says what is served), GET /demo (the shipped demo request with its reference noise, and its fp32 reference actions).

Response

{"actions": {"eef_9d":           [[0.5284, 0.0181, 0.4569, -0.0786, 0.3927, -0.9163, 0.9923, 0.1193, -0.034], "... 40 steps"],
             "gripper_position": [[0.0], [0.002], "..."],
             "joint_position":   [[-0.0051, 0.034, 0.0013, -1.917, 0.1641, 1.6101, 0.0844], "..."]},
 "action_horizon": 40, "embodiment": "oxe_droid_relative_eef_relative_joint", "seq_len": 469, "noise_source": "client", "seed": null,
 "timing_ms": {"decode": 3.4, "encode": 10.9, "device": 55.9, "decode_actions": 1.1, "total": 71.3}}

• The example is the demo request with the reference noise, as GET /demo returns it (noise_source: "client", seed: null); timing_ms from a warm request of the same serve.
• actions are physical, un-normalised targets, 40 steps × [9, 1, 7] dims, as Gr00tPolicy.get_action returns them for the DROID data config: eef_9d is the absolute end-effector pose (the relative SE(3) output composed onto the request's pose), joint_position the relative output added to the request's joints, gripper_position absolute; timing_ms.device covers upload, the four trace replays and the readback.

Demo

Input (media/demo_frames.png: both cameras at t−15 and t; DROID sample episode 1, step 40, 320×180)	Instruction	p150a actions vs the fp32 reference (media/demo_actions_vs_golden.png)
	Put the blue block in the green bowl

Accuracy and speed

Metric	Value
Action chunk vs the fp32 reference (same observation, same initial noise)	PCC eef_9d 0.99998 · joint_position 0.99996 · gripper_position max|d| 0.0031 (near-constant group) · all 17 dims × 40 steps 0.99996 · normalised valid slice 0.99995
Intermediate taps vs the fp32 reference (27 gated taps, from pixels; DiT internals are not observable inside the megakernel)	30 of 30 rows pass; tightest vit_deepstack_mergers[0] 0.999849 vs gate 0.9998. On six more DROID goldens the actions pass on five: t2_s120 has two pre-existing misses (llm_layer_first 0.98998 vs 0.991, joint_position PCC 0.99976 / max|d| 0.107 vs 0.9998 / 0.083); t1_s120 adds one megakernel-only marginal tap, pred_velocity[k=2] dim 9 per-dim PCC 0.99588 vs 0.996 (its actions pass)
Megakernel vs Stage-1 TTNN denoise (same device, 7 DROID samples)	action_pred_normalized PCC ≥ 0.999992 between the two backends; per-step velocities ≥ 0.99996
Traced vs untraced, determinism, stale state, 2 vs 1 command queue	bit-identical; 5 trace replays bit-identical; A→B→A→B reproduces A and B exactly; 2-CQ == 1-CQ on every probe (26 rows)
Inference on p150a (port benchmark, warm, batch 1, median of 50)	62.8 ms end-to-end (p90 68.6) ≈ host encode 5.6 + upload 3.2 + traces 52.4 (vision 17.5 · LLM 12.4 · adapter 6.2 · denoise 16.5 = 4 × 4.12) + readback and decode 1.3. Stage-1 path on the same tree: 79.7 ms (denoise 30.7)
Served over HTTP by this image (warm, batch 1, 50 requests)	55.9 ms device (p90 56.3) · 69.1 ms server-side incl. four PNG decodes (p90 72.7)
Same policy on an RTX 5090 (same host, NVIDIA's unmodified N1.7 policy, eager PyTorch bf16, idle machine)	get_action 59.8–65.6 ms model, 73–79 ms incl. host pre/post-processing → p150a 1.22× faster end-to-end; NVIDIA's published H100 TensorRT full pipeline: 27.9 ms (one camera)

Caveats

• Stage-2 path: one persistent generic_op runs the whole denoise inside the fourth Metal trace, streaming the 32 blocks' bfp8 weights once per step from a ~1 GB DRAM arena over the 11×10 Tensix grid (96 compute cores, 8 DRAM readers, hubs), with distributed LayerNorm, per-head flash SDPA and the cross attention over the 448-key image set split three ways. Costs: 64 KiB less allocatable L1 per core (worker_l1_size 1,395,712), the VL adapter's intermediates in DRAM (+0.6 ms), compute binaries of ~131 KB sitting ~4.5 KB under the 136 KB kernel-config ring, and no DiT-intermediate taps. Gain vs the Stage-1 path on the same tree: −19 % end-to-end, −46 % denoise (the plan's −30 % e2e was not reached). Everything outside the DiT matmuls is bf16.
• The vision tower is the port's tuned Qwen3-VL fork (HiFi3 matmuls on blocks 0–5; 17.5 ms vs 87 ms for the verbatim tower, GR00T_VIT_FORK=0 selects it).
• One embodiment and layout: DROID (oxe_droid_relative_eef_relative_joint, embodiment slot 24), exactly two cameras × two frames (t−15 and t), landscape frames that land on the 16×28-patch grid (width / height in [1.686, 1.814]: 16:9, e.g. 320×180), instruction ≤ 51 tokens; anything else is refused with 400. Batch 1.
• Noise: default seed 42 (CPU generator, deterministic). The demo's reference noise (code/gr00t_p150/demo/n17/noise.json) was drawn on CUDA and is not seed-reproducible; send it as noise to reproduce the reference actions.
• Not an OpenAI-compatible API; GET /v1/models is a stub.
• First boot is cold (weight conversion + kernel JIT, 64 s); later boots take 14 s. Validated on tt-metal 668c2907575 (port snapshot 009de892c895), single p150a only. Details: SERVING.md, GPU_COMPARISON.md.

Licensing

• Weights: nvidia/GR00T-N1.7-3B, NVIDIA License (non-commercial; copy in LICENSE). Not redistributed here.
• Tokenizer and image-processor files (code/gr00t_p150/assets/tokenizer/n17/) are byte-identical to Qwen/Qwen3-VL-2B-Instruct (Apache-2.0; hashes in PROVENANCE.md next to them).
• Port and serving code (code/): Apache-2.0, © 2026 Tenstorrent, distributed under the same non-commercial limitation as a derivative of the weights. See LICENSE-NOTICE.md.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	668c290757550588d0ce46b180c344a462a2aaf5
code/ digest	9efa5512d0634561 (sha256, first 16 hex digits)
built	2026-09-17T20:42:14+00:00 by tt-model 0.1.0