vggt-1b-blackhole

VGGT-1B (facebook/VGGT-1B) feed-forward multi-view 3D reconstruction on a single Tenstorrent Blackhole p150a via tt-nn: 1-4 RGB views of one scene in, per-pixel depth and world points (each with confidence) plus camera pose out. The transformer blocks, 2D RoPE, DPT refinenets and the final DPT convs run on the chip in bf16 with fp32 residual/softmax accumulation; the port author measured about 1.3-1.7 s per frame at S=1 (518x518) against 5.0 s for the torch CPU reference, with min-PCC 0.9947 on real CO3Dv2 input and CO3Dv2 AUC@30 86.1 vs 87.2 for the reference. Weights are CC-BY-NC-4.0 (non-commercial only); the vendored upstream code under code/vggt is covered by the VGGT License v1 shipped as code/LICENSE.txt. Port source: github.com/changh95/tt-vggt @ 0884963 (Hyunggi Chang); upstream github.com/facebookresearch/vggt @ 44b3afbd.

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/vggt-1b-blackhole --with-weights
tt-model serve changh95/vggt-1b-blackhole

pull --with-weights downloads the Docker image and the facebook/VGGT-1B weights at 860abec7937da0a4c03c41d3c269c366e82abdf9 (into your HF cache; they are not in the image). serve starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

tt-cli

tt serve changh95/vggt-1b-blackhole        # same as tt-model serve; prints the port
tt model stop changh95/vggt-1b-blackhole   # or: tt-model stop changh95/vggt-1b-blackhole

The port is the one serve printed (20000, or the next free one). This is the model's own HTTP API, not an OpenAI one: curl <endpoint>/v1/models only returns a stub and tt-model curl does not apply.

Call it

URL=http://127.0.0.1:20000
curl -s $URL/health      # {"status":"ok",...} once warm-up is done (503 from /predict before that)
curl -s $URL/info        # weights repo + revision, limits, licences, latency hints

# one scene, 1..4 ordered views (frame 0 is the world frame); media/input.png is the README frame
python3 - <<'EOF'
import base64, io, json, urllib.request, numpy as np
b64 = base64.b64encode(open("media/input.png", "rb").read()).decode()
req = {"images": [b64], "output_format": "npz", "dtype": "float16"}
r = urllib.request.Request("http://127.0.0.1:20000/predict", json.dumps(req).encode(),
                           {"Content-Type": "application/json"})
out = json.load(urllib.request.urlopen(r, timeout=600))
print(out["num_views"], out["pose_enc"][0], out["timing_ms"])
z = np.load(io.BytesIO(base64.b64decode(out["dense"]["data"])))
print({k: (z[k].shape, str(z[k].dtype)) for k in z.files})
EOF

Request fields: images (list of 1..4 base64 PNG/JPEG; image is a single-view alias), output_format (npz default | json subsampled by json_stride), dtype (float16 default | float32 for depth/conf), outputs (subset of depth, depth_conf, world_points, world_points_conf), conf_threshold (optional; low-confidence pixels become NaN).

Response: num_views, image_size [518, 518], preprocess.per_view (orig_w/h, scale_x/y, pad_left/top -- x_518 = x_orig*scale_x + pad_left), pose_enc (S, 9) = [tx, ty, tz, qx, qy, qz, qw, fov_h, fov_w], extrinsic (S, 3, 4) OpenCV cam-from-world, intrinsic (S, 3, 3) in 518-px units, dense.data = base64 np.savez_compressed with depth (S,518,518), depth_conf (S,518,518), world_points (S,518,518,3) float32, world_points_conf (S,518,518), and timing_ms (preprocess, forward, encode, total). Every frame is resized (longer side 518, shorter side to a multiple of 14) and white-padded to 518x518 server-side -- the geometry the port was validated with. Tracking (query_points) is not exposed.

A ready-made check: python3 code/models/server/smoke_test.py --url $URL prints one PASS/FAIL line with depth range, confidences and latency.

First boot

• Weights: model.safetensors (5.0 GB, fp32) is downloaded into your HF cache by tt-model serve / pull --with-weights; the server keeps the fp32 torch model (~5 GB host RAM) as the driver and uploads ~2.5 GB of bf16 weights to the chip.
• Warm-up runs one dummy forward per pre-warmed view count (VGGT_PREWARM_SEQS = 1,2,3,4). A cold JIT cache (~/.cache/tt-model/vggt-1b-blackhole/cache) compiles every kernel on the first boot -- expect several minutes before Application startup complete; later boots reuse the cache and take about a minute plus the weight load.
• Latency (port author's numbers, one p150a): S=1 ~1.3-1.7 s, S=2 ~2.4 s, S=3 ~4.4 s, S=4 ~6.2 s per request. Requests are serialised (one chip).
• Stop with tt-model stop changh95/vggt-1b-blackhole; the server closes the chip on SIGTERM. A hard kill can leave the chip wedged (tt-smi -r 0 recovers).

Results (from the port's README)

	torch CPU reference	ttnn port on p150a	ratio
latency / frame (B=1 S=1 518x518)	5037 ms	~1694 ms (later commits ~1300-1400 ms)	~3x
min-PCC (port vs ref, synthetic input)	--	0.9959	--
min-PCC (port vs ref, real CO3Dv2 apple)	--	0.9947	--
AUC@30 (CO3Dv2 apple S=2, 3 scenes)	87.2	86.1	-1.1

After the S>=3 softmax fix (BF0 in code/TODO.md): S=1 PCC 0.9997, S=2 0.9967, S=3 0.9989, S=4 0.9981 (all PASS at the 0.99 floor).

input	predicted depth	point cloud, re-rendered from a new angle

Precision profile: bf16 weights and matmul inputs; fp32 residual accumulator in every Block; fp32 attention scores + softmax (decomposed max/sub/exp/sum at N>=4000 to dodge a Blackhole fused-softmax hang); HiFi4 + fp32 dest accumulation on proj / fc2 / DPT convs. Full optimisation trajectory, CO3Dv2 write-up and backlog: code/results.tsv, code/co3d_eval_results.md, code/TODO.md.

Layout

code/models/demos/vggt/tt/ttnn_vggt.py (the port: installs bf16 weights on the chip and monkey-patches the upstream classes), code/models/demos/vggt/reference/torch_vggt.py (weights loader), code/models/server/app.py (this server), code/vggt/ (upstream facebookresearch/vggt @ 44b3afbd), code/test_vggt.py / code/eval_vggt.py / code/make_demo.py (host harnesses; set TT_METAL_HOME, VGGT_REF, TT_DEVICE_ID).

Licensing

Non-commercial only. The weights facebook/VGGT-1B are CC-BY-NC-4.0 and are not redistributed here (they are fetched from the upstream repo into your HF cache). The upstream model code vendored under code/vggt/ is Meta's, under the VGGT License v1 (code/LICENSE.txt, with its Acceptable Use Policy); redistribution must carry that licence. The port code (code/models/...) was written by Hyunggi Chang and follows the weights' non-commercial terms. tt-metal / tt-nn are Apache-2.0.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	018174b1de095d2a (sha256, first 16 hex digits)
built	2026-09-12T05:28:49+00:00 by tt-model 0.1.0