vggt-1b-p150

VGGT-1B (Meta's feed-forward multi-view 3D reconstruction transformer) running on one Tenstorrent Blackhole p150a via tt-nn: 1-4 views of a scene in, per-pixel depth, world points, confidences and camera poses out. Weights: facebook/VGGT-1B · Paper: arXiv:2503.11651 · Upstream code: facebookresearch/vggt · Port: changh95/tt-vggt

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/vggt-1b-p150 --with-weights
tt-model serve changh95/vggt-1b-p150

• Weights facebook/VGGT-1B at 860abec7937d go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/vggt-1b-p150
printf '{"images":["%s"]}' "$(base64 -w0 media/input.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/vggt-1b-p150

• POST /predict: images (1-4 base64 PNG/JPEG views of one scene, ordered; image is a single-view alias); optional output_format (npz | json), dtype (float16 | float32), outputs (subset of depth, depth_conf, world_points, world_points_conf), conf_threshold, json_stride (8).
• GET /health, GET /info.

Response

{"model": "VGGT-1B", "num_views": 1, "image_size": [518, 518],
 "preprocess": {"mode": "pad", "per_view": ["..."]},
 "pose_enc": [[0.0, -0.0, -0.0, 0.0, -0.0, 0.0, 0.998, 0.866, 0.866]],
 "extrinsic": ["..."], "intrinsic": ["..."],
 "dense": {"encoding": "npz", "dtype": "float16",
           "arrays": {"depth": {"shape": [1, 518, 518], "dtype": "float16"},
                      "depth_conf": {"shape": [1, 518, 518], "dtype": "float16"},
                      "world_points": {"shape": [1, 518, 518, 3], "dtype": "float32"},
                      "world_points_conf": {"shape": [1, 518, 518], "dtype": "float16"}},
           "data": "..."},
 "timing_ms": {"preprocess": 5.3, "forward": 1716.9, "encode": 103.8, "total": 1826.0}}

• pose_enc is [tx, ty, tz, qx, qy, qz, qw, fov_h, fov_w] per view; extrinsic (S,3,4) is OpenCV cam-from-world with frame 0 as the world frame; intrinsic (S,3,3) and every dense array are in 518×518 padded pixels (x_518 = x_orig*scale_x + pad_left, from preprocess.per_view).
• dense.data is a base64 np.savez_compressed blob: np.load(io.BytesIO(base64.b64decode(data))) yields depth (S,518,518), depth_conf, world_points (S,518,518,3) and world_points_conf. output_format: "json" returns nested lists subsampled by json_stride instead.

Demo

Input (media/input.png)	Depth and re-rendered point cloud on p150a (media/depth.png, media/point_cloud_reprojected.png)

Accuracy and speed

Metric	Value
min-PCC vs fp32 torch reference (real CO3Dv2 apple input)	0.9947 (synthetic input 0.9959)
AUC@30° camera-pose accuracy (CO3Dv2 apple, S=2, 3 scenes)	86.1 vs 87.2 reference
Inference, served over HTTP (warm, S=1, 518×518, npz float16)	~1.7-1.8 s forward · ~1.8-1.9 s end-to-end
Inference, served over HTTP (S=2, single request)	3.2 s forward · 3.4 s end-to-end

Caveats

• Every view is resized and white-padded to 518×518 server-side; 1-4 ordered views of one scene per request (VGGT_S_MAX=4), batch 1, requests serialised on the one chip.
• bf16 on device with fp32 residual/softmax accumulation: outputs differ slightly from the fp32 reference (min-PCC 0.9947, about 1.1 AUC@30° points below it on real data).
• Weights are CC-BY-NC-4.0 (non-commercial only); the 5.0 GB fp32 model.safetensors is fetched into your HF cache, and the server keeps the torch model (~5 GB host RAM) as the driver.
• Dense outputs are npz-encoded by default (see Response); point tracking is not exposed (track head not ported).
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404. Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730), single p150a only.

Licensing

• Weights: facebook/VGGT-1B, CC-BY-NC-4.0 (non-commercial only; not redistributed here).
• Port and serving code (code/models/): Apache-2.0 per changh95/tt-vggt, bound by the weights' non-commercial terms; vendored upstream code/vggt/ is Meta's under the VGGT License v1 (code/LICENSE.txt, with its Acceptable Use Policy).

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	018174b1de095d2a (sha256, first 16 hex digits)
built	2026-09-12T13:52:59+00:00 by tt-model 0.1.0