mast3r-p150

DUSt3R two-view 3D reconstruction (naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt: ViT-L/16 encoder, dual-branch decoder, two DPT heads; the backbone of MASt3R) running entirely on one Tenstorrent Blackhole p150a via tt-nn: an image pair in, two dense 512×512 pointmaps with confidence (plus optional PairViewer pose) out. Weights: naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt · Paper: arXiv:2312.14132 · Upstream code: naver/mast3r · Port: changh95/tt-mast3r

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/mast3r-p150 --with-weights
tt-model serve changh95/mast3r-p150

• Weights naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt at 61c57447d7b0 go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/mast3r-p150
printf '{"image1":"%s","image2":"%s"}' "$(base64 -w0 media/source_1.png)" "$(base64 -w0 media/source_2.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/mast3r-p150

• POST /predict: image1, image2 (base64 PNG/JPEG, view 1 = reference camera; or images: [b64, b64]); optional output_format (npz | png, default npz), return_pose (false), intrinsics (3×3 K of view 2 in its original pixels), conf_pct (50).
• GET /health, GET /info.

Response

{"model": "mast3r-p150", "frame": "camera_1", "input_size": 512, "depth_mode": "exp", "conf_mode": "1+exp",
 "preprocess": [{"orig_w": 512, "orig_h": 512, "offx": 0, "offy": 0, "pad_side": 512, "out_size": 512, "scale": 1.0}, {...}],
 "output_format": "npz", "npz_b64": "...",
 "npz_keys": {"pts3d1": [[512, 512, 3], "float32"], "conf1": [[512, 512], "float16"],
              "pts3d2": [[512, 512, 3], "float32"], "conf2": [[512, 512], "float16"]},
 "summary": {"view1": {"finite_fraction": 1.0, "conf_min": 1.0, "conf_mean": 3.70, "conf_max": 26.39,
                       "z_min": 0.287, "z_median": 0.407, "z_max": 1.102}, "view2": {...}},
 "pose": null, "pose_requested": false,
 "timing_ms": {"decode": 5.5, "preprocess": 1.9, "forward": 73.6, "encode": 160.3, "total": 241.4}}

• npz_b64 is a base64 .npz holding pts3d1/pts3d2 float32 (512,512,3) and conf1/conf2 float16 (512,512); both pointmaps are in the camera-1 frame, depth is pts3d[..., 2] in DUSt3R's own scale (not metres), conf = 1 + exp(c) >= 1. output_format: "png" returns per-view 16-bit depth + 8-bit confidence PNGs instead.
• preprocess maps original pixels to the 512 grid, (u, v) = ((x + offx) * scale, (y + offy) * scale). With return_pose: true, pose holds R, t (view 2 in the view-1 frame, X_cam2 = R @ X_cam1 + t), focal_1, focal_2 (512-px units) and used_known_intrinsics; null when PnP fails.

Demo

Input pair, CO3Dv2 apple frames 0 / 40 (media/source_1.png, media/source_2.png)	Predicted point cloud on p150a, two viewpoints (media/output.png)

Accuracy and speed

Metric	Value
End-to-end pointmap PCC vs fp32 torch reference (synthetic pair, test_mast3r.py)	0.9970 (legacy graph TT_FUSED=0: 0.9968)
Pointmap xyz PCC vs reference, 7 real image pairs, mean / min (logs: probe_multipair, 2026-09-13)	head 1: 0.9903 / 0.9718 · head 2: 0.9949 / 0.9863 (legacy graph: 0.9879 / 0.9722 · 0.9943 / 0.9856)
Pointmap xyz PCC vs reference, 12 real CO3Dv2 pairs (mean / min) — measured on the legacy graph (TT_FUSED=0), not re-run on the fused path	head 1: 0.9934 / 0.9777 · head 2: 0.9962 / 0.9901
PairViewer pose AUC@30° vs torch reference (CO3Dv2 apple, 12 pairs) — legacy graph, not re-run on the fused path	38.1 vs 35.0 (est. focal) · 40.4 vs 35.4 (known focal)
Inference, served over HTTP (warm, 30 requests, one 512×512 pair, npz; median / min / max)	73.6 / 73.2 / 79.9 ms device forward · 241 / 240 / 280 ms end-to-end (160 ms of it is the host npz encode; ~4.1 pairs/s); 1.4 s with return_pose
test_mast3r.py --layer end_to_end, best-of-25	73.0 ms per pair (legacy graph: 234.2 ms)

Caveats

• Fixed 512×512 input, exactly one image pair per request (batch 1); non-square images are gray-padded to square, which biases DUSt3R's estimated focal — pass intrinsics when you know them.
• DUSt3R backbone only: no MASt3R matcher / descriptor head and no N-view global alignment; pose is single-pair PairViewer (absolute AUC@30° ~35–40 on CO3Dv2, far below the paper's full pipeline).
• bf16 on device: pointmap xyz stays ≥ 0.99 PCC vs fp32, but confidence channels land at 0.85–0.96 PCC — loosen thresholds tuned on the reference. Dense outputs come base64-encoded (.npz, or 16-bit/8-bit PNG).
• Weights are NAVER's DUSt3R checkpoint under CC-BY-NC-SA-4.0: non-commercial use only, share-alike.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404. Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730), single p150a only.

Licensing

• Weights: naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt, CC-BY-NC-SA-4.0 (non-commercial); fetched by serve from NAVER's repo, not redistributed here.
• Port and serving code (code/), from changh95/tt-mast3r: the DUSt3R port follows the weights' CC-BY-NC-SA-4.0 (share-alike, non-commercial); the serving layer (code/models/server/) is Apache-2.0 per its SPDX headers; tt-metal / tt-nn are Apache-2.0.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	5c30db4ed8a412b6 (sha256, first 16 hex digits)
built	2026-09-13T15:25:51+00:00 by tt-model 0.1.0