mast3r-blackhole

DUSt3R two-view 3D reconstruction (naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt: ViT-L/16 encoder, dual-branch decoder, two DPT heads) running end to end on a single Tenstorrent Blackhole p150a via tt-nn: one image pair in, two dense 512x512 3D pointmaps with confidence out in 228 ms/pair (4.4 fps, 22x the fp32 CPU reference, PCC 0.9962), plus optional PairViewer pose recovery. Only the DUSt3R backbone is ported (no MASt3R matcher head). The weights are NAVER's CC-BY-NC-SA-4.0 checkpoint (non-commercial, share-alike); the port code (github.com/changh95/tt-mast3r @ 3c3c74f904defcac3d7ba4a0d9db6c91d230c356) is published under the same terms.

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/mast3r-blackhole --with-weights
tt-model serve changh95/mast3r-blackhole

pull --with-weights downloads the Docker image and the naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt weights at 61c57447d7b0adc8a1a30b2b0adec7a8935aa2a3 (into your HF cache; they are not in the image). serve starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

With tt-cli

tt serve changh95/mast3r-blackhole        # first use pulls the image + 2.3 GB of weights
tt model stop changh95/mast3r-blackhole

The server listens on the port serve printed (20000, or the next free one). It is not an OpenAI-style API: GET /v1/models exists only so the ready card and tt-model curl do not 404 -- use the routes below.

Routes

PORT=20000                                   # the port serve printed
curl -s localhost:$PORT/health               # {"status": "ok" | "starting", ...}
curl -s localhost:$PORT/info | python3 -m json.tool

POST /predict takes exactly one image pair (base64 PNG/JPEG, any size -- each view is gray-padded to square and resized to 512x512 server-side) and returns both pointmaps. With the two demo images from media/:

python3 - <<'EOF'
import base64, io, json, urllib.request, numpy as np
b64 = lambda p: base64.b64encode(open(p, "rb").read()).decode()
req = {"image1": b64("media/source_1.png"), "image2": b64("media/source_2.png"),
       "output_format": "npz", "return_pose": True}
r = urllib.request.Request("http://127.0.0.1:20000/predict", json.dumps(req).encode(),
                           {"Content-Type": "application/json"})
resp = json.load(urllib.request.urlopen(r, timeout=600))
npz = np.load(io.BytesIO(base64.b64decode(resp["npz_b64"])))
print({k: (npz[k].shape, str(npz[k].dtype)) for k in npz.files})
print("pose:", resp["pose"])
print("timing_ms:", resp["timing_ms"])
EOF

Request fields (JSON body):

field	type	default	meaning
image1, image2	base64 str	required	view 1 (reference camera) and view 2; or images: [b64, b64]
output_format	"npz" | "png"	npz	npz: one base64 .npz with pts3d1, pts3d2 float32 (512,512,3) and conf1, conf2 float16 (512,512); png: per-view 16-bit depth PNG + 8-bit confidence PNG (preview only)
return_pose	bool	false	also run the symmetric (view 2, view 1) forward and recover view 2's pose with DUSt3R's PairViewer (doubles device time)
intrinsics	3x3 list	none	pinhole K of view 2 in its original pixel coordinates; when given, PnP uses it instead of the estimated focal (known-K pose)
conf_pct	float (0, 100]	50	PnP inlier set: top conf_pct % of view-2 pixels by confidence

Response (JSON): npz_b64 (or view1/view2 PNG blocks), preprocess (per view: orig_w, orig_h, offx, offy, pad_side, out_size, scale -- canonical pixel (u, v) = ((x + offx) * scale, (y + offy) * scale)), summary (per-view conf / depth statistics), pose (R 3x3, t 3, focal_1, focal_2, used_known_intrinsics; the extrinsic of view 2 in the view-1 frame, X_cam2 = R @ X_cam1 + t; null when not requested or when PnP fails) and timing_ms (decode, preprocess, forward, forward_sym, pose, encode, total). Both pointmaps are expressed in the camera-1 frame (DUSt3R convention); depth is pts3d[..., 2], in DUSt3R's own scale (not metres). conf = 1 + exp(c) >= 1.

A ready-made check: python3 code/models/server/smoke_test.py --url http://127.0.0.1:$PORT posts the pair above and prints one PASS/FAIL line.

First boot

• Weights: model.safetensors (2.28 GB, fp32) from naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt at the pinned revision, into your HF cache (public repo, no token needed). The server holds the fp32 state dict on the host (~2.3 GB RAM) and uploads bf16 copies to the chip.
• The first forward JIT-compiles every kernel -- expect a few minutes before Application startup complete; the kernel cache persists under ~/.cache/tt-model/mast3r-blackhole/cache, so later boots are much faster.
• Errors: 400 for bad input (wrong image count, undecodable base64, bad K), 503 while warming up, 500 with the exception text on a device failure. One pair per request; requests are serialised on the single chip.

Results (from the port's README, p150a)

	torch CPU reference	ttnn port on p150a	ratio
latency / pair (B=1, 512x512)	~5000 ms (fp32)	228 ms	22x
throughput	0.20 fps	4.39 fps	+2095 %
min-PCC port vs ref, synthetic	--	0.9962	--
min-PCC port vs ref, CO3Dv2 apple xyz	--	0.9777 (head1), 0.9901 (head2)	--
AUC@30 (CO3Dv2 apple, 12 pairs, est-focal)	35.0	38.1	+3.1
AUC@30 (CO3Dv2 apple, 12 pairs, known-focal)	35.4	40.4	+5.0

PCC on 12 real CO3Dv2 pairs (512x512 pad-to-square): pcc_head1_xyz mean 0.9934 / min 0.9777, pcc_head2_xyz 0.9962 / 0.9901, conf channels 0.94-0.96 mean (min 0.85); depth rel. error 4.1 % (head1) / 5.8 % (head2). Absolute pose AUC sits far below DUSt3R's paper numbers because the minimal reference uses pad-to-square preprocessing and a single-pair PairViewer instead of the N-view global aligner -- the port-vs-reference delta is the validated signal. bf16 weights everywhere; HiFi4 + fp32 accumulation on the DPT convs. Full write-up: code/co3d_eval_results.md; optimisation log: code/results.tsv.

Sample output (CO3Dv2 apple, frames 0 / 40)

source 1	source 2	predicted point cloud, two viewpoints

PairViewer pose for this pair vs CO3D GT (63.7 deg baseline): estimated focal 14.6 deg rotation / 27.0 deg translation-direction error; known focal 9.5 deg / 14.4 deg (media/pose_accuracy.md). Known intrinsics help because pad-to-square gray bars make DUSt3R under-estimate the focal on wide images.

Limits

• Fixed 512x512 input (32x32 tokens); non-square images are gray-padded, which biases the estimated focal -- pass intrinsics when you know them.
• DUSt3R backbone only: no MASt3R matcher / descriptors, no N-view global alignment.
• Confidence channels land at PCC ~0.85-0.96 vs the fp32 reference; loosen thresholds tuned on the reference.

Licensing

Non-commercial use only. The upstream DUSt3R checkpoint (NAVER) is CC-BY-NC-SA-4.0; the weights are not redistributed here -- serve fetches them from naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt under NAVER's terms. The port code (Hyunggi Chang, github.com/changh95/tt-mast3r @ 3c3c74f904defcac3d7ba4a0d9db6c91d230c356) is published under the same CC-BY-NC-SA-4.0 terms, since a port cannot grant more than its upstream does. tt-metal / tt-nn are Apache-2.0.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	7be4a1a91d918f4b (sha256, first 16 hex digits)
built	2026-09-12T05:15:03+00:00 by tt-model 0.1.0