moge-2-p150

MoGe-2 (Microsoft; checkpoint Ruicheng/moge-2-vitl-normal, MIT) -- monocular metric geometry from one RGB image: metric depth, camera-space point map, surface normals, validity mask and recovered intrinsics. The DINOv2 ViT-L/14 encoder and the full ConvStack decoder run on a single Tenstorrent Blackhole p150a via tt-nn; the decoder is metal-traced. At the validated 1920x1080 / 1800-token setting the port measures 3.85 FPS (single command queue, as served; 4.13 FPS with MOGE_2CQ=1) with point-map PCC 0.9999 vs the torch reference. Vendored microsoft/MoGe and utils3d are MIT, the port glue is Apache-2.0; port source github.com/changh95/tt-MoGe @ b64091f2 (the serving adapter under code/tt_moge/server lives in this repo).

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/moge-2-p150 --with-weights
tt-model serve changh95/moge-2-p150

pull --with-weights downloads the Docker image and the Ruicheng/moge-2-vitl-normal weights at cb0e8bbd6b1e243589717c78e750b1ba4c093acf (into your HF cache; they are not in the image). serve starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

With tt-cli

tt serve changh95/moge-2-p150          # pulls the image + weights on first use
tt model stop changh95/moge-2-p150

The server listens on the port serve printed (20000, or the next free one). It is the model's own HTTP API -- not an OpenAI endpoint: GET /v1/models only exists so the ready card does not 404, and tt-model curl does not apply. The real routes are GET /health, GET /info and POST /predict.

Call it

PORT=20000
curl -s localhost:$PORT/health          # {"status":"ok",...} once warm (503s from /predict before)
curl -s localhost:$PORT/info            # weights revision, canonical canvas, limits, license

# Any RGB PNG/JPEG. Dense outputs come back at the ORIGINAL resolution.
IMG=media/source.png
python3 - "$IMG" "$PORT" <<'EOF'
import base64, io, json, sys, urllib.request
import numpy as np
img, port = sys.argv[1], sys.argv[2]
req = {"image": base64.b64encode(open(img, "rb").read()).decode(),
       "output_format": "npz",     # npz (default) | png | json (<= 512x512 only)
       "fit": "pad",               # pad = letterbox onto the 1920x1080 canvas (keeps geometry) | stretch
       "fov_x": None,              # known horizontal FoV in degrees -> only the depth shift is solved
       "apply_mask": True,         # invalid pixels -> inf depth/points, zero normal
       "force_projection": True}   # points re-projected from depth + intrinsics
r = urllib.request.Request(f"http://127.0.0.1:{port}/predict", data=json.dumps(req).encode(),
                           headers={"Content-Type": "application/json"})
out = json.loads(urllib.request.urlopen(r, timeout=600).read())
print({k: v for k, v in out.items() if k not in ("outputs", "encoding")})
z = np.load(io.BytesIO(base64.b64decode(out["outputs"]["npz"])))
print({k: (z[k].shape, str(z[k].dtype)) for k in z.files})
EOF

Response fields: height, width (original), canonical (canvas, fit, the region the image occupied, num_tokens, token_grid), metric_scale, intrinsics (3x3, normalized: multiply row 0 by W and row 1 by H; also given as intrinsics_pixels), fov_x_deg/fov_y_deg, shift, mask_coverage, depth_m {min, median, max}, timing_ms {decode, preprocess, device, postprocess, encode, total} and outputs:

• npz -- base64 of np.savez_compressed: points float32 [H,W,3] metres (camera space, OpenCV axes), depth float32 [H,W] metres, normal float16 [H,W,3], mask uint8 [H,W] (1 = valid), intrinsics float32 [3,3], metric_scale float32.
• png -- depth_png16 (16-bit PNG, metres = value x encoding.depth_png_scale, 0 = invalid), normal_png (8-bit RGB, n = v/255*2-1), mask_png (255 = valid). Add include_depth_png: true to attach depth_png16 to the other formats.
• json -- nested lists (null = invalid); refused above 512x512 pixels.

Errors: 400 for an undecodable image or bad fields, 503 while the server is still loading/warming, 500 with the exception text if the forward fails. One image per request, requests are serialised on the chip.

Smoke test (what the hardware validation runs): python code/tt_moge/server/smoke_test.py --url http://127.0.0.1:$PORT -> one PASS ... line.

First boot

pull --with-weights fetches model.pt (1.32 GB) into your HF cache. The first serve loads the fp32 torch reference on the host (1.3 GB RAM resident, ~2.6 GB peak), uploads the encoder weights (bf16) to the chip, JIT-compiles the ViT-L (seq 1825) and conv kernels and captures the decoder trace on a 1920x1080 warm-up image -- several minutes, all before Application startup complete. Kernels persist under `/.cache/tt-model/moge-2-p150/cache`, so later boots take ~1-2 minutes. No gated repos, no tokens, no extra files.

How inputs are handled

The decoder trace is bound to one token grid (32x57 for 1920x1080 at 1800 tokens), so every request is placed on that canvas: fit: pad letterboxes with a neutral gray border and the letterbox is masked out of the focal/shift solve and cropped away before the outputs are resized back to your image; fit: stretch squashes instead. Batch is 1. Changing MOGE_NUM_TOKENS / MOGE_CANONICAL_SIZE (serve env) re-captures the trace for that setting at boot; the numbers below are for the default.

Demo (python -m scripts.make_demo, single p150a)

Input (media/source.png)	Depth (media/depth.png)	Normal (media/normal.png)

Results (num_tokens = 1800, 1920x1080, single p150a)

Per-stage PCC vs the torch reference (tt_moge/tests/test_relative_pcc.py, random image):

Stage	PCC	Shape
encoder_x (summed DINOv2 features)	0.9940	(1, 1024, 32, 57)
encoder_cls (scale-head input)	0.9944	(1, 1024)
neck_L0 / L1 / L2 / L3 / L4	1.0000 / 0.9995 / 0.9999 / 0.9997 / 1.0000	pyramid
points_finest	1.0000	(1, 3, 512, 912)
e2e points / depth / normal / mask	0.9997 / 0.9980 / 0.9974 / 0.9966	(1, 1080, 1920, ...)

Real image (tt_moge/tests/test_pretrained_eval.py): point map 0.9999, depth 0.9998, normal 1.0000, mask 1.0000 (PCC in float64 -- z = exp(z_raw) overflows a float32 Pearson).

Performance (python -m tt_moge.benchmark --impl ttnn, warm, device-synchronised median):

Metric	Value
Median latency / throughput, 2 CQs (MOGE_2CQ=1)	242 ms / 4.13 FPS
Throughput, single CQ (the served default)	3.85 FPS
Point-map PCC vs torch	0.9999
Peak DRAM	846 MiB

Optimisation trajectory: torch CPU 0.55 FPS -> ViT-L encoder on chip 1.32 -> conv decoder on chip (bfp8 weights) 2.11 -> cached UV maps 2.49 -> metal-traced decoder 3.85 -> 2 CQs 4.13. Rejected: HiFi2 in the encoder (no speedup, PCC 0.9943) and tracing encoder + decoder as two traces (hangs).

Caveats

• Encoder eager, decoder traced; host does token prep, projection+sum, scale MLP, final resize/remap and the focal/shift recovery (scipy least-squares on a 64x64 grid).
• Depth/points are stored as float32 on purpose: the raw exp remap can reach ~1e11 in invalid regions, which overflows float16.
• The reference import chain needs opencv-python-headless and scipy even though the TT path never calls cv2.

Source layout (code/)

tt_moge/tt/ (ttnn encoder, ConvStack decoder, TtMoGe forward + trace), tt_moge/reference/ (vendored microsoft/MoGe + utils3d, checkpoint loader), tt_moge/server/ (this ASGI app, post-processing, smoke test), tt_moge/tests/ (PCC, pretrained eval, Tracy perf), tt_moge/benchmark.py, scripts/ (download_weights.sh, make_demo.py), conftest.py (pytest device fixture). Run the tests from code/ with PYTHONPATH=$TT_METAL_HOME:$TT_METAL_HOME/ttnn:$PWD.

Licensing

Upstream model and weights: MIT (https://github.com/microsoft/MoGe/blob/main/LICENSE; checkpoint https://huggingface.co/Ruicheng/moge-2-vitl-normal). Vendored moge and utils3d (https://github.com/EasternJournalist/utils3d): MIT. DINOv2 backbone code: Apache-2.0 (Meta). Port glue and this server: Apache-2.0, written by Hyunggi Chang (https://github.com/changh95/tt-MoGe). Weights are not redistributed here; they are fetched from the upstream repo under its own terms.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	c4f03dbab72e4ff7 (sha256, first 16 hex digits)
built	2026-09-12T11:34:18+00:00 by tt-model 0.1.0