sam3-p150

Meta's SAM3 (Segment Anything with Concepts, 840.5M params) for text-prompted open-vocabulary instance segmentation, with the ViT-H/14 backbone and fusion encoder running on one Tenstorrent Blackhole p150a via tt-nn: image plus noun phrases in, per-instance masks, boxes and scores out. Weights: facebook/sam3 · Paper: arXiv:2511.16719 · Upstream code: facebookresearch/sam3 · Port: changh95/tt-sam3

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/sam3-p150 --with-weights
tt-model serve changh95/sam3-p150

• Weights facebook/sam3 at 3c879f39826c go to your HF cache; the image does not contain them.
• Serves on port 20000 (or the next free port); ready when the log says Application startup complete.

Run with tt-cli

tt serve changh95/sam3-p150
printf '{"image":"%s","prompts":["car"]}' "$(base64 -w0 media/source_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/sam3-p150

• POST /predict: image (base64 PNG/JPEG), prompts (1–4 noun phrases); optional threshold (0.5), max_instances (50), mask_format (png | rle | none).
• GET /health, GET /info.

Response

{"width": 1920, "height": 1080, "input_size": 1008, "threshold": 0.5, "mask_format": "png",
 "results": [{"prompt": "car", "presence": 0.995, "num_instances": 3, "instances": [
    {"score": 0.9727, "box_xyxy": [545.8, 447.9, 1159.1, 854.4], "mask_area": 175127, "mask_png": "..."},
    {"score": 0.818, "...": "..."}, {"score": 0.788, "...": "..."}]}],
 "timing_ms": {"decode": 26.5, "preprocess": 7.5, "forward": 570.8, "forward_per_prompt": 570.8, "postprocess": 20.6, "total": 625.5}}

• One results entry per prompt; instances are sorted by score, box_xyxy is [x0, y0, x1, y1] in original image pixels.
• mask_png is a base64 8-bit 0/255 PNG at the original H×W; rle gives COCO uncompressed column-major {size, counts}, none omits masks.

Demo

Synthetic street scene, prompt "car".

Input (media/source_1.png)	Masks on p150a (media/target_1_ttnn.png)

Accuracy and speed

Metric	Value
Top-mask IoU vs torch CPU reference (demo image, "car")	0.998 (top score 0.978 vs 0.976)
End-to-end mask PCC vs torch (200 queries, random-weight shadow) · backbone PCC	0.977 · 0.988
Inference, served over HTTP (warm, batch 1, one prompt, 1920×1080)	575 ms forward · ~610–635 ms end-to-end (1.6 req/s)

Caveats

• Every image is squashed to 1008×1008; one image per request, batch 1; text prompts only (no box/point prompts), and each of the 1–4 prompts costs a full forward.
• facebook/sam3 is gated under the Meta SAM License: request access and hf auth login on the host before serve (3.45 GB sam3.pt).
• Only the ViT backbone and fusion encoder run on device (bfloat8_b/bf16); the FPN neck, text encoder, decoder and mask head run eagerly on the host CPU and dominate served latency.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
• Validated on tt-metal v0.78.0-dev20260820 (main 8b98410e730), single p150a only.

Licensing

• Weights: facebook/sam3, Meta SAM License (gated, not OSI).
• Port and serving code (code/): from changh95/tt-sam3, distributed under the same SAM License terms; code/tt_sam3/assets/bpe_simple_vocab_16e6.txt.gz is the MIT-licensed CLIP BPE vocabulary vendored from the upstream sam3 repo.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	669b8cdb523d4ba1 (sha256, first 16 hex digits)
built	2026-09-12T13:33:23+00:00 by tt-model 0.1.0