changh95/sam3-p150

sam3-blackhole

Meta's SAM3 (Segment Anything with Concepts, 840.5M params) for text-prompted open-vocabulary instance segmentation: an image plus noun-phrase prompts such as "car" gives per-instance masks, boxes and scores. The ViT-H/14 backbone and the 6-layer fusion encoder run on a single Tenstorrent Blackhole p150a via tt-nn (FPN neck, text encoder, decoder and segmentation head stay on the host CPU): 0.77 s end-to-end on a 1920x1080 image vs 3.20 s on CPU (4.15x), ~330 ms / 3.0 FPS warm, 0.998 top-mask IoU vs the CPU reference. Weights are facebook/sam3 (gated, Meta SAM License); port source github.com/changh95/tt-sam3 @ cb728e85275334973b39d5acf6ae376ff2a94a68.

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/sam3-blackhole --with-weights
tt-model serve changh95/sam3-blackhole

pull --with-weights downloads the Docker image and the facebook/sam3 weights at 3c879f39826c281e95690f02c7821c4de09afae7 (into your HF cache; they are not in the image). serve starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

With tt-cli

tt serve changh95/sam3-blackhole          # pulls the image + weights, boots, waits for READY
tt model stop changh95/sam3-blackhole

The port is the one serve printed (20000, or the next free one). This is the model's own HTTP API: the ready card's /v1/models hint and tt-model curl are OpenAI-shaped and do not apply (GET /v1/models exists only so they do not 404).

Prerequisite: gated weights

facebook/sam3 is gated under the Meta SAM License. Request access on the Hub, then hf auth login on the host before serve; the 3.45 GB sam3.pt (+ config.json) lands in your HF cache at the pinned revision and is bind-mounted into the container.

Call it

PORT=20000   # the port serve printed
curl -s localhost:$PORT/health      # {"status":"ok","model":"sam3-blackhole","device":{...}}
curl -s localhost:$PORT/info        # weights repo + revision, limits, source commit

python3 - <<'EOF'
import base64, json, urllib.request
img = base64.b64encode(open("media/source_1.png", "rb").read()).decode()
req = {"image": img, "prompts": ["car"], "threshold": 0.5, "max_instances": 50, "mask_format": "png"}
r = urllib.request.Request("http://127.0.0.1:20000/predict", data=json.dumps(req).encode(),
                           headers={"Content-Type": "application/json"})
out = json.load(urllib.request.urlopen(r))
top = out["results"][0]["instances"][0]
print(out["width"], out["height"], out["timing_ms"], top["score"], top["box_xyxy"])
open("mask0.png", "wb").write(base64.b64decode(top["mask_png"]))   # 8-bit 0/255 PNG, HxW
EOF

Request (POST /predict, JSON): image (base64 PNG/JPEG, any size; it is squashed to the fixed 1008x1008 the port validated), prompts (1-4 noun phrases; each is an independent query and costs one forward), threshold (0.5; keep score > threshold as upstream), max_instances (50, per prompt, highest score first), mask_format (png default | rle COCO uncompressed column-major | none).

Response: width, height, input_size (1008), results[] one per prompt with prompt, presence (image-level presence probability), num_instances and instances[] sorted by score desc, each with score, box_xyxy (pixels in the ORIGINAL image), mask_area (pixels) and mask_png (base64 8-bit 0/255 PNG at the original resolution) or mask_rle; plus timing_ms (decode / preprocess / forward / forward_per_prompt / postprocess / total). 400 for bad input, 503 while starting, 500 with the exception text on failure. Box or point prompts are not supported by this port (text prompts only, one image per call).

First boot

Weights: 3.45 GB download once (HF cache). The first start JIT-compiles 40 ttnn kernels (several minutes) into `/.cache/tt-model/sam3-blackhole/cache/; the server reports READY (Application startup complete) only after two warm-up forwards, so the first request is already fast. Later boots reuse the kernel cache (roughly a minute: load the 3.45 GB checkpoint, convert ~840M params to bfloat8_b/bf16, upload). Host RAM: ~8 GB transient while loading. The decoder runs in eager mode (SAM3_TORCH_COMPILE=0`, ~3% slower than the README's compiled rows) because the runtime image has no C++ compiler.

Demo (from the port README)

Street scene, prompt "car", full pipeline on a p150a.

Input (media/source_1.png)	CPU prediction (media/target_1.png)	TT-NN prediction (media/target_1_ttnn.png)

Results (from the port README)

Per-stage PCC, tt-nn vs the torch reference (random-weight shadow tests):

Stage	PCC	Notes
patch_embed	0.999	CPU F.conv2d + fast H2D (pre-bf16 + ROW_MAJOR + device tilize)
vit_block (single)	0.99	LN + attention + MLP, bfloat8_b weights, HiFi2
vit_backbone (32 blocks)	0.988	window attention (ws=24) + 4 global blocks
FPN neck (per scale)	0.99	CPU conv, bf16 cached weights
Fusion encoder (6 layers)	0.98	device SDPA + cross-attention + FFN
RoPE	exact	atol 1e-6 vs complex reference
E2E masks (200 queries)	0.977	full pipeline mask PCC
E2E scores	0.997	classification score PCC
Top mask IoU	0.998	binary mask overlap of the highest-scoring query

Real data, 1920x1080 street scene, prompt "car":

Impl	Top score	Top mask IoU	Pipeline time
Torch CPU reference	0.9762	-	3.20 s
TT-NN on p150a	0.9779	0.9984	0.77 s

Warm throughput at batch 1 (tt_sam3/benchmark.py): ViT backbone 168 ms (5.97 FPS, 16x CPU); full E2E ~330 ms (3.0 FPS, ~4x CPU). Layout: image 1008x1008 -> patch embed (CPU) -> ViT-H/14 [device, bfloat8_b/HiFi2/fused SDPA/RoPE] -> FPN neck (CPU bf16) -> text encoder (CPU, cached per prompt) -> fusion encoder [device] -> decoder (CPU, bf16 autocast) -> segmentation head (CPU) -> 200 query masks (288x288) + scores. Known caveats: batch 1; each prompt re-runs the full forward (no image-feature caching); non-square images are squashed like upstream; the CPU stages dominate on a small-core host (torch threads capped at min(17, cpu_count)).

Licensing

The weights (facebook/sam3) and the upstream model code (sam3 PyPI package) are governed by Meta's SAM License (2025-11-19; not an OSI licence) and the repo is gated - request access from Meta before pulling. The port code under code/tt_sam3 was written by Hyunggi Chang (github.com/changh95/tt-sam3 @ cb728e85) and is published under the same terms, since a port cannot grant more than its upstream does. code/tt_sam3/assets/bpe_simple_vocab_16e6.txt.gz is the CLIP BPE vocabulary from the upstream sam3 repository (MIT-licensed CLIP asset), vendored because the PyPI wheel omits it. Weights are never redistributed by this package; they are fetched from the upstream repo under its own terms.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest	19f718f9cdbd063a (sha256, first 16 hex digits)
built	2026-09-12T05:52:10+00:00 by tt-model 0.1.0