gpt-oss-20b-p150x8
OpenAI gpt-oss-20b (21B MoE, 3.6B active, 32 experts top-4, 131K context, harmony chat format) served on eight Blackhole P150 boards with tensor parallelism through vLLM, up to 32 concurrent users.
Runs on p150x8 (mesh P150x8) — 32,768-token context, up to 32 concurrent sequences.
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
Quickstart
tt-model pull changh95/gpt-oss-20b-p150x8 --with-weights
tt-model serve changh95/gpt-oss-20b-p150x8
pull --with-weights downloads the Docker image and the openai/gpt-oss-20b weights at 6cee5e81ee83917806bbde320786a8fb61efebee (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.
Performance
Measured on eight Blackhole P150 boards (logical 1x8 mesh, TP=8) with real openai/gpt-oss-20b weights, bfp8 experts and KV cache,
greedy decoding, warm caches, on the tt-metal branch this image was built from (final regression sweep, 2026-09-05, commit f91dc77e50f).
Rows are concurrent users, columns are input/output tokens per user; every user gets a distinct prompt. Cells marked – were not run
(context budget of the sweep). Decode steps are traced; prefill runs one user at a time, so TTFT grows with the number of users.
The image is built from tt-metal ed28fce3a4c, which adds the serving fixes made after that sweep (trace-safe prefill programs, the
padded-prefill page-table fix, tenstorrent/tt-metal#56324, and keeping the server's padding rows out of the expert union); served
through vLLM in this container the model streams 61 tok/s to a single user and 1730 tok/s aggregate to 32 users (54 tok/s each) on 128-token prompts.
Per-user decode, tokens per second:
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 58.1 | 57.8 | 56.8 | 57.1 | 56.8 | 56.5 | 56.9 | 56.4 | 55.3 |
| 2 | 55.6 | 55.1 | 57.1 | 57.5 | 57.4 | 55.8 | 56.1 | 55.7 | 55.2 |
| 4 | 54.5 | 53.9 | 57.8 | 57.5 | 56.5 | 56.6 | 56.2 | 56.0 | 53.2 |
| 8 | 50.1 | 49.6 | 55.2 | 55.1 | 54.6 | 53.7 | 54.0 | 53.9 | 49.4 |
| 16 | 48.7 | 47.8 | 54.2 | 54.0 | 53.4 | 54.0 | 53.9 | 50.9 | 49.1 |
| 32 | 48.9 | 48.2 | 56.7 | 56.7 | 54.6 | 51.3 | 52.3 | 44.9 | 40.2 |
Aggregate decode throughput, tokens per second:
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 58 | 58 | 57 | 57 | 57 | 56 | 57 | 56 | 55 |
| 2 | 111 | 110 | 114 | 115 | 115 | 112 | 112 | 111 | 110 |
| 4 | 218 | 216 | 231 | 230 | 226 | 227 | 225 | 224 | 213 |
| 8 | 401 | 396 | 441 | 440 | 437 | 430 | 432 | 431 | 395 |
| 16 | 778 | 764 | 867 | 864 | 854 | 864 | 863 | 814 | 786 |
| 32 | 1564 | 1542 | 1813 | 1816 | 1746 | 1642 | 1675 | 1438 | 1288 |
Time to first token, mean over users, ms (sequential prefill: first user = one prefill, mean = (B+1)/2 prefills):
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 59 | 57 | 191 | 250 | 452 | 891 | 890 | 1787 | 3685 |
| 2 | 77 | 77 | 189 | 345 | 664 | 1322 | 1324 | 2663 | 5526 |
| 4 | 118 | 121 | 314 | 570 | 1094 | 2199 | 2191 | 4430 | 9526 |
| 8 | 218 | 222 | 818 | 1273 | 2029 | 4024 | 4000 | 8358 | 20869 |
| 16 | 477 | 524 | 1114 | 1962 | 3835 | 7829 | 7823 | 18792 | 31318 |
| 32 | 728 | 738 | 2056 | 3803 | 7697 | 17954 | 18896 | 42505 | 115754 |
Provenance
The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | ed28fce3a4cd5f14f17fbb3b1a58e9903d40ed86 |
| vLLM | v0.25.1 |
| vllm-tt-plugin | ba73d09 (branch solar-fixes; the GPT-OSS registration is upstream built-in) |
code/ digest |
ed4b547cf1cf447f (sha256, first 16 hex digits) |
| built | 2026-09-12T13:31:38+00:00 by tt-model 0.1.0 |