solar-open-100b-p150x8
Upstage Solar-Open-100B (102B MoE, 12B active, 128 experts + 1 shared, 131K context) served on eight Blackhole P150 boards with tensor parallelism through vLLM.
Runs on p150x8 (mesh P150x8) — 16,384-token context, up to 32 concurrent sequences.
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
Quickstart
tt-model pull changh95/solar-open-100b-p150x8 --with-weights
tt-model serve changh95/solar-open-100b-p150x8
pull --with-weights downloads the Docker image and the upstage/Solar-Open-100B weights at 1f591439b14055004d1a5d1a975608953a022fea (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.
Performance
Measured on eight Blackhole P150 boards (logical 1x8 mesh, TP=8) with real weights, bfp8 experts and KV cache, greedy decoding, warm caches, on the tt-metal commit this image was built from (2026-09-10). Rows are concurrent users, columns are input/output tokens per user. The 128-token columns use 32 distinct prompts; the 1K-32K columns feed every user the same text and are upper bounds.
Per-user decode, tokens per second:
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 74.6 | 74.1 | 74.1 | 72.5 | 71.4 | 69.0 | 68.0 | 64.5 | 57.5 |
| 2 | 53.2 | 53.2 | 56.5 | 55.9 | 54.3 | 52.9 | 52.4 | 49.5 | 45.5 |
| 4 | 47.6 | 48.1 | 54.9 | 53.8 | 52.6 | 50.0 | 51.0 | 48.8 | 44.1 |
| 8 | 41.5 | 40.7 | 54.1 | 53.2 | 50.3 | 47.4 | 47.4 | 42.9 | 33.0 |
| 16 | 35.1 | 35.3 | 52.4 | 49.8 | 46.5 | 42.9 | 41.7 | 33.6 | 20.8 |
| 32 | 29.0 | 28.3 | 51.0 | 48.1 | 42.6 | 35.6 | 33.6 | 20.5 | 14.1 |
Aggregate decode throughput, tokens per second:
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 75 | 74 | 74 | 73 | 72 | 69 | 68 | 64 | 57 |
| 2 | 106 | 106 | 113 | 112 | 109 | 106 | 105 | 99 | 91 |
| 4 | 191 | 192 | 219 | 216 | 211 | 200 | 204 | 195 | 176 |
| 8 | 332 | 325 | 433 | 426 | 402 | 379 | 379 | 344 | 264 |
| 16 | 561 | 566 | 835 | 796 | 744 | 687 | 666 | 537 | 333 |
| 32 | 926 | 906 | 1632 | 1542 | 1361 | 1141 | 1074 | 657 | 451 |
Time to first token, mean over users, ms (sequential prefill: first user = one prefill, mean = (B+1)/2 prefills):
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 163 | 166 | 416 | 1194 | 1604 | 2927 | 2942 | 7518 | 12562 |
| 2 | 232 | 235 | 596 | 1084 | 2135 | 5827 | 6514 | 10255 | 20064 |
| 4 | 384 | 386 | 1005 | 2197 | 3712 | 9200 | 8015 | 18341 | 31798 |
| 8 | 665 | 665 | 2324 | 3345 | 6407 | 13651 | 16280 | 26913 | 60260 |
| 16 | 1299 | 1258 | 3452 | 6424 | 12341 | 26017 | 26427 | 54661 | 145439 |
| 32 | 2431 | 2490 | 7078 | 12254 | 23506 | 52219 | 51151 | 127929 | 295550 |
Provenance
The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | 584d9f294781f2f9d9ce02d6d2b5db4be72d53a2 |
| vLLM | v0.25.1 |
| vllm-tt-plugin | ba73d09 (branch solar-fixes: async-decode EOS fix, plain-architecture registration) |
code/ digest |
312c02815cac2ae3 (sha256, first 16 hex digits) |
| built | 2026-09-11T04:17:58+00:00 by tt-model 0.1.0 |