exaone-4.5-33b-p150x8
LG AI Research EXAONE-4.5-33B (dense 33B, hybrid sliding-window/global attention, reasoning model with thinking on by default, 131K context) served on eight Blackhole P150 boards with tensor parallelism through vLLM, up to 32 concurrent users.
Runs on p150x8 (mesh P150x8) — 131,072-token context, up to 32 concurrent sequences.
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
Quickstart
tt-model pull changh95/exaone-4.5-33b-p150x8 --with-weights
tt-model serve changh95/exaone-4.5-33b-p150x8
pull --with-weights downloads the Docker image and the LGAI-EXAONE/EXAONE-4.5-33B weights at 570aa4b15a4f45ba1133072b45f50198f6e3b4fd (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.
Performance
Measured against this container as served (vLLM 0.25.1 through the Tenstorrent plugin) on eight Blackhole P150 boards (logical 1x8 mesh,
TP=8), real LGAI-EXAONE/EXAONE-4.5-33B weights, greedy decoding, 2026-09-12. Rows are concurrent users, columns are input/output tokens per
user; each user sends a distinct natural-text prompt and all users of a cell are submitted together, so the time to first token includes waiting
for the users prefilled earlier (the server prefills one user at a time). Numbers are steady state: the server captures a new prefill trace the first time it sees a (users, prompt length) pair for the traced
lengths 128 and 1024, which costs a few seconds once, so those cells were run again after the grid and the repeat is reported. Per-user tokens per second are measured from the streamed responses, so they include the
server's per-token overhead; the batch-1 decode step of the bare model is ~28 ms (36 tok/s).
Sampling runs on device for decode (sample_on_device_mode: decode_only); with the earlier host-side sampling the streamed rate
alternated between about 23 and 30 tok/s across runs, with on-device sampling three repeats of the key cells agreed within 1 %. Five 32-user
cells with prompts of 4096 tokens and longer are marked – : that burst exceeds the 133K-token KV cache of this profile and the scheduler stalled
instead of queuing, so they were not measured.
The grid was measured on the image built from tt-metal 39a7870be5d (the merged EXAONE pull request); this image is built from tt-metal
f72689310a4, the same code plus a fix for a serving bug in the shared prefill page-table code (the padded prefill tail could be written into KV blocks the request did not own, giving non-deterministic
or corrupted answers for prompts longer than 128 tokens). The fix changes what the pad tail writes, not the work per token; a repeat of the
batch-1 and batch-32 cells on the fixed image matched the grid within the run-to-run spread.
Per-user decode, tokens per second:
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 37.2 | 36.2 | 35.1 | 34.2 | 31.6 | 30.5 | 30.2 | 28.4 | 25.0 |
| 2 | 37.2 | 36.4 | 35.6 | 34.1 | 31.5 | 30.3 | 30.3 | 28.3 | 25.0 |
| 4 | 37.2 | 36.4 | 35.6 | 34.1 | 31.4 | 30.5 | 30.3 | 28.5 | 25.1 |
| 8 | 37.2 | 36.4 | 35.6 | 34.1 | 31.7 | 30.6 | 30.3 | 28.7 | 25.2 |
| 16 | 37.0 | 36.2 | 35.5 | 34.1 | 31.5 | 29.6 | 28.6 | 28.6 | 25.2 |
| 32 | 36.7 | 35.5 | 34.6 | 32.9 | – | – | – | – | – |
Aggregate decode throughput, tokens per second (users x per-user rate while decoding):
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 37 | 36 | 35 | 34 | 32 | 30 | 30 | 28 | 25 |
| 2 | 74 | 73 | 71 | 68 | 63 | 61 | 61 | 57 | 50 |
| 4 | 149 | 146 | 142 | 137 | 126 | 122 | 121 | 114 | 101 |
| 8 | 297 | 291 | 285 | 273 | 253 | 245 | 243 | 230 | 202 |
| 16 | 593 | 579 | 567 | 545 | 504 | 473 | 458 | 457 | 404 |
| 32 | 1174 | 1136 | 1107 | 1053 | – | – | – | – | – |
Time to first token, mean over users, ms (all users of a cell submitted together; the server prefills them one at a time):
| users \ in/out tokens | 128/128 | 128/1024 | 1024/128 | 2048/128 | 4096/128 | 8192/128 | 8192/1024 | 16384/128 | 32768/128 |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 77 | 89 | 169 | 324 | 489 | 1028 | 1032 | 2206 | 5224 |
| 2 | 124 | 149 | 267 | 519 | 952 | 2033 | 2010 | 4393 | 10489 |
| 4 | 265 | 281 | 731 | 1320 | 2441 | 5124 | 4964 | 11212 | 21664 |
| 8 | 532 | 432 | 1374 | 2696 | 4485 | 9384 | 9559 | 18708 | 35054 |
| 16 | 1086 | 1093 | 2794 | 5415 | 11318 | 15750 | 15733 | 28071 | 60324 |
| 32 | 2046 | 2317 | 6054 | 10780 | – | – | – | – | – |
Provenance
The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | f72689310a45c3fbc753a2dc3d4f06dbe3161e77 |
| vLLM | v0.25.1 |
| vllm-tt-plugin | ba73d09 (branch solar-fixes) |
code/ digest |
724aaf6f137056a7 (sha256, first 16 hex digits) |
| built | 2026-09-12T07:16:06+00:00 by tt-model 0.1.0 |