D3-Turbo-NPU2
vllm-sr/d3-turbo v1.1 (689b3af1; Decision 3.0, the 27B multimodal decision
model of vLLM Semantic Router, Qwen3.5-27B text tower) for AMD Ryzen AI NPUs, served by
FastFlowLM 1.0.11+.
It answers System One requests -- choice, yes / no and score questions about a state, optionally with images and videos,
with a probability for every option; box questions (where is it in an image or a video) and track questions (a box per
moment over a whole video) as well -- over the same POST /v1/systemone API as the upstream package. Up to 32,768 input
tokens per question. Like upstream d3-turbo it reads the state once per request and answers every question on it
("ask once, answer many"), so many questions about one input cost far less than one request per question.
flm pull d3:turbo # 16.6 GB download
flm serve --decision d3:turbo # optionally next to a chat model: flm serve qwen3.5:9b --decision d3:turbo
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
-d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
"criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'
Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most
1.6 MP), shared by every question of the request. d3:lite (0.8B), d3:nano (2B) and d3:mini (4B) are the smaller
siblings; they answer a single short question in well under a second, d3:turbo needs a few seconds (see Speed).
What this package supports on the NPU
| Question / input | Status |
|---|---|
choice, noul (yes / no), score on text |
supported |
| the same with images in the state (several images per request) | supported |
| Maximum length | 32,768 tokens per question (a longer prompt is answered with max_length_exceeded; nothing is truncated) |
long_context=True (upstream: up to 131,072 tokens) |
not supported (the request is refused with HTTP 422) |
| Question / input | Status |
|---|---|
box (where is it in an image; also at a moment of a video with at) |
supported: one box per question with a calibrated confidence |
assets (images inside the state, a question's instructions or its options) |
supported |
| videos in the state (2 frames per second, 4 to 32 frames, at most 0.2 MP per frame) | supported; a video is at most 60 s and 16,384 vision tokens per request; decoding mp4 / webm / mkv / mov / gif needs the FFmpeg shared libraries on the machine (without them the request is refused with HTTP 422; frame lists {"frames": [...]} always work) |
track (a box per moment over a whole video, 1 or 2 updates per second) |
supported; at most 256 moments per request (D3_MOMENT_ROWS) |
GET /v1/systemone/models lists exactly what the running build accepts.
Requirements
- An AMD Ryzen AI processor with an XDNA 2 NPU (this package was tested on a Strix Halo machine), Linux with the XRT stack, FastFlowLM as above.
- Memory: the package is 16.6 GB on disk and about 17.9 GB resident once loaded. Measured peak resident memory of the server process: 17.4 GB for prompts up to 4k tokens, 22.4 GB at 24k and 26.3 GB at 32,768 tokens (a fresh server of the final FastFlowLM build, one request; 25.3 GB on the development build); the FLM catalog reserves a footprint of 26 GB (plus 8.6 GB of locked scratch memory). All numbers are from a 128 GB machine; smaller machines were not tested, plan for a 32 GB or larger system and do not run it next to other large models.
- Load time: about 8 s for the engine with the files in the page cache;
flm servealso checks the 16.6 GB of files against their hashes first (76 s), so it is ready about 85 s after the start.
What was changed
- Weights: the Qwen3.5-27B text tower of d3-turbo quantized to FastFlowLM's Q4NX format: every projection Q4_K with
GPTQ-calibrated rounding (our own sequential GPTQ over all 64 layers, 400 tensors; calibration: 300 synthetic System
One questions of up to 2k tokens with the model's
prefix_bidirmask, none from the Decision Index), token embedding Q8_0, norms and the Gated DeltaNetalpha/betaparameters kept in BF16. - Vision tower:
vision_weight.q4nxis made from d3-turbo's own vision encoder (27 layers, 1152 wide, 5120 out), which differs from stock Qwen3.5-27B's in every tensor (it was fine-tuned), then quantized to Q4_K. It runs on FastFlowLM's existing vision kernels. - Attention numerics (
prefix_bidir): as upstream, the state block attends bidirectionally to itself, each question block attends to the state and to itself, never the other way, and the Gated DeltaNet layers stay causal. The engine runs the state once and the question blocks on its cached keys / values and recurrent state; a question's answer does not depend on the other questions of the request.usage.input_tokenscounts the state once plus every question block, as upstream. - Unchanged:
readout.safetensors,decision_config.json,tokenizer.json,tokenizer_config.json,chat_template.jinja,LICENSEare the upstream files byte for byte (MANIFEST.sha256);NOTICEadds the modification statement. Prompts, token ids and answer objects follow the upstream runtime, with one known difference: FastFlowLM tokenizes with the model's owntokenizer.json, while upstream'stransformerstokenizer drops the combining-mark class from its pre-tokenizer, so text with combining marks (for example an emoji variation selector) can have one token more upstream (the token counts differ by one on 3 of 1,480 test prompts).
Accuracy (against the upstream BF16 model)
All comparisons use the same requests: the upstream runtime in BF16 as the reference (run on an AMD iGPU, the vision tower of the image references on the CPU) and this package on the NPU.
| this package on the NPU | upstream BF16 | |
|---|---|---|
| Final-norm hidden state vs BF16, cosine (736 text questions, median / minimum) | 0.9990 / 0.981 | -- |
| Probability error vs BF16 (736 text questions, median / 99th percentile / maximum) | 0.011 / 0.13 / 0.50 | -- |
| Same top answer as BF16 (736 text questions) | 95.7% | -- |
| Same, 5 long prompts of 6k to 24k tokens (cosine median) | 5 / 5 (0.9980) | -- |
| Same, 60 questions with images (cosine median) | 91.7% (0.9941) | -- |
The plain Q4_K quantization of the same model without GPTQ agreed on 91.0% of the text questions (cosine 0.9964), so GPTQ is what makes this package usable. Rounding noise of 4-bit weights accumulates over 64 layers, so this 27B model sits at the level of d3:nano (95.7%) and below d3:mini (97.0%) on this measure. The image rows are lower than the text rows mainly because of the vision tower: on FastFlowLM's current vision kernels its features differ from the BF16 tower's by a relative L2 error of 0.31 (the same kernels give 0.58 on the d3:mini tower, which ships).
| Decision Index public-suite subset | this package on the NPU | upstream BF16 (same requests) |
|---|---|---|
| mean benchmark score (782 requests, 42 benchmarks) | 77.18 | 76.12 |
| top-1 agreement with BF16 on the same requests (7,026 choice answers) | 98.9% | -- |
Mean benchmark score, this package minus BF16: +1.06 pt (standard error 0.48 pt; request-weighted +1.31 pt). The package is not worse than the BF16 model on this slice (d3:lite -0.59 pt, d3:nano -1.25 pt, d3:mini -0.99 pt); the mean absolute probability difference is 0.0086. The slice is the first 800 requests of the same 4,002-request sample as d3:nano and d3:mini (round-robin over 43 benchmarks).
| Decision Index public vision suite subset (600 questions, 10 benchmarks) | this package on the NPU | upstream BF16 |
|---|---|---|
| accuracy | 79.0% | 81.0% |
| top-1 agreement with BF16 on the same questions | 88.8% | -- |
Paired accuracy, BF16 minus this package: +2.0 pt (95% CI -0.3 to +4.3; d3:mini: +2.5 pt, CI +0.5 to +4.5). The largest gap is Hateful Memes moderation (63.3% vs 76.7% on 60 questions). The image rows are limited by the vision tower: on FastFlowLM's current vision kernels its features differ from the BF16 tower's by a relative L2 error of 0.31.
The subset is a seeded round-robin sample over the suite's benchmarks (the same rows as for the d3:lite, nano and mini cards), not the full suite, so its scores are not comparable with the upstream board numbers (d3-turbo: 70.2 text, 72.9 vision, upstream's own estimates). The vision subset is 60 seeded questions from each of 10 benchmarks (the suite without the gated Winoground).
Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM)
The engine columns were measured on a loaded machine (load average 10 to 80 from other jobs on the same box during most
runs, CPU and memory bandwidth shared), so treat them as upper bounds. The idle machine columns were measured later over
HTTP (flm serve, FastFlowLM pull request build with the CI-built libraries and xclbins, this GPTQ package, nothing else
running on the NPU): median of 5 requests after a warm-up request, the range of the five is under 1% except where shown. The runs of the per-length and
per-question-count tables used the plain Q4_K package of identical shape (the GPTQ package differs only in the rounded
weight values); the HTTP rows used this package.
Time to answer one request of one question of n tokens (the state is about 57% of the prompt):
| question + state | median latency (engine, loaded machine) | idle machine, HTTP (range of 5) |
|---|---|---|
| 64 tokens | 5.6 s | -- |
| 128 tokens | -- | 2.85 s (2.83 to 2.88) |
| 330 tokens (HTTP, loaded: 4.9 s) | 5.5 s | 3.22 s (3.18 to 3.24) |
| 1,024 tokens | 11.9 s | 6.86 s (6.78 to 6.88) |
| 2,048 tokens | 23.4 s | -- |
| 4,096 tokens | 42.4 s | 27.4 s (27.3 to 27.6) |
| 8,192 tokens | 84.3 s | -- |
| 16,384 tokens | 186 s | -- |
| 32,768 tokens | 430 s (360 to 411 s in the correctness runs) | 314 s (314 to 315) |
The floor is about 3 s per request on the idle machine (4 to 5 s on the loaded one; the 15.5 GB of weights are streamed from
memory once per pass, so a short question is memory-bound; d3:mini needs 0.5 s for the same question), then roughly 5 to 10 ms
per token on the idle machine (7 to 12 ms on the loaded one). Over HTTP
(flm serve), the 736 text questions of the accuracy set (212 requests) took 4.4 s per request at the median.
Questions per request (state once). One request about the same state, wall time of the engine (state 300 / 1,024 / 4,096 tokens, each question about 110 tokens), against answering every question in its own pass:
| questions | state 300 tokens | state 1,024 tokens | state 4,096 tokens |
|---|---|---|---|
| 1 | 6.3 s | 12.4 s | 39.8 s |
| 4 | 9.3 s (one pass each: 19.2 s, 2.1x) | 15.4 s (57.5 s, 3.7x) | 53.9 s (202 s, 3.7x) |
| 16 | 21 s (53 s, 2.5x) | 25.9 s (243 s, 9.4x) | 72.7 s (635 s, 8.7x) |
| 64 | 74 s (305 s, 4.1x) | 88.2 s (917 s, 10.4x) | 110 s (3,505 s, 32x) |
| 255 | 318 s (1,220 s, 3.8x) | -- | -- |
On the idle machine, over HTTP, a 330-token state (a 4 and a 16 question request are 597 and 1,247 tokens in all): 1 question 3.22 s, 4 questions 6.09 s (6.06 to 6.14), 16 questions 9.52 s (9.52 to 9.58).
State once gives bit-identical answers to the one-pass-per-question path. Peak resident memory with 255 questions about a 300-token state: 18 GB.
| request (one box or track request, GPTQ package, Strix Halo) | median latency, loaded machine | median latency, idle machine (one pass over the request set) |
|---|---|---|
box questions about one image (1 to 8 boxes, 29 requests) |
11 s (3-19 s) | 11 s (3-20 s) |
| a plain question about a video in the state (11 requests) | 18 s (4-26 s); 28 s for a 60 s 720p clip | 18 s (4-27 s) |
assets (images in options or instructions, 15 requests) |
28 s (11-114 s) | 28 s (11-114 s) |
box at one moment of a video (9 requests) |
29 s for a 60 s 720p video | 22 s (7-41 s; clips of up to 11 s) |
track (11 requests: clips of 4 to 11 s, 1 or 2 updates per second) |
-- | 14 s (7-103 s) |
track over a 60 s 720p video, 1 update per second (60 moments, 4 windows) |
145 s | not repeated |
track over a 60 s 720p video, 2 updates per second (120 moments, 8 windows) |
278 s | not repeated |
Requests with images on the idle machine, over HTTP (median of 5 after a warm-up, range in brackets; the first request that carries an image also loads the vision tower):
| request | median latency |
|---|---|
| 1 question + one 1,024 x 768 image (879 tokens) | 7.59 s (7.55 to 7.66) |
| 1 question + two 1,024 x 768 images (1,649 tokens) | 14.61 s (14.58 to 14.63) |
| 1 question + one 2,000 x 1,500 image (scaled to 1.6 megapixels, 1,675 tokens) | 15.30 s (15.23 to 15.33) |
| 8 questions about one 1,024 x 768 image (1,242 tokens) | 10.60 s (10.57 to 10.66) |
| 1 box question about one 1,024 x 768 image (906 tokens) | 7.62 s (7.61 to 7.66) |
| 4 box questions about one 1,024 x 768 image (1,159 tokens) | 10.45 s (10.42 to 10.52) |
Soak: 1,000 mixed requests (text, image and long-prompt requests, several questions per request, repeated requests and malformed bodies) first one after the other, then the same schedule from 4 concurrent clients next to a streaming chat model: 0 status and 0 answer mismatches between the two runs (905 answered, 95 refused with 422 as expected), no crash; resident memory of the server 18.1 GB after the load, a transient peak of 29.4 GB, 21.9 GB afterwards.
From 1,536 tokens attention can run on the NPU's attention kernel; end to end it made no clear difference on this model (within the +-30% noise of the loaded machine), so the threshold was left at the default. The 32,768-token prompt is split into two GEMM parts internally (a single 2.3 GB buffer cannot be allocated on this driver).
Comparison
- Against upstream d3-turbo on one AMD Instinct MI325X (AutoModel, 255 questions about one photo in 2.12 s, one question in 302 ms): the NPU is memory-bandwidth bound and about two orders of magnitude slower in wall time (state 300 tokens + 255 questions: 318 s). Its use is local, private, low-power operation next to the CPU and GPU, not throughput.
- Against the smaller NPU siblings: d3:mini answers a 330-token question in 0.5 s and d3:nano in 0.25 s; d3:turbo takes about 5 s, but reads a state once for any number of questions (the NPU siblings in FastFlowLM read the prompt again for every question). On accuracy, upstream reports (its own estimates) 70.2 text / 72.9 vision on the Jev Decision Index for d3-turbo against 66.3 / 71.3 for the d3 board model.
- Against the BF16 model on the same requests: see Accuracy. The quantization costs a few points of top-1 agreement, more than for d3:mini, and the image path costs somewhat more than text.
License
Apache-2.0, as the upstream model; see LICENSE and NOTICE. Built on Qwen/Qwen3.5-27B (Apache-2.0). Quantized and
repackaged for FastFlowLM by the FastFlowLM team; the readout, decision config, tokenizer and chat template are unmodified.
Citation
Please cite the upstream model (vllm-sr/d3-turbo):
@misc{d3turbo_2026,
title = {{d3-turbo}: A Multimodal Decision Model that Reads Once},
author = {{vLLM Semantic Router Team}},
year = {2026},
howpublished = {\url{https://huggingface.co/vllm-sr/d3-turbo}}
}
- Downloads last month
- 30
Model tree for FastFlowLM/D3-Turbo-NPU2
Base model
vllm-sr/d3-turbo