D3-Edge-NPU2

vllm-sr/d3-edge (Decision 3.0, the decision model of vLLM Semantic Router; Qwen3.5-0.8B architecture cut to 12 layers, revision c454ee39) for AMD Ryzen AI NPUs, served by FastFlowLM 1.0.12 or later. It answers System One requests -- choice, yes / no and score questions about a state, optionally with images, with a probability for every option -- over the same POST /v1/systemone API as the upstream package. Up to 32,768 input tokens per question. Video is not supported (a request with videos is refused with HTTP 422).

flm serve --decision d3:edge          # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:edge
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
  -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
       "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'

Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most 1.6 MP), shared by every question of the request. d3:lite (0.8B), d3:nano (2B), d3:mini (4B) and d3:flash (9B) are the larger siblings.

What was changed

• Weights: the 12-layer text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with GPTQ-calibrated rounding (calibration: 192 synthetic System One prompts, none from the Decision Index), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. vision_weight.q4nx holds the vision tower (Qwen3.5-0.8B's, merger included), used for requests with images.
• Unchanged: readout.safetensors, decision_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja and LICENSE are the upstream files byte for byte (MANIFEST.sha256).
• Same contract: prompts, token ids and answer objects follow the upstream runtime.

Accuracy (against the upstream BF16 model)

	this package on the NPU	upstream BF16
Decision Index public-suite sample, mean benchmark score (1,600 requests, 37 benchmarks)	49.8%	47.2%
Top-1 agreement with BF16 on the same requests (4,452 choice answers)	93.8%	--
Long prompts, 835 to 31,545 tokens (8 requests), same top answer as BF16	6 of 8	--
Decision Index public vision suite subset, accuracy (300 questions, 10 benchmarks)	53.0%	55.7%
Top-1 agreement with BF16 on the same image questions	82.3%	--

Mean benchmark score, this package minus BF16: +2.6 pt (standard error 1.1 pt; about 42 requests per benchmark, so single benchmarks are noisy), i.e. no accuracy loss is measurable. Answers agree with BF16 less often than for the larger sizes (93.8%; plain Q4_K without GPTQ: 91.9%): a small model has many near-tied options, and the two long prompts that differ are such near-ties. The sample is a seeded round-robin over the public suite's requests whose prompts are at most 8,000 tokens (37 of its 40 benchmarks have such requests); RouterBench and SGD requests are left out because their locally rebuilt payloads differ from the lab's. The BF16 scores are the upstream BF16 run's own answers on the same requests. The vision subset is 30 seeded questions from each of 10 benchmarks (the suite without the gated Winoground); paired accuracy, BF16 minus this package: +2.7 pt (95% CI -0.7 to +6.0 pt), so images may cost a little accuracy.

Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.12)

request	median latency
1 question, 128 tokens	60 ms
1 question, 330 tokens	83 ms
1 question, 1,024 tokens	0.26 s
1 question, 4,096 tokens	1.0 s
1 question, 16,384 tokens	6.0 s
1 question, 31,545 tokens	16 s
1 question, 32,767 tokens	17 s
4 questions, 330 tokens each	0.29 s
16 questions, 330 tokens each	1.0 s
1 question + a 1024 x 768 image (880 tokens, HTTP)	0.61 s
1 question + two images (HTTP)	0.89 s
1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP)	1.4 s
8 questions + one image (one image encoding, HTTP)	2.0 s

From 1,536 tokens attention runs on the NPU's attention kernel. Prompts over 32,768 tokens are answered with max_length_exceeded (nothing is truncated). Resident memory of the server: 2.0 GB after loading, up to 4.8 GB at 32,768 tokens with images in the mix. Median of 3 (16,384 tokens and longer: 1 to 2 runs) over HTTP on an otherwise idle machine.

License

Apache-2.0, as the upstream model; see LICENSE and NOTICE. Built on Qwen/Qwen3.5-0.8B.