FastFlowLM build: Nemotron-3.5-Bolt-E4B-DFlash-NPU2

The DFlash speculative-decoding draft of NVIDIA's Nemotron-3.5-Bolt-E4B, converted for the AMD Ryzen AI NPU and run by FastFlowLM (flm). It does nothing on its own: it is the second half of the nemotron3.5-bolt-dflash:e4b tag, whose main weights are the existing nemotron3.5-bolt:e4b package (Nemotron-3.5-Bolt-E4B-NPU2, unchanged). flm pull nemotron3.5-bolt-dflash:e4b downloads both.

flm run nemotron3.5-bolt-dflash:e4b

Needs FastFlowLM 1.0.12 or later (the release that adds DFlash and two-package tags). FLM_DFLASH=0 turns speculation off (the tag then decodes exactly like nemotron3.5-bolt:e4b).

Upstream	nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash (BF16 GGUF, the source of the weights here)
Requires	nemotron3.5-bolt:e4b (main model, tokenizer, chat template, vision and audio towers)
License	OpenMDW-1.1 (LICENSE.OpenMDW-1.1, kept with this package as the license requires); the Gemma 4 base is Apache-2.0
Draft	5 Qwen3-style layers, hidden 2560, FFN 5120, 32 query / 8 key-value heads of 128, attention window 1024, block of 8 tokens, 0.36 B parameters; shares the target's embedding table and output head
Weights	dflash_weight.q4nx: the 5 layers' matrices in Q4_1 (5 bits per weight with scales and minimums); the fc feature projection and the norms in BF16. About 0.27 GB
Taps	features of target layers 5, 11, 17, 23 and 35 (zero-based layer outputs), fused by the draft's fc layer

How it is used

Each step the draft proposes up to 7 tokens in one pass; the main model checks all of them together and keeps the longest prefix it agrees with, plus one token of its own. With a sampler (temperature above 0) every position is sampled and compared with the draft, so the output distribution is the main model's.

Measured draft quality (acceptance length, tokens emitted per step)

Offline replay of this draft's reference implementation on real target features (BF16 target, thinking on, 16 MT-Bench and SPEED-Bench prompts, 400 generated tokens each, greedy):

Draft weights	Mean acceptance length
BF16 (NVIDIA's file)	3.83
all matrices Q4_1	3.78 (-1.3%)
this package (Q4_1, fc in BF16)	3.83

NVIDIA reports 3.6 to 3.9 on MT-Bench and SPEED-Bench; llama.cpp with this draft gives 3.71 on our 40-prompt set, FastFlowLM 3.71.

FastFlowLM measurements

Ryzen AI MAX+ PRO 395 (128 GB), Linux, default power mode, flm serve, 512 generated tokens, 3 prompts per context, decode tokens/s. nemotron3.5-bolt:e4b is the same main model without the draft.

Prompt context	nemotron3.5-bolt:e4b	DFlash greedy, think off	greedy, think on	sampled (default), think off	sampled, think on
~30	13.3	1.74x	1.89x	1.59x	1.71x
1k	12.7	1.54x	1.78x	1.51x	1.76x
4k	11.8	1.33x	1.55x	1.27x	1.38x
16k	9.1	1.43x	1.64x	1.40x	1.61x

The speedup depends on how predictable the text is (code and structured answers gain the most). The NPU checks the 8 positions of a step in one pass; that pass computes in a different order than single-token decoding, so greedy output can differ from nemotron3.5-bolt:e4b at near-ties of the two best tokens (97% of verified positions agree exactly with token-by-token decoding, 99.5% outside near-ties).