HelloSun/Nemotron74ba9

Nemotron 3.5 Bolt 74B-A9B (24 GB IQ2 GGUF) — CPU-optimized build

One model, one quant, one machine: NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf (unmodified file from nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF) plus mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf (unmodified vision projector) and a llama.cpp patch (base commit de5122d) tuned for an Intel Xeon Platinum 8559C. The GGUF files are not changed; all changes are code. Both files are redistributed under the NVIDIA OpenMDW-1.1 license.

What is in this repo

file	purpose
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf	model (unchanged, 20.5 GiB)
mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf	vision encoder projector (unchanged)
patches/nemotron-nvfp4-cpu-opt.patch, patches/BASE_COMMIT	llama.cpp patch and base commit
patches/optional-iq2xxs-vnni.patch	experimental IQ2_XXS kernel, no end-to-end gain (apply after the main patch)
build.sh	clone, patch, and build llama.cpp
benchmarks/	raw A/B throughput, KL-divergence log, multimodal check

What the patch changes

The Nemotron GGUF is a Mamba-2 / Transformer / LatentMoE hybrid. Its size comes from three weight types: IQ2_XXS (routed and shared experts, 16.3 GiB), NVFP4 (all Mamba-2 ssm_in / ssm_out projections, 2.3 GiB), and Q8_0 (attention, latent projections, output). Profiling the stock build on this CPU showed the NVFP4 projections as 39% of decode op time and 67% of prefill op time. Stock llama.cpp has only an AVX2 vec_dot for NVFP4 and no batched kernel, so this patch adds:

1. AVX512-VNNI vec_dot for NVFP4 x Q8_0 (ggml/src/ggml-cpu/arch/x86/quants.c). One 64-weight block per iteration. The E2M1 magnitude comes from a 16-byte shuffle, the sign from nibble bit 3 (applied to the q8 side with a mask so dpbusd takes unsigned magnitudes). UE4M3 scales are decoded in SIMD with a bit move plus a multiply by 2^119 (0x7F maps to 0, as in the reference). The 16 int32 lanes are scaled by one per-lane float vector, so no horizontal reduction is needed inside the loop.
2. 2 x 2 kernel (nrc == 2) for prefill: two weight rows and two activation columns per call, so each weight block is decoded once and used for both columns. Enabled with .nrows = 2 for NVFP4 in the CPU type traits. Non-AVX512 builds fall back to four nrc == 1 calls, so the patch is safe on other CPUs.

Unit tests (random data, double-precision reference): relative error about 5e-7 for both kernels, and for the 2 x 2 path.

Results (16 cores: taskset -c 0-15, -t 16 -fa on -lm mlock -b 1024 -ub 1024, llama-bench)

Interleaved runs on a shared host (see benchmarks/ab_results.csv; 12-thread repeat in ab_12threads.csv: pp256 25.2 -> 36.3 t/s, tg64 9.51 -> 9.53 t/s):

build	pp256 t/s (median, n=5)	tg64 t/s (mean, n=3)
stock llama.cpp de5122d	30.9	10.2
this patch	45.1	9.1 (5.8 / 9.9 / 11.6)
speedup	~1.46x	no reliable decode gain measured

• Prefill: consistent across all five rounds (stock 30.6–33.2, patch 43.0–45.4 t/s).
• Decode: the noise across rounds (5.8 to 11.7 t/s) is larger than any effect, so this patch does not claim a decode speedup. The single-thread NVFP4 gemv is about 25% faster in a microbenchmark (4.3 vs 5.6 ns per 64-weight block), but the end-to-end decode gain did not show up on this host because decode is memory-bandwidth bound (next section).

Why decode does not speed up, and what does help (measured on this machine)

• Decode is memory-bandwidth bound. The NVFP4 ssm_in matvec moves about 43 MB in 0.53 ms (~80 GB/s) against a measured ceiling of 120-150 GB/s, so a faster kernel cannot raise tok/s much. An AVX512-VNNI IQ2_XXS kernel (gather grid + sign bytes rebuilt with pdep and parity, patches/optional-iq2xxs-vnni.patch) is correct (rel. error 5e-7) and ~10% faster on one thread (4.4 vs 4.9 ns per 64 weights), but made no end-to-end difference (tg64 median 9.53 vs 9.51 t/s, 12 threads), so it is not in the main patch.
• Thread count under the CPU quota (benchmarks/thread_sweep.csv, tg64): 16 threads got CFS-throttled and ranged 7.0-10.9 t/s; 12 threads were never throttled and gave 9.6-10.3 t/s. Use -t 12 for steadier decode; prefill is faster at 16 threads (45 t/s vs 36 t/s at 12) when the quota allows.
• Speculative decoding (benchmarks/spec_decoding.jsonl, greedy, 200 tokens, -t 12; baseline 10.9 t/s):

mode	code	prose	repeated text
none	10.9	10.7-11.0	10.9
--spec-type draft-dflash (LiLiCorr drafter, n-max 6)	11.6 (78% accepted)	5.5 (28% accepted)	13.4
--spec-type ngram-mod (no draft model)	10.7 cold / 13.5 warm	10.3 / 9.0	10.1 cold / 22.2 warm

DFlash is not recommended on CPU: verifying 7 tokens through the MoE reads many more experts, so low-acceptance text gets much slower. ngram-mod costs little on novel text and helps when output repeats earlier text (editing files, echoing tool output). "Warm" means the same text had already been generated once in the session, so it is a best case.

• Runtime knobs: nothing to gain (benchmarks/knobs_decode.csv, tg64, 12 threads, 3 interleaved rounds, medians): default --poll 50 10.37, --poll 0 10.16, --poll 100 10.39, --cpu-strict 1 9.96, -lm none 10.20 t/s. All differences are inside the run-to-run noise (single outliers of ~6 t/s appeared once for --cpu-strict 1 and -lm none). --prio 2 did not run in this container. Transparent hugepages are madvise system-wide and were not changed.

• -ub for prefill (benchmarks/ubatch_prefill.csv, pp512, 12 threads, 2 rounds): 128 -> 33-35 t/s, 256 -> 37, 512 -> 40, 1024 -> 39-41 t/s. The default 512 is already the best choice.

• Smaller output.weight (benchmarks/lm_head_quant.txt): the Q8_0 lm_head (570 MB) is ~4.4% of decode op time. A standalone 12-thread gemv takes 6.4-6.8 ms (Q8_0), 4.1-4.9 ms (Q6_K), 2.4 ms (Q4_K). That caps the decode gain at roughly 1-3%, below the measurement noise, and costs logit accuracy, so no requantized GGUF was made.

Suggested server flags on a quota-limited shared host: -t 12 -fa on -lm mlock --spec-type ngram-mod.

Accuracy

Against stock logits, wikitext-2 test split, 16 chunks x 512 tokens (benchmarks/kld.log):

comparison	mean KLD	same top-1
this patch vs stock (-fa on)	0.00270 ± 0.00009	97.75%
stock -fa off vs stock -fa on (noise floor)	0.00257 ± 0.00010	97.40%

The patch's deviation is the same size as the stock flash-attention on/off difference. Stock PPL on this text is 6.604 ± 0.264. Results are not bit-exact (float accumulation order differs). Short factual prompts were answered correctly (see the multimodal check); an explicit greedy-output comparison against stock was not run.

Multimodal check

llama-server --mmproj ... with the patched build correctly names all six icons in a test image, and text answers are correct (17*23 -> 391, capital of France -> Paris). See benchmarks/multimodal_check.md. The image check is coarse: colors are only answered as "teal".

Build and run

./build.sh
taskset -c 0-15 llama.cpp/build/bin/llama-server -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf -t 16 -fa on -lm mlock --jinja -c 8192 --port 8080

Needs about 22 GB of lockable memory for the model plus the projector.

Limitations / next steps

• Validated only on Xeon 8559C with 16 cores.
• IQ2_XXS expert matmul stays on the upstream AVX2 kernel in the main patch (see the optional patch above).
• Decode gain is not established (see above). Prefill is the measured win.