Nanbeige4.2-3B-NVFP4A16 — ⚠ summarization-scoped

Uniform NVFP4 weight-only (W4A16, group-16) quantization of Nanbeige/Nanbeige4.2-3B. 3.6 GB, the fastest artifact in the family (175 tok/s freeform / 220 summarize on RTX 5090) — but read the gate table: use it for summarization/extraction-class work, NOT for reasoning. The failures are published on purpose; they are the first FP4 datapoints for a looped/weight-shared LLM anywhere (prior art, LoopQ arXiv:2605.16343, is INT-only).

Quality gate table (RTX 5090, vLLM v0.25.1; champion = my FP8-Dynamic quant)

bench	mode	bf16 original	FP8-Dynamic	this repo
GSM8K strict (n=100)	thinking	not measured	89	81 ❌
GSM8K flexible	thinking	not measured	93	89
Blind-judge summarization (closed, same-judge pair)	non-thinking	n/c (separate judging pass)	3.93	4.00 ✅ (tied)
Judged faithfulness (closed)	non-thinking	n/c	4.95	5.0
Judged fabrication / leaks (closed)	non-thinking	0% / 0%	2% / 0%	0% / 0%
Dictation-rewrite taxonomy (closed)	non-thinking	18/20	18/20	16/20 ❌
JSON parse rate (closed, /48)	non-thinking	44	42	39

Speed (RTX 5090, batch-1, vLLM v0.25.1, 64k ctx, fp8 KV)

Decode tok/s, single stream, this artifact's headline. Both rows measured WITH ngram speculation (its intended copy-heavy workloads):

workload	spec decode	bf16	FP8-Dynamic	this repo
freeform	ngram, 8 tok	85	128	175
summarize / RAG (2k+ ctx prompts)	ngram, 8 tok	136	206	220

Two workload anchors, not an ISL sweep. The Marlin W4A16 kernel plus the looped arch's double weight-read per token is where the speedup comes from: weight compression pays out twice per token on this model.

Why reasoning breaks: this architecture executes each layer twice per token (num_loops: 2) — 4-bit weight error compounds across both passes. Shallow extractive work (summarization stays judge-tied with FP8 at perfect faithfulness) survives; long reasoning chains don't. A contributing mechanism, verified by super-weight scan (Apple 2411.07191): this model's super weight (layers.1.mlp.down_proj.weight[1252, 6883], the single most load-bearing scalar) sits in a 4-bit region under uniform quantization; the LoopShield recipe holds it in FP8. A mixed-precision recipe recovers full reasoning parity at 4.5 GB — see the companion repo Nanbeige4.2-3B-NVFP4-FP8-LoopShield.

Serving (vLLM)

pip install --no-deps ./vllm_plugin   # arch not upstream yet (vLLM PR #49433)
vllm serve <this-repo> --trust-remote-code \
  --max-model-len 65536 --kv-cache-dtype fp8 \
  --speculative-config '{"method":"ngram","num_speculative_tokens":8,"prompt_lookup_max":4,"prompt_lookup_min":2}'

Serves via the Marlin W4A16 kernel (fastest batch-1 path on consumer Blackwell — do not convert to W4A4: slower at batch 1 AND activation-quantization is this arch's worst failure mode). Non-thinking mode recommended for its intended workloads: chat_template_kwargs: {"enable_thinking": false}.

Download just this artifact:

hf download NullSense/Nanbeige4.2-3B-NVFP4A16 --local-dir Nanbeige4.2-3B-NVFP4A16

Creation

llmcompressor QuantizationModifier(targets="Linear", scheme="NVFP4A16", ignore=["lm_head"]), data-free. GPTQ was attempted and is not usable on this architecture (llmcompressor 0.12: Hessian inversion fails on every module — root cause unknown; the double-fire hook was checked and is not it → "Failed to invert hessian" on 154/154 modules, silent RTN fallback — reported upstream); AWQ lacks arch mappings. So RTN is currently the only plain uniform-4-bit route for looped models, which is precisely why the reasoning gate fails and why I scope this artifact.

Limitations

Not for reasoning/agentic use — see gate table.
vLLM-only until PR #49433 merges; bundled modeling file carries two one-line transformers-5 compat patches.
English-only evals, English-only calibration-free quant on a bilingual model.

Which artifact should I choose?

artifact	size	tok/s (freeform / summ)	pick when
FP8-Dynamic	4.9 GB	154 / 206	default: no measured quality loss on any gate. Recommended.
NVFP4-FP8-LoopShield	4.5 GB	159 / 202	smallest artifact that keeps reasoning at FP8 parity; best JSON reliability. Recommended for tight VRAM.
NVFP4A16 (this repo)	3.6 GB	175 / 220	fastest; summarization/extraction only (reasoning drops 8 GSM8K points). Not for math/agentic.
EAGLE3 draft	+1.5 GB	+12-41% decode	add-on speculator for any of the above; thinking-aware retrain (2026-07-24), thinking-mode acceptance 0.41, lossless. Serve with TRITON_ATTN.

All three serve identically (same plugin, same flags); only the checkpoint differs.

Comparison chain: the columns here use my FP8-Dynamic quant as reference; the FP8 card carries the bf16-original matrix linking the chain back to the unquantized model.

Links & provenance

Base model: Nanbeige/Nanbeige4.2-3B
Family: FP8-Dynamic · NVFP4-FP8-LoopShield · NVFP4A16 · EAGLE3 draft
Arch serving support: vLLM PR #49433 (until merged, the bundled vllm_plugin/ registers it out-of-tree)
Upstream tooling reports I filed from this work: llm-compressor#2952 (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · llm-compressor#2953 (AWQ lacks arch mappings)
Quantized with llm-compressor 0.12.0 (compressed-tensors format)

Benchmark provenance (ordered: public-harness first, then my closed harnesses)

Public, reproducible (lm-eval-harness local-chat-completions, exact configs in each row's annotation):

GSM8K — grade-school math, the reasoning gate (thinking mode, n=100, max_tokens 8192)
IFEval — verifiable instruction following (non-thinking, n=250)
MMLU-Pro — 10-choice knowledge/reasoning (non-thinking, 25/category)
BBH — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask)
MultiHop-RAG — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public)

Closed/personal harnesses (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores):

Blind-judge summarization — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality?
Rewrite taxonomy — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing.
JSON parse rate — structured-output emission over the summarization set. Tests: format discipline under quantization.

Citation

@misc{peciukonis2026nanbeige42nvfp4a16,
  author       = {Pe{\v{c}}iukonis, Matas (NullSense)},
  title        = {Nanbeige4.2-3B-NVFP4A16: uniform 4-bit quantization with published failure modes},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16},
  note         = {Speed/summarization-scoped artifact; reasoning regression (GSM8K 89->81) documented on-card as a negative result for looped-arch uniform 4-bit.}
}