Qwen3.8-27B-NVFP4

Post-training NVFP4 (W4A4) quantization of Qwen/Qwen3.8-27B, produced on a RTX 5090 (32 GB, sm_120) + Ryzen 9 9950X3D + 64 GB DDR5 system with NVIDIA ModelOpt 0.46.0. The checkpoint ships the full multimodal architecture (language + vision tower) plus the Multi-Token-Prediction (MTP) draft layer, and is verified serving 160k context with an FP8 KV cache on vLLM 0.28.0.

Weight footprint drops from 55.56 GB (bf16) to 20.59 GB (NVFP4: 19.7 GB main + 0.85 GB MTP shard) — a 62.9 % reduction — while the precision-critical module families (embeddings, lm_head, GDN conv/in_proj, norms, vision tower, MTP layer) are deliberately kept at full precision. Every number on this card is traceable to a file in this repository (config.json, quantization_config.json, hf_quant_config.json, model.safetensors.index.json, provenance.json) or to a command in the Reproducibility appendix.

A note on how to read this card. The base model publishes no perplexity or benchmark table, and its 55.6 GB bf16 weights cannot be co-resident with this checkpoint on a 32 GB GPU. A base-vs-quant delta is therefore not reported here; the quality section gives absolute numbers instead. That limitation is stated rather than papered over.

---

1. Identity & scope

Property	Value
Base model	Qwen/Qwen3.8-27B (Apache-2.0)
Architecture	Qwen3_5ForConditionalGeneration (model_type qwen3_5), image-text-to-text
Parameters	27,781,427,952 (~27.78 B), identical to the base model — quantization changes the storage format, not the parameter count; on-disk footprint 20.59 GB (W4A4)
Quantization	NVFP4 — W4A4: FP4 E2M1 weights and FP4 E2M1 activations, FP8 E4M3 block scales per 16 values, FP32 global scales
Layers	64 — 48 Gated-DeltaNet linear-attention + 16 full-attention
Native context	262,144 tokens (verified serving at 163,840)
MTP	1 draft layer, 15 tensors, kept bf16, shipped in a dedicated shard
License	Apache-2.0 (base and derived)
Quantized by	Bernhard Brieger
Quantization date	2026-08-30

Sidebar note — the HF "Model size" box is a display artifact, not the model size. The auto-generated Safetensors widget on the HF page counts raw stored elements and does not unpack packed FP4 (each U8 byte holds two 4-bit values) — so it reports ~15 B, not the 27.78 B logical parameter count above. The model is 27.78 B either way; the box is an HF UI quirk for packed FP4 storage, not a model property.

Intended use. General instruction following, coding, tool/function calling, vision, and long-context workloads (retrieval over documents >100k tokens) on hardware that fits ~20 GB of weights, with a runtime that supports NVFP4 (W4A4) execution — vLLM's modelopt_fp4 backend (Cutlass GEMMs) is the verified path. NVFP4 is a Blackwell-generation format: the W4A4 path requires native FP4 tensor cores, which only Blackwell (sm_100/sm_120) has — it cannot run on pre-Blackwell GPUs.

Out of scope. This is a plain quantization of the base model — no fine-tuning, instruction distillation, or safety retraining was applied. It is not intended as a substitute for the base model where full precision is required, and the numbers below characterize this specific checkpoint, not the base model in general.

What is quantized vs. kept at bf16. The W4A4 compression covers all 400 linear mlp and attention-projection modules across the 64 layers (400 input_scale tensors were captured during calibration). The remaining module families stay at full precision — token-embedding and lm_head matrices (1.27 B each), the MTP draft layer (425 M), the vision tower (461 M), and the per-layer layernorms / GDN conv1d / in_proj_a / in_proj_b / output-gate parameters.

---

2. Quantization recipe

Self-contained recipe — the exclusion list and calibration settings are stored in quantization_config.json and hf_quant_config.json in this repo.

Toolchain

Component	Version
NVIDIA ModelOpt	0.46.0
transformers	5.14.1
PyTorch	2.13.0+cu130
Python	3.12
GPU	1× NVIDIA RTX 5090, 32 GB (sm_120)
CPU	AMD Ryzen 9 9950X3D, 16C/32T
RAM	64 GB DDR5

NVFP4 configuration

Parameter	Value	Note
preset	nvfp4	W4A4 end-to-end (weights and activations)
algorithm	max	per-tensor global scales derived from the weight tensors (amax)
group_size	16	FP8 E4M3 block scale per 16 values
Activation scaling	input_scale (amax)	captured on the calibration corpus
Excluded modules	147 entries (+ mtp.* in the serving config)	see table below

Modules kept at bf16 (exclusion families)

NVFP4 exclusions are an explicit module list (not regex patterns): a listed module is left at full precision. quantization_config.json carries 147 entries; the serving-time hf_quant_config.json extends the same set with mtp.* so the draft layer stays exact.

Family	Count	Why
lm_head, embed_tokens	2	output/input embeddings — load-bearing
*.linear_attn.conv1d	48	GDN causal convolution — not quantized (the W4A4 path does not cover causal conv)
*.linear_attn.in_proj_a / in_proj_b	96	GDN projection pairs — load-bearing
*.norm (incl. GDN / vision-merger norms)	—	not quantized; normalization parameters are kept in their original precision
model.visual*	1	entire vision tower stays bf16
mtp.*	15 tensors	keeps the speculative-decoding draft layer exact

Why the MTP exclusion matters for serving. vLLM gates MTP on any exclusion key whose text contains mtp. This repo lists mtp.* in the serving-time exclusion set — hf_quant_config.json (exclude_modules, 148 entries), mirrored in config.json (quantization_config.ignore) — which is what keeps the 15 bf16 MTP tensors loadable and lets --speculative-config work (see §4). A missing mtp.* entry is what originally broke MTP loading on this checkpoint.

Calibration corpus

Metric	Value
Sequences	2,026
Total tokens	1,802,645
Longest sequence	132,968 tokens
Random seed	42
Chunk size	8,192 tokens (32 activation-scaling blocks)
Purpose	activation input_scale (amax) for the W4A4 activations; weight scales come from the weight tensors themselves

Built from a curated pool of public, non-gated sources (see License & provenance): 11 programming languages, an agent/tool-calling pool, general web/wiki prose, a multilingual (DE/FR/IT) pool, a math pool, and a 2×128k long-document tail, so the deepest context regimes are actually calibrated, not just short sequences. The corpus manifest is referenced in provenance.json; the raw sequences are not redistributed with this checkpoint.

Run statistics

Stage	Wall time
Model load (bf16)	13 s
Calibrate + compress	3.5 min (155 s calibrate / 52 s compress)
Export (2 shards + MTP)	13 s + injection

Total wall time ~4 min (240 s in the build log) on a single RTX 5090 — the NVFP4 scheme only needs per-tensor activation scales (amax) collected in one calibration forward pass, with no per-module iterative error compensation.

---

3. Quality verification

Three independent checks: a quantizer-statistics note, an absolute perplexity, and a functional smoke suite (text and vision) run against the live server.

(a) Quantizer statistics

ModelOpt does not emit a per-module loss table; the build log's per-block records serve as the audit instead. The log records input_scale: 400 tensors — every quantized linear module received a per-tensor activation scale from the calibration corpus — and the verify step confirms exclude mods: 147 and weights: 19.7 GB. No module fell back to a degraded strategy: W4A4 ModelOpt has no per-module fallback concept — either the scale is captured or the module is excluded. The 147-entry exclusion list in quantization_config.json (which includes model.visual*) is the complete accounting; mtp.* is added only in the serving-time hf_quant_config.json.

(b) Perplexity (absolute)

Metric	Value
Perplexity	2.799
Tokens scored	n = 260
Method	max_tokens=1 + prompt_logprobs=1 over a fixed, neutral English prose sample; PPL = exp(−mean logprob) over the input tokens
Runtime	live vLLM (this checkpoint), scripts/measure_ppl_quant.py

This is an absolute perplexity on a single 260-token sample — a sanity check that the dequantized model assigns sensible probabilities to plain prose, not a quality benchmark. The base model publishes no perplexity, so there is nothing to diff against; the number is reported as-is.

(c) Functional smoke (3/3 pass, live server)

Probe	Result	Detail
Coding generation	PASS	2,490 chars in 18 s (enable_thinking: false)
Long-context needle	PASS	target located at 46,037-token context, returned ZEBRAX42 (7 s)
Tool / function calling	PASS	exact JSON: get_weather(city="Graz", unit="celsius") (1 s)

The needle probe exercises the long-context regime that the 2×128k calibration tail was built to protect. The tool-call probe verifies the qwen3_coder parser path end-to-end.

(d) Vision probe (PASS, live server)

Probe	Result	Detail
Solid-color image	PASS	128×128 solid red PNG via the chat-API image input → model answered "Red" (1.1 s)

The probe is deterministic (scripts/vision_smoke.py): a pure-red image with a strict "reply with the color word only" instruction, so any answer other than "Red" is a fail. This confirms the vision tower (kept at bf16, §1) and the image-processor path (Qwen3-VL image processor + processor_config.json shipped in this repo) work on this checkpoint.

What is not verified here

• No base-vs-quant benchmark table. The base card publishes no benchmarks or perplexity, and the 55.6 GB base cannot be co-resident with this checkpoint on the 32 GB test GPU. A head-to-head would need ≥80 GB of VRAM.
• No per-module loss table — ModelOpt does not emit one (see §3a); quality evidence is the absolute PPL plus the 4 live probes.
• MTP speculative decoding is preserved (15/15 tensors, bf16) but the performance numbers in §4 were taken without the speculative flag; see §4 for the opt-in flag and the caveat.
• Single-sample PPL is a sanity check, not a benchmark (§3b).

---

4. Deployment

Verified serving configuration on vLLM 0.28.0 (Cutlass NVFP4 GEMMs, FP8 KV cache, FlashInfer attention, CUDA graphs active), 163,840-token context on an RTX 5090 32 GB:

vllm serve bernhardbrieger/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --quantization modelopt_fp4 \
  --linear-backend cutlass \
  --max-model-len 163840 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --attention-backend FLASHINFER \
  --max-cudagraph-capture-size 64 \
  --trust-remote-code

• --linear-backend cutlass is required, not cosmetic. It routes the W4A4 GEMMs to precompiled Cutlass kernels; without it the default path triggers a FlashInfer JIT that OOMs the host on 32 GB systems and hangs the engine.
• Do not pass --enforce-eager. Disabling CUDA graphs amortizes nothing here and measurably lowers W4A4 throughput (eager runs on this checkpoint landed well below the ~48–56 tok/s decode of the graphed config). The verified config runs with CUDA graphs on.
• --max-cudagraph-capture-size 64 bounds graph capture to the bench/agent-size batch range — the mitigation currently recommended for sm_120 + hybrid GDN models on vLLM 0.28.x (vllm-project/vllm#52540, #54331).
• Weights: 19.7 GB (2 shards) resident. At 160k context with an FP8 KV cache the server holds ~31.4 GB of the 32 GB GPU — it fits, with max_num_seqs 2.
• Reasoning / tool parsing: --reasoning-parser qwen3 + --tool-call-parser qwen3_coder are what make structured tool-call output work (see the smoke probes in §3c).

Optional: MTP speculative decoding

The MTP draft layer is present (15 tensors, bf16, in model-mtp.safetensors, excluded from quantization). To use it:

  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

The MTP weights remain fully usable without the flag (they are simply unused), so the checkpoint runs as a normal model on any setup — speculative decoding is opt-in. Caveat: the MTP-speculation + CUDA-graphs + concurrent-batch corner was not part of this verification; run a low-concurrency smoke before relying on it in production.

Serving (verified path)

The W4A4 runtime path is vLLM (§4 flags). A plain from_pretrained of the packed FP4 weights in transformers is not a verified path — the checkpoint is distributed for the vLLM modelopt_fp4 backend, and that is the only serving configuration verified for this repo.

Performance (measured)

One-shot measurement, vLLM 0.28.0, 2026-08-30, batch 1, on the exact §4 flags above (RTX 5090 32 GB). Per context length: 1 warmup request + 3 timed runs, median reported, max_tokens=256, temperature 0, thinking mode off. Prompts were drawn from the §2 calibration corpus such that no prompt is a prefix of another — with --enable-prefix-caching active the server-side hit rate over the whole run was 0.0 %, i.e. every number below is full prefill. Prefill throughput = prompt tokens / TTFT; decode throughput = completion tokens / (total time − TTFT). The KV-cache column is derived from the model config, not measured: FP8 KV holds 32 KiB per token (16 full-attention layers × K+V × 4 KV heads × 256 head-dim × 1 byte), so each entry equals its context length × 32 KiB.

Context	TTFT (s)	Prefill (tok/s)	Decode (tok/s)	KV cache (GiB)
8,192	0.607	13,503	55.9	0.25
16,384	1.350	12,149	55.5	0.50
32,768	3.364	9,746	54.9	1.00
65,536	9.297	7,045	53.0	2.00
131,072	29.762	4,404	49.5	4.00
163,560	43.006	3,801	47.8	4.99

The W4A4 path's signature shape is in the first column pair: prefill throughput stays in the 3.8–13.5k tok/s range across all six tiers (bandwidth-bound W4A4 GEMMs), while decode settles around 48–56 tok/s — i.e. the checkpoint is prefill-heavy by design. If your workload re-processes long contexts on every call (agent loops, RAG refreshes), that is exactly where it pays; if your workload is mostly long generations from short prompts, plain 4-bit decode-bound formats will usually feel snappier per output token.

Single-shot snapshot: absolute numbers will differ on other vLLM releases (kernel and scheduling changes). Reproduce with scripts/bench_serve.py against a server launched with the §4 flags.

---

5. Hardware requirements

Deployment	Minimum VRAM
Weights only (short context)	~20 GB
160k context, FP8 KV, 2 seqs (verified)	32 GB (RTX 5090)

The verified configuration uses the full 32 GB. The 19.7 GB main-shard footprint (+0.85 GB MTP shard) is a fixed cost: a GPU below ~20 GB cannot load this checkpoint at any context length. The remaining budget goes to the KV cache, which scales with context length and concurrency — it determines how much context a given card can serve. The verified path is the vLLM modelopt_fp4 backend (Cutlass GEMMs) on Blackwell hardware (sm_100/sm_120) — the only GPUs with native FP4 tensor cores, and a hard requirement for the W4A4 path. NVFP4 is a Blackwell-generation format: the W4A4 path (FP4 compute on both weights and activations) cannot run on pre-Blackwell GPUs, which have no FP4 tensor cores — running this checkpoint there is out of scope for this release.

---

6. Reproducibility appendix

Every number in §1–§3 is produced by one of the following. Paths are relative to this repository; the raw calibration corpus is documented in provenance.json but not redistributed with this checkpoint.

6.1 Quantization

The checkpoint is produced by ModelOpt 0.46.0 over the 2,026-sequence corpus (documented in provenance.json: seed 42, 1,802,645 tokens; the raw sequences are not redistributed); the build uses its first 32 8,192-token blocks for the amax calibration:

# scripts/build_nvfp4.py (abridged)
from modelopt.recipe.presets import QUANT_CFG_CHOICES
from modelopt.torch.quantization import quantize, compress
from modelopt.torch.export.unified_export_hf import export_hf_checkpoint

model = Qwen3_5ForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3.8-27B", torch_dtype=torch.bfloat16,
    device_map="auto",
    max_memory={0: "28 GiB", "cpu": "32 GiB"},  # caps leave VRAM for the calib forward
)
model = quantize(model, QUANT_CFG_CHOICES["nvfp4"], forward_loop)  # 32-block calibration
compress(model)
export_hf_checkpoint(model, "output/Qwen3.8-27B-NVFP4", max_shard_size="10GB")

The nvfp4 preset is W4A4 with the max algorithm: weight scales come from the weight tensors, activation input_scale (amax) from the calibration forward. Before the export, the script consolidates every tensor to a single device (modelopt's export-time dummy forward is CUDA-only) and clears the stale hf_device_map so the save takes the whole-state-dict path.

ModelOpt's export writes weights + config only. Two post-steps complete the checkpoint (both in scripts/inject_mtp_and_processors.py, both idempotent):

1. MTP injection — transformers' Qwen3_5 discards ^mtp.* at load, so the 15 bf16 draft-layer tensors are copied from the base shard into model-mtp.safetensors (+ index entry). Without this step MTP loading breaks.
2. Processor injection — tokenizer/processor files (tokenizer.json, preprocessor_config.json, processor_config.json, video_preprocessor_config.json, …) are copied from the base snapshot; the export writes none of them. Without preprocessor_config.json the vLLM engine fails to boot; without processor_config.json, AutoProcessor.from_pretrained on the repo fails.

quantization_config.json (exclusions, calibration metadata) and hf_quant_config.json (serving-time quant config, mtp.* in both exclusion lists) are emitted by the builder and verified in step 5 of build_nvfp4.py.

6.2 Perplexity

Against a running server on :8000, scoring the fixed 260-token sample with input-token logprobs:

PPL_MODEL=bernhardbrieger/Qwen3.8-27B-NVFP4 python scripts/measure_ppl_quant.py
# -> quant_ppl  n=260  perplexity=2.799

6.3 Functional smoke (text + vision)

python scripts/smoke_test_api.py
# -> 3 probes: coding / needle_40k / tool_call

python scripts/vision_smoke.py
# -> deterministic solid-red probe, expect reply "Red"

Both scripts are checkpoint-agnostic (target via the SMOKE_MODEL / PPL_MODEL env vars), so the same suite can be run against any sibling checkpoint of this model.

---

7. License & provenance

Model weights. Apache-2.0. The base Qwen/Qwen3.8-27B is Apache-2.0 and this derived checkpoint is released under the same license.

Calibration data. The corpus is drawn entirely from public, non-gated datasets: seed 42, the pools listed in §2, and the 2×128k long-document tail. Every source, its license, and (for code) its pinned commit are listed in provenance.json in this repository. The raw calibration sequences are not redistributed with this checkpoint.

Source	License	Used in
28 code repositories	Apache-2.0 (13) · MIT (13) · BSD-2-Clause (1) · BSD-3-Clause (1)	code pools (11 languages)
codeparrot/codeparrot-clean	per-file upstream license (no dataset-level license declared)	Python pool
wikimedia/wikipedia (EN/DE/FR/IT)	CC BY-SA 3.0 + GFDL	wiki prose, long-document tail
openai/gsm8k	MIT	math pool
allenai/tulu-3-sft-mixture	ODC-BY	agent / tool-calling pool
HuggingFaceTB/SmolLM-Corpus (FineWeb-Edu dedup)	ODC-BY	web prose, long-document tail

Each code repository is pinned to a specific commit in provenance.json (repo + commit SHA + license), so the code portion of the corpus is reconstructable byte-for-byte.

Data attribution. Wikipedia text © Wikipedia contributors, released under CC BY-SA 3.0 and the GFDL. FineWeb-Edu and the Tülu-3 SFT mixture are used under ODC-BY. Attribution is provided here in lieu of per-file notices.

Base model references.

• Base model card: https://huggingface.co/Qwen/Qwen3.8-27B
• Qwen3.5 / Qwen3.8 technical report: see the links on the base model card.