bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4

FP4 (NVFP4) quantization of bottlecapai/ThinkingCap-Qwen3.6-27B — capability of Qwen3.6-27B with 50% less thinking tokens on average, achieved by finetuning Qwen3.6-27B (Qwen Team, 2026) while preserving the original answer quality and style.

➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.6-27B.

About this quantization

Weights are stored at 4 bits in NVIDIA's NVFP4 format (E2M1 values with per-group-of-16 FP8 scales) while activations stay in bf16 — weight-only NVFP4A16 (W4A16), produced data-free with llm-compressor in the compressed-tensors nvfp4-pack-quantized format that vLLM loads natively. ≈19 GB instead of ≈55 GB bf16 — ~2.9× smaller. Being weight-only (W4A16), it serves through vLLM's MarlinNvFp4LinearKernel on both Hopper (H100/H200) and Blackwell: the 4-bit weights unpack to bf16 for compute, so the decode win is memory bandwidth (≈2× bf16 tok/s), not FP4 tensor cores (native cutlass FP4 compute would require a W4A4 / FP4-activation checkpoint). The model's MTP (multi-token-prediction) head is preserved, so self-speculative decoding works (≈3.2 accepted tokens/step).

Kept in bf16: lm_head, the MTP head, the vision tower, and the two small Gated-DeltaNet input gates linear_attn.in_proj_a / in_proj_b (48-wide output, not a multiple of the 4-bit kernel's tile size and fused to width 96 at serve time, so quantizing them breaks loading on non-Blackwell GPUs; keeping them bf16 costs ~60 MB). The remaining group-quantizable Linear layers — MLP, self-attention, and the other linear-attention projections (in_proj_qkv / in_proj_z / out_proj) — carry the 4-bit weights.

Usage

vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4

Speculative decoding (MTP)

The MTP (multi-token-prediction / NextN) head is kept in bf16, so vLLM can run self-speculative decoding for a decode speed-up — no separate draft model needed. Add --speculative-config when serving (requires vLLM ≥ 0.24.0):

vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Speculative decoding is lossless — the output is identical to standard decoding. It accepts ≈3.2 drafts per verify step here, for a further ≈1.5× on top of the finetune's token savings (see below).

Served on vLLM; SGLang cannot currently load weight-only NVFP4 for this architecture. For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF; for near-lossless FP8 at half the memory (loads in vLLM and SGLang), see bottlecapai/ThinkingCap-Qwen3.6-27B-FP8.

Expected performance

Measured on our internal serving harness on 8× RTX PRO 6000 (Blackwell) with vLLM 0.25.0, one configuration per GPU. Each cell is N=200 questions/dataset × 3 generation seeds over a fixed problem subset, batch size 16, sampled decoding (temperature 1.0, top_p 0.95, top_k 20); acc is the mean ± 95% CI across the seeds. Generation budgets are 24,576 tokens (MMLU-Pro) and 16,384 tokens (RealWorldQA).

FP4 matches the bf16 finetune's accuracy (all 95% CIs overlap) while decoding ≈2.3× faster (52.7 vs 22.6 tok/s, MMLU-Pro standard — the memory-bandwidth win of 4-bit weights via the Marlin kernel), and MTP self-speculative decoding (≈3.1–3.2 accepted tokens per verify step) adds a further ≈1.5× — stacking with the finetune's token savings to ≈4.7–8.4× faster per task than the unquantized base (MMLU-Pro 11.6 s vs 97.3 s; RealWorldQA 7.3 s vs 34.2 s).

All timings are measured per request during the eval. median tokens = median completion length (the finetune's lever); tok/s = per-request steady-state decode rate (prefill excluded via time-to-first-token); task s = measured end-to-end wall-clock per request under the batch-size-16 concurrency; speedup = task s of the base model in standard decoding ÷ task s of the row. unsloth NVFP4 and NVIDIA NVFP4 are community NVFP4 quants of the base model, shown for comparison; ThinkingCap-FP8 is our FP8 sibling (bottlecapai/ThinkingCap-Qwen3.6-27B-FP8).

NVFP4 kernel (important for reading tok/s). On this hardware the vLLM NVFP4 kernel is chosen by the checkpoint's FP4 scheme, not a flag, so the NVFP4 rows are not on identical footing: this model and NVIDIA NVFP4 are weight-only W4A16 → MarlinNvFp4LinearKernel (4-bit weights unpacked for compute; a memory-bandwidth decode win), whereas unsloth NVFP4 is W4A4 (weights and activations FP4, mixed-precision) → FlashInferCutlassNvFp4LinearKernel, the cutlass/flashinfer FP4 GEMM. vLLM has no cutlass path for weight-only W4A16 on sm_120, so those fall back to Marlin — a checkpoint-format property, bit-exact-ish and not a source of accuracy difference. On Hopper (H100/H200) there is no native FP4 compute at all, so every NVFP4 serves via Marlin weight-unpack; the cutlass FP4 tensor-core advantage (W4A4) shows mostly on compute-bound prefill / high batch and on datacenter Blackwell.

MMLU-Pro (reasoning)

config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base bf16 · standard	0.902 ± 0.019	2186	22.6	97.3	1.00×	—
Qwen3.6-27B base bf16 · MTP	0.892 ± 0.031	2117	47.7	45.6	2.13×	3.23
unsloth NVFP4 (base) · standard	0.878 ± 0.038	2186	44.5	49.1	1.98×	—
unsloth NVFP4 (base) · MTP	0.888 ± 0.047	2123	87.4	25.4	3.83×	3.21
NVIDIA NVFP4 (base) · standard	0.892 ± 0.026	2032	51.0	40.0	2.43×	—
NVIDIA NVFP4 (base) · MTP	0.907 ± 0.036	1996	93.7	21.6	4.50×	3.23
ThinkingCap bf16 · standard	0.890 ± 0.022	963	23.4	41.6	2.34×	—
ThinkingCap bf16 · MTP	0.895 ± 0.012	910	51.1	18.8	5.18×	3.26
ThinkingCap-NVFP4 · standard	0.882 ± 0.031	926	52.7	17.7	5.50×	—
ThinkingCap-NVFP4 · MTP	0.885 ± 0.033	878	81.3	11.6	8.39×	3.24

RealWorldQA (vision)

config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base bf16 · standard	0.802 ± 0.019	718	21.7	34.2	1.00×	—
Qwen3.6-27B base bf16 · MTP	0.798 ± 0.064	752	42.2	18.7	1.83×	3.09
unsloth NVFP4 (base) · standard	0.788 ± 0.019	777	43.8	18.7	1.83×	—
unsloth NVFP4 (base) · MTP	0.783 ± 0.040	731	79.2	10.9	3.14×	3.08
NVIDIA NVFP4 (base) · standard	0.785 ± 0.012	732	46.6	17.3	1.98×	—
NVIDIA NVFP4 (base) · MTP	0.790 ± 0.050	700	79.8	10.5	3.26×	3.09
ThinkingCap bf16 · standard	0.818 ± 0.047	344	21.5	17.4	1.97×	—
ThinkingCap bf16 · MTP	0.805 ± 0.045	372	42.5	9.9	3.45×	3.09
ThinkingCap-NVFP4 · standard	0.802 ± 0.007	317	40.9	9.2	3.72×	—
ThinkingCap-NVFP4 · MTP	0.800 ± 0.012	320	55.5	7.3	4.68×	3.09