Avuja/Qwen3.8-27B-int4-AutoRound

Qwen3.8-27B INT4 AutoRound

A W4A16 (INT4 weight, FP16 activation) quantization of https://huggingface.co/Qwen/Qwen3.8-27B, produced with Intel's AutoRound.

Built as a drop-in model slot for the club-3090 dual-RTX-3090 vLLM stack — format-identical to https://huggingface.co/Lorbus/Qwen3.6-27B-int4-AutoRound, whose recipe this replicates for the new 3.8 release.

TL;DR

• Base: Qwen3.8-27B (27B dense VLM, hybrid Gated DeltaNet + full attention, Aug 2026)
• Quant: INT4 W4A16, group_size 128, symmetric
• Tool: auto-round 0.14.2 (default recipe, 200 iters, torch.compile)
• Size: 19.6 GB (down from ~54 GB BF16) — **2.7x reduction**
• MTP preserved: the entire Multi-Token Prediction head (mtp.fc + all mtp.layers.0 linears) is kept in BF16, enabling native speculative decoding in vLLM — 1.55x measured decode throughput (127 vs 82 tok/s, single stream, RTX 5090) at 0.69 greedy draft acceptance on vLLM 0.27.1
• Purpose: the stock club-3090 dual compose serves this repo with a two-line change

Quantization details

Field	Value
Base	Qwen/Qwen3.8-27B (BF16 — never quantize from the FP8 repo)
Method	AutoRound (intel/auto-round), default recipe
Scheme	W4A16 (4-bit weights, FP16 activations)
Bits	4
Group size	128
Symmetric	yes
Packing format	auto_round:auto_gptq
Unquantized layers	linear_attn.in_proj_a/b, the full MTP head (mtp.fc, mtp.layers.0.* linears), all LayerNorms/RMSNorms, router gates, vision tower, embeddings, lm_head
Calibration samples	128 (default)
Iterations	200
torch.compile	enabled
GPU used for quant	1× RTX 5090 (32 GB, SM120), low_gpu_mem_usage=True
auto-round version	0.14.2
Quant wall time	~1h 10m (peak VRAM 27.7 GB, peak RAM 23.7 GB)

Unquantized layers — why

• linear_attn.in_proj_a/b: low-rank projections in the Gated DeltaNet blocks; shapes not divisible by 32 (group_size), so AutoRound skips them. Tiny parameter fraction.
• mtp.fc: the Multi-Token Prediction fusion layer — see the fix below.
• Norms, routers, vision tower, embeddings: precision-sensitive and small; kept at base precision (BF16), matching the reference quant and the official FP8 repo's own exclusion list.

MTP fix — what's different from a vanilla AutoRound run

A plain auto-round run on this architecture packs mtp.fc as INT4. In that form, vLLM's Qwen3_5MTP loader skips the layer entirely (it expects fc.weight, finds fc.qweight), and MTP speculative decoding silently runs at 0% acceptance.

This build goes one step further than the 3.6 reference quant and keeps the entire MTP head in BF16:

• mtp.fc: excluded from quantization natively via --fp_layers mtp.fc (auto-round ≥ 0.14 supports this), so it was never packed. Recorded in quantization_config.extra_config as 16-bit fp.
• mtp.layers.0 linears (7): auto-round 0.14.2's exporter RTN-packs these (untuned) at export time; they were replaced with the original BF16 weights from the base model post-hoc (single-shard surgery in model_extra_tensors.safetensors) and marked as 16-bit fp in extra_config.

The whole head is only ~220 MB, so size impact is negligible. Result: MTP works out of the box. If you serve without --speculative-config, the BF16 MTP weights are simply unused and harmless.

Validation (RTX 5090, vLLM 0.27.1)

Gate	Result
Code generation	PASS — correct, coherent Python
MTP acceptance	PASS — 0.69 greedy / ~0.58 sampled (non-thinking) / ~0.47 (thinking, temp 1.0); fail mode ≈0 excluded. An A/B test (INT4 vs BF16 MTP internals) scored identically, so the gap vs older ~0.8–0.9 reports is a vLLM-version difference, not quant quality
Decode speedup	1.55x (900-token completion: 7.1 s with MTP vs 11.0 s without, single stream)
Vision	PASS — correct scene description of an unlabeled photo

Note: on a single desktop-shared 32 GB GPU, a 131072 ctx OOMs at KV-cache init (hybrid-GDN state pool scales with max_num_seqs); validated at 32768 ctx / 32 seqs. The dual-3090 TP=2 target rig has sufficient headroom for full context.

Quantization environment

python 3.12 · auto-round 0.14.2 · transformers 5.15.0 · torch 2.13.0+cu130 · quantized from a local BF16 checkout of Qwen/Qwen3.8-27B