antonyMox/Qwen3.6-27B-AutoRound-INT4-wide-MTP

// ANTONYMOX · PARK RECORD · SMARTEST BUILD

Qwen3.6-27B wide — INT4 · MTP

The clean Qwen3.6-27B base, quantized our way — big linear-attention projections kept in BF16. Highest hard-coding score in our fleet (78), MTP head preserved, sees images, up to 256k context (native).

hard 78 · record MTP preserved vision 256k ctx 26 GB Apache-2.0

78

LCB-HARD 🏆

83

TOOL-USE

+8

PP vs NARROW

83

TOK/S · MTP3

26GB

ON DISK

📊 All benchmarks & throughput measured on a single RTX 4090 (48 GB) · vLLM 0.19 · MTP n=3 · seed 42

The clean Qwen/Qwen3.6-27B base (no distill, no abliteration — creator alignment intact), quantized by us with AutoRound INT4 using a wide recipe: the large linear-attention (SSM/GDN) projections are kept in BF16 instead of INT4. That single choice lifts hard-coding from 70 → 78 (+8 pp) over the standard narrow quant — the highest hard score in our fleet, at a third the size of the 80B it beats. MTP head preserved → speculative decoding works in vLLM out of the box.

---

Highlights

🏆 Fleet record — hard 78

our smartest build; beats the 80B Coder-Next (62) at ⅓ the size

🧠 Clean base, no distill

raw Qwen3.6-27B intelligence, not a fine-tune

🔬 The "wide" trick: +8 pp

big SSM projections kept BF16 → long-range logic survives

🧬 MTP preserved

15-tensor head kept → speculative decoding just works

🖼️ Multimodal

images & video — vision tower full BF16

📏 256k context (native)

supports 256k; full 256k fits on 48 GB at mem 0.80

⚡ 83 tok/s — brisk for dense

MTP n=3; not a slow model despite being dense

🧾 Proof in-repo

the per-task benchmark JSON ships with it

---

Benchmarks — the wide trick pays off

Same clean base Qwen/Qwen3.6-27B, one harness — only the quant recipe changes. Keeping the big SSM projections in BF16 lifts hard +8 pp over the standard narrow quant:

Model (our harness)	Recipe	hard	tools	Size
Lorbus Qwen3.6-27B narrow INT4 (reference)	narrow	70	83	18.5 GB
abliterated sister	narrow + Heretic	70	87	19 GB
➡️ wide (this)	wide	78 🏆	83	26 GB
antonyMox 35B-A3B (coming soon)	MoE narrow	48	83	22 GB
Qwen3-Coder-Next 80B (UD-IQ4_XS, reference)	80B MoE	62	80	~38 GB

+8 pp hard for +7.5 GB over narrow — and it beats Qwen3-Coder-Next 80B (hard 62 on our harness), three times its size.

Numbers are % of tasks passed (40 hard coding + 30 tool tasks; raw pass/total in benchmark-results/). Conditions: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 · seed 42 · non-thinking · vLLM 0.19 · RTX 4090 (48 GB) · MTP n=3. The abliterated & wide rows share the exact 40 hard tasks the narrow quant scored 70 on → clean apples-to-apples. The Lorbus row is our own measurement of their public quant, not their claim. The 80B Coder-Next row is likewise our own measurement of the UD-IQ4_XS GGUF (via LM Studio) on the same task set — a bigger, different model shown for scale. Reproduce it, don't trust it.

---

🔬 The "wide" trick — why it scores higher

Qwen3.6 is a hybrid model: most layers are linear-attention (GDN/Mamba), where long-range logic flows through the large projections in_proj_qkv, in_proj_z, out_proj. Standard "narrow" INT4 quants compress these to 4-bit too — and that quietly costs reasoning. We keep them in BF16:

Kept in BF16 (not quantized)	Tensors	Why
🔬 Big SSM projections (the wide trick)	in_proj_qkv, in_proj_z, out_proj	carry long-range logic — INT4 here costs ~8 pp hard
🌊 Small SSM control	A_log, conv1d, dt_bias, in_proj_a/b, norm	SSM recurrence stability
🧬 MTP head	15× mtp.*	makes speculative decoding work
👁️ Vision tower	333× visual.*	full image/video, untouched

Everything else (full-attention q/k/v/o, MLP gate/up/down) is INT4. The recipe was validated on Qwen3.5 first (+6 pp) and confirmed here on 3.6 (+8 pp).

---

⚡ Speed — MTP speculative decoding

Honest 768-token × 3-run measurement on a single RTX 4090 (48 GB), vLLM 0.19:

MTP n	short tok/s	28k-ctx tok/s	acceptance @28k
1	55	33	79%
2	71	39	64%
3 ⭐	83	40	52%
4	92	42	44%
5	96	40	37%

Use num_speculative_tokens: 3 — long-context throughput is already on its plateau (~40 tok/s) while acceptance stays highest (52%), so the draft head misfires least. Spec-decode is lossless — n changes speed only, never output.

---

🖼️ Multimodal — sees images & video

The full vision-language model. The entire vision tower (333 tensors) is kept in full BF16 — quantization never touched it — so image/video understanding is identical to the pristine base. Serve with --limit-mm-per-prompt '{"image":4,"video":1}'. Published benchmarks cover text/coding/tool-use; vision quality is inherited from the base, not separately re-scored.

---

🚀 Quickstart — this is a big-card model

Weights are 26 GB, so this is a big-card model. A 48 GB card is its home; a 32 GB card runs it but with only a small context window; 24 GB can't load it at all.

GPU	VRAM	Script	max-model-len	gpu-mem-util	Status
RTX 4090 48G / A6000 / L40S / RTX 6000	48 GB	run_48gb.sh	150 000	0.72	✅ tested (benchmark rig)
RTX 5090	32 GB	run_32gb.sh	16 384	0.95	✅ verified (tight; dedicated card)
RTX 3090 / 4090	24 GB	—	—	—	❌ 26 GB weights don't fit

32 GB is tight. The 26 GB weights leave only ~1.7 GB for KV even at mem-util 0.95, so the safe window is ~16k tokens (measured by emulating a 32 GB budget on a 48 GB card: concurrency ~1.5x at 16k; 24k dropped to 1.03x = the hang zone). mem 0.95 assumes a dedicated/headless card — lower it if a display is attached. Want long context on ≤ 32 GB? Use the 19 GB abliterated sister (fits 24 GB with 40k ctx, or 32 GB with much more).

vllm serve antonyMox/Qwen3.6-27B-AutoRound-INT4-wide-MTP \
  --quantization auto-round \
  --max-model-len 150000 \
  --gpu-memory-utilization 0.72 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --trust-remote-code

Long context is cheap: Qwen3.6 is a hybrid (Mamba/GDN + attention), KV costs ~37 MB/1k tokens at fp8. The model's native maximum is 256k (262,144) tokens — how much fits on 48 GB depends on mem-util:

Context vs mem-util (48 GB card, measured)

gpu-mem-util	context that fits	notes
0.72 ⭐ (default preset)	~150k (conc 1.15x)	balanced; leaves VRAM free
0.80	256k (conc 1.11x)	full native context

Raise --gpu-memory-utilization toward 0.80 for the full 256k; the 0.72 default keeps more VRAM free (this variant's 26 GB weights leave less room than the 19 GB abliterated sister).

Never set --max-model-len at exact KV capacity. On vLLM 0.19 hybrid models, requests in the last ~3 % of the window pass validation then hang forever (no error, GPU idle) — the scheduler can't allocate the final KV block. Keep the window a couple of blocks below capacity: after launch, the log line Maximum concurrency for N tokens: Yx must be ≥ ~1.10x (ours is 1.15x at 150k), never 1.00x. Also, the GPU KV cache size: X tokens log line understates real capacity ~3× on hybrids — trust only the Maximum concurrency line.

---

🧬 The family

Clean quants of the same base — pick by hardware:

Model	Size	hard	tools	Best for
➡️ wide (this)	26 GB	78 🏆	83	48 GB+ cards — maximum reasoning
abliterated	19 GB	70	87	24 GB cards · uncensored
35B-A3B (coming soon)	22 GB	48	83	fast MoE (~188 tok/s)

The 27B sisters: AutoRound INT4, MTP preserved, multimodal. All benchmarked against the community reference Lorbus narrow INT4 (hard 70) — see the table above.

---

🍳 Recipe (reproducible)

1. Base — pristine Qwen/Qwen3.6-27B BF16.
2. Quant — AutoRound W4A16, group 128, 200 iters, v2 calibration.
3. Wide fp_layers (kept BF16) — mtp.*, visual.*, small SSM (in_proj_a/b, A_log, conv1d, dt_bias, linear_attn.norm) + the wide trick: in_proj_qkv, in_proj_z, out_proj.
4. MTP — 15 mtp.* tensors preserved (BF16), n=3 recommended.
5. Sampler — ships temp 1.0 / top_p 0.95 / top_k 20 (generation_config.json).

🔢 Why HuggingFace shows "~11.5B params"

Not a small model — it's how the widget reads a partially-packed INT4 quant. The INT4 weight tensors are packed 8× int4 into each int32 (HF counts fewer slots), while the large wide projections, vision, and MTP stay full BF16 (counted fully). The true model is Qwen3.6-27B, 26 GB on disk. Verified breakdown: language_model 9.35 B · other 1.27 B · vision 0.46 B (333 t, BF16) · mtp 0.42 B (15 t, BF16). (The wide sister shows more BF16 than a narrow quant precisely because it keeps those big projections full-precision — that's the whole point.)

📜 License & attribution

Apache-2.0, same as the base. Quantized from Qwen/Qwen3.6-27B with a wide AutoRound INT4 recipe. This is the clean base (creator safety alignment intact). Community build; benchmark numbers are from our own harness — JSON included, methodology open. Provided "as is", without warranty.