Qwen3.6-27B-Alis-MLX-Dynamic

Apple Silicon (MLX) mixed-precision quantizations of Qwen/Qwen3.6-27B — a 27B dense hybrid (3:1 Gated DeltaNet : full attention, qwen3_5). One repo, one card — each quantization lives on its own branch.

branch	base	eff. bpw	size	target Mac	decode*	KL vs bf16 (top-1 flip)	status
main — multimodal	Qwen3.6-27B	4.704	16.09 GB	24 GB+	~36 tok/s	0.325 (same text weights)	✅ golden + vision tower at 8-bit
text	Qwen3.6-27B	4.601	15.47 GB	24 GB+	36.4 tok/s	0.325 (11.3%)	text-only artifact
mtp-bf16	—	bf16	0.85 GB	—	—	—	MTP head sidecar for self-speculative decoding

* single-stream decode on an M3 Ultra, short prompt; longer contexts decode slower. † measured against its own ThinkingCap bf16 reference (different base than the other rows).

Everything below is measured, not projected — KL / top-1 flip on a fixed 4096-token EN/code/ZH/KO slice (teacher-forced vs the bf16 base), strided perplexity (ctx 2048 / stride 1024), and single-stream decode on an M3 Ultra.

Which build?

• 24 GB+ Mac → main. 14 GiB weights + 15.8 GB measured peak at short context; int8 KV gives ~48K context in 1.5 GiB of cache.
• 16 GB Mac → temporarily withdrawn. The 2.9bpw ThinkingCap tier passed teacher-forced KL/PPL checks but free-running generation on hard long-CoT problems showed early stopping and repetition loops (its calibration mix contained no long thinking traces — teacher-forced metrics cannot catch this failure mode). CoT-heavy recalibration fixed generation form (MMLU-300 77.0%) but GPQA-class reasoning stays collapsed at 3bpw (30.3%) — a precision floor. No 16 GB reasoning tier is feasible in this family; for reasoning at minimum size use the ThinkingCap 4.6bpw quant (15.47 GB, GPQA 83.3).

Multimodal (default)

main carries the full multimodal stack — 4.6bpw text weights + the vision tower at 8-bit (verified with EN/KO OCR tests). Plain mlx-lm loads main fine for text-only use (vision weights are skipped); the text branch saves 0.62 GB if you never need images.

pip install mlx-vlm
python -m mlx_vlm generate --model avlp12/Qwen3.6-27B-Alis-MLX-Dynamic \
  --image photo.jpg --prompt "이미지의 텍스트를 읽어줘." --max-tokens 300

Usage

from mlx_lm import load, generate

model, tokenizer = load("avlp12/Qwen3.6-27B-Alis-MLX-Dynamic")                      # main

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "뮤텍스와 세마포어의 차이를 설명해줘."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

CLI / server:

mlx_lm.generate --model avlp12/Qwen3.6-27B-Alis-MLX-Dynamic --prompt "..." -m 512
hf download avlp12/Qwen3.6-27B-Alis-MLX-Dynamic --local-dir qwen36-27b
mlx_lm.server --model ./qwen36-27b --port 8080

Long context: quantize the KV cache in the generate path (--kv-bits 8 --kv-group-size 64) — measured to be quality-free (below).

Recipes (verified from each build's config.json)

Per-tensor bits assigned by marginal utility from a 2–6-bit + nvfp4/mxfp4 weight-space sensitivity scan of all 498 text tensors, then validated end-to-end against uniform baselines (the scan alone is not trustworthy — see the nvfp4 finding below).

tier	bulk	upgrades	pins
main (4.6bpw)	90.1% affine 4-bit g64	9.8% sensitivity-ranked → 5-bit g64	routers/small gates 6-bit; embed & lm_head ≥4-bit

The golden spot for vanilla Qwen3.6-27B (survey of 102 MLX quants)

We surveyed every vanilla Qwen3.6-27B MLX quant on the Hub (102 repos; finetunes, abliterations and GGUF excluded) and measured the serious ≤16 GB contenders on our fixed KL ruler (4096-token EN/code/ZH/KO, teacher-forced vs bf16):

build	weights	KL vs bf16 ↓
text (ours, 4.6bpw affine)	15.47 GB	0.325
deepsweet oQ4-FP16	15.80 GB	0.385
mlx-community / lmstudio-community 4-bit g64	16.05 GB	0.383

Nothing at or below this size class quantizes vanilla Qwen3.6-27B better — every lower-KL build we found is strictly larger (mlx-community 5-bit 19.4 GB, OptiQ-4bit 20.0 GB, Intel AutoRound int4 19.0 GB, …). This build is the quality-per-byte knee of the curve for the vanilla model. (For higher absolute quality at this size, use the ThinkingCap sibling: GPQA 83.3 vs this build's 72.7 — token-efficient bases quantize far better. This repo is the golden spot for the vanilla weights specifically.)

The nvfp4 finding. The scan's weight-space rel_error consistently rates nvfp4 (g16) above affine 4-bit (g64) at equal cost on this model — but end-to-end KL says the opposite: pure-nvfp4 4.5bpw scored 0.450 vs uniform affine-4bit's 0.383. Qwen's activation outliers appear to favor affine's per-group bias term. main is therefore affine-only; trust weight-space scans for ranking within a format, not across formats. Note the finding is architecture-dependent: on the 35B-A3B MoE sibling the reverse was measured — its nvfp4-mix beats an affine-only rebuild (0.292 vs 0.309).

DWQ. All tiers went through layerwise DWQ (KL distillation vs the bf16 teacher; 260-sample EN/KO/ZH/code chat-template mix, seq 512, validation-gated rounds — see alis-dwq). At ≥4bpw the gates rejected every round (the quantized model is already too close to the teacher for lr ≥2e-6 to help), so main carries its conversion-time weights. At 2.9bpw DWQ is decisive: the 16gb tier's valid KL dropped 0.436 → 0.298 (−32%).

Quality

KL(bf16‖quant) / top-1 flip on the fixed 4096-token slice, per domain:

build	overall	EN	code	ZH	KO
main 4.6bpw	0.325 / 11.3%	0.146 / 6.7%	0.861 / 12.5%	0.122 / 12.7%	0.171 / 13.1%
uniform 4-bit g64 (4.5bpw)	0.383 / 12.7%
v1 nvfp4-mix 4.2bpw (unreleased)	0.479 / 15.5%	0.229 / 9.7%	1.094 / 15.3%	0.260 / 18.4%	0.332 / 18.5%
uniform 3-bit g64 (3.5bpw)	0.808 / 24.6%
16gb-thinkingcap 2.9bpw†	1.375 / 35.5%	0.651 / 25.3%	1.967 / 23.9%	1.397 / 45.0%	1.487 / 47.8%

Strided perplexity (ctx 2048, stride 1024; wikitext / code / Korean):

build	wikitext	code	KO
main	5.95 [5.86, 6.03]	2.01 [1.99, 2.03]	6.67 [6.60, 6.73]
16gb-thinkingcap†	7.96 [7.84, 8.08]	2.62 [2.59, 2.65]	13.67 [13.51, 13.83]

† different bf16 reference (ThinkingCap); PPL is comparable across rows, KL is not.

Generative benchmark (GPQA-Diamond, temp 1.0, our harness)

Long-CoT hard reasoning is far more quantization-sensitive than teacher-forced metrics suggest — measure generatively before trusting any low-bit build:

model	weights	GPQA-D acc	avg thinking tokens
ThinkingCap bf16 (harness check; card reports 83.8)	55.56 GB	83.8	3,019
TC 4.6bpw quant	15.47 GB	83.3	2,838
vanilla 5.5bpw (unreleased, available on request)	18.49 GB	79.8	11,138
main 4.6bpw (vanilla base, 16K cap)	15.47 GB	72.7	5,987
3.0bpw CoT-recalibrated (withdrawn tier, ref.)	10.09 GB	30.3	2,312

At ~3bpw generation form is fixable with CoT-heavy DWQ calibration (no more early stops/loops; MMLU-300 77.0%) but GPQA-class reasoning collapses — a precision floor, not a calibration artifact.

KV cache — measured recommendations

KV quantization drift is 2–3 orders of magnitude below weight-quantization drift (same-model logit KL, 2048-token chunked forward on the 16gb build):

KV config	KL vs bf16-KV	KV bytes/token	~ctx in 1.5 GiB
bf16	0	64 KB	24 K
int8 g64 (recommended)	0.0006	32 KB	48 K
K4/V8 (long-context)	0.0032	24 KB	64 K

Qwen3.6-27B is a 3:1 hybrid — only 16 of 64 layers carry KV at all; the other 48 are Gated DeltaNet with a fixed-size recurrent state. KV is already ~4× cheaper than a dense transformer of this depth before any quantization.

Notes & provenance

• main carries the full multimodal stack (mlx-vlm); text is the mlx-lm text-only artifact; mtp-bf16 carries the MTP head as a bf16 sidecar.
• Calibration/eval data: tulu-3 SFT chat (EN), Korean chat, code/ZH mix — chat-template rendered.
• Built on an M3 Ultra 512 GB · mlx-lm 0.31.3 · mlx 0.31.2 · 2026-07.
• Sibling repo: Qwen3.6-35B-A3B-Alis-MLX-Dynamic (MoE, 102–111 tok/s decode).

Credits

• Qwen team — the Qwen3.6-27B base model and its hybrid DeltaNet architecture.
• bottlecapai — ThinkingCap-Qwen3.6-27B, the token-efficiency finetune behind the 16gb-thinkingcap tier.
• Apple MLX team — mlx / mlx-lm, including the DWQ trainer this pipeline builds on.
• Unsloth — their NVFP4 releases motivated the diverse-chat calibration mix and served as quality reference points.
• AllenAI — tulu-3 SFT mixture used in the calibration set.
• Quantization recipe, DWQ retune and evals: Alis (avlp12) with the alis-dwq pipeline; built with assistance from Claude (Anthropic).