JC1DA/Qwen3.8-27B-INT4-INT8-Mixed

Qwen3.8-27B — Mixed W4/W8 Quantized (compressed-tensors)

Mixed-precision quantized checkpoint of Qwen/Qwen3.8-27B (hybrid GDN linear-attention + full-attention, qwen3_5 architecture). Built by taking inspiration from Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 (full W8A8 via SmoothQuant) and pushing a subset of layers further down to W4A16 to make the model smaller and faster to load/serve.

Close to FP16 quality, much faster. The mixed W4/W8 scheme targets near-FP16 accuracy (exact numbers in Benchmarks below) while the smaller weights and INT8 tensor-core GEMMs make it meaningfully faster to load and to serve than the FP16 weights.

Quantization scheme

Compressed-tensors, format: mixed-precision, three weight groups plus BF16 exclusions:

Group	Scheme	Modules	Weight bits (logical)	Activation	Storage format
group_1	W4A16	204	4-bit, group-128, symmetric	BF16	pack-quantized
group_0	W8A16	78	8-bit, per-channel, symmetric	BF16	pack-quantized
group_2	W8A8	118	8-bit, per-channel, symmetric	INT8, per-token, dynamic	int-quantized

Kept BF16 (not quantized): embed_tokens, lm_head, all layer norms, linear_attn.in_proj_a / in_proj_b / conv1d / A_log / dt_bias, the full vision tower (model.visual.*), and the MTP head (mtp.*, shipped in model_extra_tensors.safetensors).

Per-layer layout

64 layers: 48 GDN linear-attention + 16 full-attention (layers 3, 7, 11, …, 63). Quantization is assigned per module, not per layer — most layers mix schemes:

Layers	attn in_proj_qkv/in_proj_z or q/k/v	attn out_proj/o_proj	mlp.gate/up	mlp.down_proj
0, 1, 2, 4, 5, 50	W4A16	W8A16	W4A16	W4A16
6, 8, 9	W4A16	W8A16	W4A16	W8A16
3	W4A16	W4A16	W4A16	W4A16
7	W4A16	W8A8	W4A16	W4A16
11, 15, 19, 23, 27, 31, 35, 43	W8A8	W8A8	W4A16	W8A16
14, 16, 17, 18, 20	W8A8	W8A16	W4A16	W8A16
21	W4A16	W8A16	W4A16	W8A16
22, 36–38, 40–42, 44–46	W4A16	W8A16	W8A8	W8A16
24, 25, 26, 28–30, 32–34	W8A8	W8A16	W8A8	W8A16
39, 47	W4A16	W8A8	W8A8	W8A16
48, 49, 52–54, 56–58, 60–62	W4A16 (incl. out_proj)	W4A16	W4A16	W4A16
51, 55, 59	W4A16	W8A8	W4A16	W4A16
63	W4A16	W4A16	W4A16	W4A16

Pattern: mid-depth layers (roughly 11–47) carry the W8A8 modules (only group with dynamic INT8 activations — INT8×INT8 GEMMs on tensor cores); early and late layers are mostly W4A16. down_proj never uses W8A8.

Size

Component	Tensors	Logical weight size
W4A16	204	5.90 GiB
W8A16	78	4.49 GiB
W8A8	118	6.37 GiB
BF16 (embed, lm_head, norms, vision, MTP)	522	5.64 GiB
Total weights (logical)		~22.4 GiB

On disk: 23.4 GiB (8 model shards + model_extra_tensors.safetensors with the BF16 MTP head). vs. ~52 GiB for the BF16 original.

Model specs (inherited from Qwen3.8-27B)

• model_type: qwen3_5, architecture Qwen3_5ForConditionalGeneration (vision + language)
• 64 layers, hidden 5120, 24 attention heads / 4 KV heads, MLP intermediate 17408
• Vocab 248,320, max position 262,144, rms_norm_eps 1e-06, no tied embeddings

Files

File	Contents
model-0000X-of-00008.safetensors	Quantized language-model weights (W4A16/W8A16 packed, W8A8 INT8, BF16 exclusions)
model_extra_tensors.safetensors	BF16 MTP speculative head (mtp.*)
quantization_config.json	Compressed-tensors group definitions (source of truth for the table above)
config.json, generation_config.json, tokenizer files, chat_template.jinja	Standard HF artifacts
preprocessor/processor_config.json	Vision preprocessing (vision tower kept BF16)

Serving notes

• quant_method: compressed-tensors → load with vLLM (compressed-tensors kernels): W8A8 modules route to CUTLASS INT8 scaled-mm (needs Ampere SM86+ for INT8 tensor-core GEMMs), W4A16/W8A16 packed groups route to Marlin.
• --dtype bfloat16 (activations stay BF16 for W4A16/W8A16 groups; INT8 activations are quantized dynamically at runtime for W8A8 groups).
• Add --language-model-only if the vision tower is not needed.
• MTP head available for {"method": "mtp"} speculative decoding if your build supports it.

Serving with SGLang

python3 -m sglang.launch_server \
    --model-path JC1DA/Qwen3.8-27B-INT4-INT8-Mixed \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --kv-cache-dtype fp8_e4m3 \
    --mamba-ssm-dtype bfloat16 \
    --mem-fraction-static 0.85 \
    --tp 2 --host 0.0.0.0 --port 8000 \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/Qwen3.8-27B-DFlash2

Credits

Inspired by Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 (SmoothQuant + activation-aware GPTQ, full W8A8). This checkpoint starts from that W8 recipe and demotes selected modules to W4A16 (group-128) to shrink the checkpoint and speed up memory-bound serving.

Benchmarks

AIME 2026 (I + II, 30 problems)

0-shot, \boxed{} extraction, EvalScope AIME adapter. This benchmark caps max_tokens at 100000 per request.

Model	Precision	Score
Qwen3.8-27B-W4/W8 (this repo)	mixed W4/W8	96.7 % (29/30)
Qwen3.8-27B-FP16 (reference)	BF16	96.7 % (29/30)