GreenBitAI/Qwen3.8-27B-4bit

Qwen3.8-27B, 4-bit, with its draft head

Qwen3.8-27B quantized to four bits for MLX, carrying the model's own multi-token prediction head in mtp/. Speculative decoding therefore works from this repository alone -- nothing else to fetch, no environment variable pointing somewhere else.

Generation runs 1.8-2.3x faster with the head on, and says the same thing. Every token it proposes is checked by the model itself, so the reply is the model's own either way; the head only saves passes over the weights.

Speed

Measured on a 48 GB MacBook Pro (M4 Pro), greedy decoding, 96 tokens, with gbx_lm. Decode is timed from the first token, so prefill is not in it.

context	decode, head off	decode, head on	draft acceptance
1,024	14.7 tok/s	33.8 tok/s	0.95
4,096	14.3	27.3	0.78
16,384	13.5	24.8	0.78

Use

pip install gbx-lm

# the head is off unless asked for, and found in `mtp/` without any path
GBX_QWEN35_MTP=on python -m gbx_lm.generate \
    --model GreenBitAI/Qwen3.8-27B-4bit --max-tokens 256 --prompt "..."

The same switch works for the server:

GBX_QWEN35_MTP=on python -m gbx_lm.fastapi_server --model GreenBitAI/Qwen3.8-27B-4bit

What is in here

mtp/mtp.safetensors is built from the draft head Qwen/Qwen3.8-27B ships under mtp., quantized to match these weights. Apache 2.0, as the original is.