Qwen

/

Qwen3.8-27B

like 9.84k

Follow

Qwen 97.5k

The measurement

If an engine performs speculative decoding, predictable content generates faster than unpredictable content, because several tokens are validated per weight read. If it does not, speed is content-independent.

Two prompts, 200 tokens each, temperature 0, three repetitions, machine quiet:

engine              speculation   predictable   unpredictable   ratio
------------------  -----------   -----------   -------------   -----
GGUF q4_K_M         off           11.91 tok/s     11.78 tok/s   1.01x
GGUF mtp-q4_K_M     on (n=4)      11.40 tok/s      5.14 tok/s   2.22x
MLX  nvfp4          on            38.36 tok/s     20.26 tok/s   1.89x

The first row is the negative control. Note what it actually is: Ollama's 27b-q4_K_M and 27b-mtp-q4_K_M tags point at the same weights blob (f5f1dd8920d4, 16.81 GB). Their only difference lives in the params file — the second one carries draft_num_predict: 4, the first carries nothing. So this is not two models, it is one inference with speculation off, then on. Exactly one variable moves: how many tokens are drafted ahead. With speculation off, speed is content-independent. That is what tells us the protocol measures the intended mechanism and not word length, vocabulary size, or chance.

What this means

Ollama does activate the MTP heads. I initially assumed the opposite, and Ollama's own params file says otherwise in plain text.

Take q4_K_M without heads as the baseline: 11.78 tok/s is the cost of one token with no speculation.

• When drafts are rejected, mtp-q4_K_M falls to 5.14 tok/s — 2.3x slower than no speculation at all. The draft alone costs slightly more than a full pass of the model.
• When drafts are accepted, it climbs back to 11.40 tok/s — barely the no-heads baseline.

So under Ollama, speculation repays its own overhead in the best case and never more. Pure downside risk. Under MLX the same mechanism gives 20.26 when it fails and 38.36 when it succeeds, the failure case is not penalized at all.

Inference, not verified in code: this is what you would expect from lightweight heads reusing the trunk's hidden state, versus an implementation doing a full extra pass. I have not read either implementation; I observe their effects.

Physics check (independent path)

Weights in nvfp4 for 27.8B ≈ 13.9 GB. Bandwidth ≈ 270 GB/s. Ceiling = 19.4 tok/s.

                                    measured    vs ceiling
---------------------------------   ---------   ----------
MLX, unpredictable                  20.26       104 %
MLX, predictable                    38.36       198 %
GGUF no heads, any content          11.8         70 %
GGUF with heads, unpredictable       5.14        30 %

198% of the bandwidth ceiling is only possible by emitting several tokens per weight read, confirming the ratio through a path that depends on no software. (The 104% is likely residual acceptance even on rare words: commas, articles and word endings stay predictable.)

Reproducing it

ollama pull qwen3.8:27b-mlx
ollama pull qwen3.8:27b-mtp-q4_K_M
ollama pull qwen3.8:27b-q4_K_M

All three matter, without the no-heads control the result proves nothing.

Two calls to /api/generate per model, with "stream": false, "think": false, "options": {"temperature": 0, "num_predict": 200}.

Prompts as actually measured (French : I have not re-run them in English):

Predictable

Compte de 1 jusqu'à 200. Écris uniquement les nombres séparés par une virgule et un espace, sans aucun autre mot, sans saut de ligne. Commence par : 1, 2, 3,

Unpredictable

Écris 200 mots français rares choisis au hasard, séparés par une virgule et un espace. Aucun lien de sens entre eux, aucune phrase, aucun autre texte. Commence par : zythum, obombrer, cagnotte,

Read speed as eval_count / eval_duration. Three precautions: throw away a warm-up call; keep one model in memory at a time; make sure nothing else is querying the server. Verify that last point against a wall clock , my first pass showed 50% discrepancies from background processes, and without the second measurement path I would have published wrong numbers believing I had measured them.

Two integration traps found along the way

• think has no effect on the /v1 OpenAI-compatible endpoint. You must use reasoning_effort there. A system passing think: false on that path keeps reasoning active, pays the compute, and gets a cleaned-up answer — the bill without the trace.
• chat_template_kwargs: {enable_thinking: false}, documented for vLLM and SGLang, is a no-op under Ollama.

Scope

One machine, one model, two prompts, Ollama 0.32.13 and its MLX variant. I did not measure token acceptance rate directly, I infer it from throughput. I have not tested llama.cpp built manually with --spec-type draft-mtp, which may behave differently.

Happy to discuss further, I'm @sugaarak on X. If someone can run the same three-model protocol on another Apple Silicon config or on CUDA, I would like to know whether the GGUF draft cost is platform-specific or general.