AtomicChat/Qwen3.8-27B-GGUF

How to Run Qwen3.8 27B Locally

Built from Qwen's original weights with our own importance matrix. The calibration corpora behind our builds are public.

• See our quantization analysis below for measurements and instructions.
• You can now run Qwen3.8 in Atomic Chat with toggles for thinking.

---

Every quantization of this model we could find, ours and everyone else's, measured against the original weights on the same held-out text. The dashed line is the best file available at each size from anyone but us.

Pick a file

Every number below is measured, not estimated. How we measured it is at the bottom, and the raw logs are in the metrics repo so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the original weights. Lower is better, and zero means identical. top-1 is how often it picks the same next word the original would have picked.

File	Size	KL divergence	top-1
Q8_0	28.9 GB	0.00064	98.92%
AD-Q6_K	25.0 GB	0.00107	98.67%
AD-Q6_K-Q5_K	23.1 GB	0.00252	97.94%
AD-Q5_K	20.2 GB	0.00419	97.34%
AD-Q5_K-Q4_K	18.6 GB	0.00730	96.43%
AD-Q4_K	17.1 GB	0.01126	95.59%
AD-IQ4_XS	16.5 GB	0.01248	95.39%
AD-IQ4_XS-IQ3_S	14.4 GB	0.02660	93.15%
AD-IQ3_S	13.8 GB	0.03247	92.41%
AD-IQ3_S-IQ3_XXS	13.0 GB	0.04337	91.33%
AD-IQ3_XXS	12.1 GB	0.06972	89.13%
AD-IQ2_S	11.1 GB	0.09832	87.18%
AD-IQ2_S-IQ2_XS	10.2 GB	0.13807	84.77%
AD-IQ2_XS	9.9 GB	0.16170	83.48%
AD-IQ2_XXS	9.0 GB	0.25663	79.44%
AD-IQ1_M	8.5 GB	0.34212	76.34%

AD- marks an Atomic Dynamic layout. The name says what the two largest tensor groups got: AD-<ffn_down>-<ffn_up>, collapsed to one name when both match. Nothing is named after a type it does not contain.

Which one fits your card

The file has to fit, and so does the context. This model keeps 256 KB of attention cache per token, which is 2 GB at 8k context and 8 GB at 32k. Budget for both.

Your card	File	Leaves room for
12 GB	AD-IQ2_S	short context only, or move some layers to CPU
16 GB	AD-IQ3_S	around 8k context
24 GB	AD-Q5_K-Q4_K	around 16k context
32 GB	AD-Q6_K	around 24k context
48 GB and up	Q8_0	full context

If you are choosing between two neighbouring files, take the larger one. The steps between them cost one or two gigabytes and buy noticeably fewer wrong words, especially below 14 GB where the curve gets steep.

Running it

llama-server -m Qwen3.8-27B-AD-Q4_K.gguf -ngl 99 -c 8192

Prompt format:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

The model ships a multi token prediction head. It is inside every file here and needs no extra download:

llama-cli -m Qwen3.8-27B-AD-Q4_K.gguf --spec-type draft-mtp -ngl 99 -c 8192

How these compare to other builds

Numbers taken against different references cannot be put in the same table, so we did not copy anyone's published figures. We downloaded their files and measured them ourselves, against the same original weights, on the same held-out text. Their logs are in the metrics repo next to ours.

At sizes where the comparison is direct:

Size	Ours	Best other build at that size
20.2 GB	AD-Q5_K 0.00419	unsloth UD-Q5_K_XL 0.00437
25.0 vs 25.9 GB	AD-Q6_K 0.00107	unsloth UD-Q6_K_XL 0.00110 at 0.9 GB more
16.5 vs 16.1 GB	AD-IQ4_XS 0.01248	Q4_K_S 0.01707
12.1 vs 11.9 GB	AD-IQ3_XXS 0.06972	unsloth UD-IQ3_XXS 0.07330

One honest exception. At 17.9 GB unsloth's UD-Q4_K_XL reaches 0.00955, which is better than our 17.1 GB file and close to our 18.6 GB one. Around 18 GB their build and ours are within a few percent of each other. We are ahead across most of the range, not all of it.

One thing worth seeing on its own. Three publishers ship a file called Q4_K_M, and they are not the same file:

Publisher	Size	KL divergence
lmstudio-community	16.8 GB	0.02094
ggml-org	19.0 GB	0.01470
ours, AD-Q4_K	17.1 GB	0.01126

Same name, two gigabytes apart, and nearly a factor of two in accuracy. A quant name tells you which recipe was requested, not what you are getting.

What we found while building these

Where the bits go matters more than how many there are

We built ten versions of the same 17 to 19 GB file, changing only which tensors got the extra bits, and measured each one against the original weights.

Layout	Size	KL divergence
every layer treated the same	16.8 GB	0.01580
4 layers lifted	17.1 GB	0.01449
more bits on ffn_down everywhere	17.8 GB	0.01189
more bits on attention	18.2 GB	0.01010
16 layers lifted, first and last	17.8 GB	0.00981
32 layers lifted instead	18.4 GB	0.00826
16 lifted, plus the attention gate	18.4 GB	0.00821
16 lifted, plus a richer output head	18.8 GB	0.00800
24 layers lifted	18.6 GB	0.00743
24 lifted, plus attention gate and state output	18.6 GB	0.00730

Half the divergence disappears at the same file size, purely from moving bits around. Every file in the ladder above uses the last layout.

The ends of the network are worth more than the middle

Lifting the first four and last twelve layers helped more than anything else we tried. Widening that band to 32 layers did not help further, and spending the same bits on the output head helped less. The importance matrix agrees: the highest activation energy in the whole model sits on layers 52 to 62, with a second peak on layer 0.

This model is a hybrid, and two small groups carry a lot

Alongside ordinary attention, Qwen3.8 has an attention gate and a state output path. They are 5.5% of the weights each. Giving both one extra step of precision cost 0.16 GB and removed 11% of the remaining divergence. That was the single best trade we found.

The embedding table is cheaper than it looks

The token embedding table and the output head weigh the same, 4.7% each, and behave nothing alike. The head decides the next word directly and has to stay precise. The embedding table is a lookup whose error stays inside one token, so it can be cut hard. Paying for the head out of the embedding table is a net gain at every size we tested.

Q8_0 is not lossless

It is very close, but it is not the original: 0.00064 divergence and 98.92% top-1 agreement. We say so because our whole table is measured against the real BF16 weights rather than against Q8_0, and that changes every number in it. Anyone measuring against a Q8_0 reference will get smaller numbers than these for the same files.

The prediction head collects no calibration data

The multi token prediction head is never executed during a normal forward pass, so the importance matrix has nothing to say about it at any corpus size. Quantize it low and llama.cpp refuses partway through rather than guess. It is pinned to q5_k in every file here.

Every tensor row divides by 256

Worth checking before you plan a ladder. K and I quants store weights in superblocks of 256, and a model whose rows do not divide by that number silently falls back to a coarser type while keeping the name you asked for. On this model everything divides cleanly, so the whole range from IQ1_M upward is genuinely available. That is not true of every recent release.

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe qwen3.8-27b. It is 4,967,044 tokens across 3,004 documents:

Part	Share
agentic and tool use	24.7%
code	17.8%
reasoning	14.8%
multilingual	13.8%
long context	11.9%
vocabulary sweep	9.9%
structured data	4.1%
graphics	3.0%

Two details that matter more than the mix.

Every conversation is rendered through this model's own chat template, so <|im_start|>, <think>, <tool_call> and <tool_response> appear as the single tokens the model actually sees. Text rendered for another model is excluded from the build rather than reused.

The vocabulary sweep is regenerated for this tokenizer. It covers 246,919 of the 247,133 vocabulary entries that can stand alone, at about two tokens per entry. A sweep built for a different model covers a different vocabulary and does nothing here.

If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation, and the agentic and reasoning parts of the corpus calibrate on text the model never sees.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out set is 4.5219 plus or minus 0.0238.

Held-out text: eval_neutral from our calibration dataset, never used for calibration. 87 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement.

Hardware: 4x RTX 5090, CUDA 13.0. The reference needs all four cards because the BF16 file is 51 GB.

The raw logits of the reference are published in the metrics repo, split into parts because of the file size limit. With them you can measure your own build against exactly the same point we did:

cat base-neutral.kld.*.part > base-neutral.kld

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, split across seven workers and merged. Splitting by chunk range gives the same result as one long run, because the statistic is a sum:

llama-imatrix -m Qwen3.8-27B-bf16.gguf -f calib_train.txt -o shard-0.gguf \
  -ngl 99 -c 512 -b 4096 -ub 4096 --parse-special --from-chunk 0 --chunks 1400

llama-imatrix -m Qwen3.8-27B-bf16.gguf --in-file shard-0.gguf,shard-1.gguf,... \
  -o imatrix.gguf

Then, for AD-Q4_K:

llama-quantize --imatrix imatrix.gguf \
  --tensor-type 'blk\.64\.=q5_k' \
  --tensor-type 'blk\.([0-3]|5[2-9]|6[0-3])\.ffn_.*=q5_k' \
  --tensor-type 'blk\.([4-9]|1[01])\.ffn_.*=q5_k' \
  --tensor-type ffn_down=q4_k --tensor-type ffn_gate=q4_k --tensor-type ffn_up=q4_k \
  --tensor-type attn_q=q4_k --tensor-type attn_gate=q5_k --tensor-type ssm_out=q5_k \
  --tensor-type output=q6_k --tensor-type token_embd=iq4_xs \
  Qwen3.8-27B-bf16.gguf Qwen3.8-27B-AD-Q4_K.gguf Q8_0

The rules for every other file are in its quantization log in the metrics repo.

Also in this repo

baseline/ holds the stock Q4_K_M, Q5_K_M and Q6_K mixes, built from the same importance matrix and measured against the same reference. They are the control: without them, the claim that a hand-assigned layout beats the standard recipe at equal size has nothing to stand on. For actual use, take the AD- file of the same size instead.

Model details

64 layers, hidden size 5120, feed-forward size 17408, vocabulary 248,320, context 262,144, no sliding window. Hybrid attention with an added gate and a state path. One multi token prediction head.

Needs a recent llama.cpp with qwen35 architecture support.