ThinkingCap-Qwen3.8-27B-abliterated-GGUF

GGUF quantizations of ThinkingCap-Qwen3.8-27B-abliterated, the uncensored variant of bottlecapai/ThinkingCap-Qwen3.8-27B, for llama.cpp and apps built on it.

Every file is quantized directly from the BF16 GGUF, never from another quant. All quants below Q8_0 use an importance matrix computed on this model.

Files

File	Type	Size	Mean KLD vs BF16	Notes
…-Q8_0.gguf	Q8_0	29.1 GB		effectively lossless
…-Q6_K.gguf	Q6_K	22.5 GB		recommended if it fits
…-Q5_K_M.gguf	Q5_K_M	19.7 GB		excellent quality
…-Q4_K_M.gguf	Q4_K_M	19.5 GB		the usual default
…-IQ4_XS.gguf	IQ4_XS	18.1 GB		smallest with little loss
…-IQ3_M.gguf	IQ3_M	17.3 GB		smallest offered; visibly degraded
mmproj-…-f16.gguf	F16	0.93 GB	—	vision tower; add for image input
…imatrix.gguf	—	0.01 GB	—	importance matrix used for all quants

KLD method: llama-perplexity --kl-divergence against the BF16 GGUF on WikiText-2 test, context 2048, 60 chunks. Under 0.01 is imperceptible, 0.01–0.05 is good, above 0.1 is noticeable. Refusal behaviour was measured on the bf16 model (6/100 on Heretic's evaluator), not on each GGUF.

Why these files are larger than usual for their type

This is a hybrid model. 48 of its 64 layers are Gated DeltaNet (linear attention with a recurrent state), which holds 5.6B of the 28B parameters. Three groups of tensors are pinned to higher precision than the file's nominal type:

Tensors	Pinned to	Applied to	Why
MTP head (blk.64.*)	Q8_0	every file	it never runs in a normal forward pass, so the imatrix has no data for it; Q8_0 keeps speculative-decoding acceptance high
DeltaNet (blk.*.attn_qkv, attn_gate, ssm_*)	Q8_0	Q4_K_M and below	the recurrent block is the part of this architecture most sensitive to low-bit quantization
token_embd, output	Q8_0	below Q4	standard for sub-4-bit builds

The pins make sizes non-monotonic across type names. Measured on these files:

• Q5_K_S (19.1 GB) is smaller than Q4_K_M (19.5 GB). Q5 types keep DeltaNet at Q5_K; Q4 types pin it to Q8_0.
• Q3_K_M (18.0 GB) and IQ4_XS (18.1 GB) are the same size, as are Q3_K_L and Q4_K_S (18.8 GB).
• Below that, sizes fall slowly: IQ3_M 17.3 GB, IQ2_M 15.3 GB, IQ1_S 12.9 GB, because about 9 GB of every sub-4-bit file is pinned at Q8_0 (DeltaNet 5.9 GB, embeddings + output 2.7 GB, MTP 0.45 GB).

So choose by the KLD column at your size budget, not by the type name. Types not listed here were not better than a listed file of similar size.

Usage

llama-server, with MTP speculative decoding:

llama-server -m ThinkingCap-Qwen3.8-27B-abliterated-Q6_K.gguf \
  -ngl 99 -c 32768 -fa on --jinja --reasoning-format deepseek \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 \
  --chat-template-kwargs '{"reasoning_effort":"xhigh"}'

With image input, add the vision projector:

llama-server -m ThinkingCap-Qwen3.8-27B-abliterated-Q6_K.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.8-27B-abliterated-f16.gguf \
  -ngl 99 -c 32768 -fa on --jinja --reasoning-format deepseek

Notes:

• MTP support in llama.cpp is recent and flag names have changed between versions. Check llama-server --help in your build. At the time of writing, MTP could not be combined with --mmproj or with -np above 1, so run vision and speculative decoding as separate server instances.
• reasoning_effort accepts xhigh (default, recommended), medium or low.
• Sampling follows Qwen3.8's recommendations, which ThinkingCap uses unchanged: thinking mode temp 1.0, top_p 0.95, top_k 20, min_p 0; non-thinking mode temp 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.
• KV cache: only the 16 full-attention layers have one, so -ctk q8_0 -ctv q8_0 saves less memory here than on a dense 27B. Leave it at the default unless you're running long contexts.
• Other apps (LM Studio, Ollama, etc.) work if they bundle a llama.cpp recent enough to support the Qwen3.8 (qwen3_5) architecture.

How it was made

python convert_hf_to_gguf.py <bf16-dir> --outtype bf16 --outfile model-BF16.gguf
python convert_hf_to_gguf.py <bf16-dir> --mmproj --outtype f16 --outfile mmproj-f16.gguf
llama-imatrix -m model-BF16.gguf -f calibration_datav3.txt -o imatrix.gguf -ngl 99 -c 512 --chunks 200
llama-quantize --imatrix imatrix.gguf \
  --tensor-type 'blk\.64\..*=q8_0' --tensor-type 'blk\.[0-9]+\.(attn_qkv|attn_gate|ssm_).*=q8_0' \
  model-BF16.gguf model-Q4_K_M.gguf Q4_K_M

The DeltaNet pin is dropped for Q5_K_S and above (those types keep DeltaNet at their own precision); the embedding/output pin applies only below Q4. Calibration text: bartowski's calibration_datav3.

Limitations

Everything from the bf16 card applies: no safety filter, 6/100 residual refusals on the bf16 model, thinking mode not separately evaluated. You are the safety layer. Lower quants add the usual degradation on long multi-step reasoning and code. If a task works on Q8_0 and fails on IQ4_XS, the cause is the quantization.

License

PolyForm Small Business License 1.0.0 + BottleCap personal-use grant, inherited from ThinkingCap (see LICENSE). Upstream Qwen materials and the abliteration adapter are Apache-2.0 (see NOTICE). Commercial use beyond the PolyForm terms: contact BottleCap AI.

Credits

bottlecapai (ThinkingCap) · MuXodious (abliteration adapter) · p-e-w/heretic · ggml-org/llama.cpp · bartowski (imatrix calibration data) · Qwen team