Qwen3.6-27B-MTP-Custom-IQ6_K

Overview

This repository provides a highly optimized, custom-quantized GGUF model of Qwen3.6 27B, specifically engineered for local deployment on Dual-GPU setups (e.g., 2x RTX 3090 24GB). The primary research objective of this quantization is to achieve an extreme context length (up to 200K tokens in F16 KV Cache) while maximizing inference speed through Multi-Token Prediction (MTP / Self-Speculative Decoding) and retaining most of the original model's capacities. To achieve this, the base network was quantized to IQ6_K using a custom iMatrix, while the critical NextN layers and embeddings were strictly preserved in Q8_0. The base model used for this requantization is unsloth/Qwen3.6-27B-GGUF-MTP.

Engine Compatibility

Important: This model has been specifically designed for and tested on ik_llama.cpp. Standard llama.cpp releases currently only support the specific handling of MTP layers through one specific branch (see Unsloth's official repo) but ik_llama.cpp brings combined with mixed custom quantizations and a better splitting for multi-GPU setups. Using ik_llama.cpp is highly recommended to achieve the expected draft acceptance rates and multi-GPU VRAM distribution and might be necessary due to the custom quantization used here.

Research & Methodology

Preserving Speculative Decoding (MTP)

Most standard quantization pipelines compress the entire model, which severely degrades the NextN layers responsible for Multi-Token Prediction or completely suppress it. This results in poor speculative decoding acceptance rates. During the quantization process, the following custom layer overriding was applied:

--custom-q "blk\\.64\\.nextn\\..*=q8_0"

This forces the speculative heads to remain in high precision. In practical testing, this setup achieved a Draft Acceptance Rate of ~89.9% (438 accepted / 487 generated), effectively doubling the generation speed (~35 tokens/s) on a heavy ~200K token context load.

Chat Template Optimization

The original Qwen3.6 chat template contains known formatting bugs that can affect inference behavior and lacks the Developer role. To resolve this, the previous custom GGUF I built was manually patched using the Hugging Face tool CISCai/gguf-editor but this new one is directly quantized from Unsloth's MTP ready BF16 model. It therefore integrate directly the Unsloth corrected chat_template into the GGUF metadata, ensuring stable and correct out-of-the-box formatting without the need for manual template overrides.

iMatrix Calibration

The model was calibrated using a custom, shuffled iMatrix to ensure high fidelity across coding, instruction-following, and bilingual tasks (English/French). The dataset was built by merging and shuffling the following subsets from eaddario/imatrix-calibration:

• code_small
• tools_small
• text_en_small
• text_fr_small

Recommended Usage

To replicate the optimal performance (200K context, F16 Cache, Multi-GPU) using ik_llama.cpp, use the following llama-server command. Note the specific use of --split-mode graph and --tensor-split 3,2 for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across as I do not own an NVLink at the moment.

/path/to/ik_llama.cpp/build/bin/llama-server \
    -m /path/to/qwen3.6-27B-MTP-Custom-IQ6_K.gguf \
    --mmproj /path/to/mmproj-F16.gguf \
    --split-mode graph \
    --tensor-split 3,2 \
    --max-gpu 2 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 200231 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v f16 \
    --context-shift on \
    --flash-attn on \
    -b 2048 -ub 2048 \
    -amb 512 -rtr -sas -smgs -muge \
    -mtp --draft-max 4 --draft-p-min 0.70 \
    --graph-reduce-type f16 \
    --cache-ram 16384 \
    --cache-ram-similarity 0.85 \
    --cache-ram-n-min 2048 \
    --parallel-tool-calls \
    --recurrent-ckpt-mode per-step \
    --image-min-tokens 1024 \
    --mtmd-kq-type f16 \
    --alias Qwen3.6-27b \
    --jinja

If I may add, I also developed a proxy to enable users to select thinking or non-thinking behaviors and applied the recommended sampling parameters AND the "Preserve Thinking" option. You may find it on my GitHub.

Hardware Requirements

• Target VRAM: 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
• RAM: Minimum 32GB system RAM (Prompt caching and system overhead).
• Context limit: The command above loads ~13GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing --ctx-size or using 8-bit cache (--cache-type-k q8_0 --cache-type-v q8_0).

Acknowledgments

This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:

• Radamanthys11: For providing the high-quality base Qwen3.6-27B-MTP-Q8_0-GGUF model used as the foundation on my original requantizations. This model was not used anymore for this new repo.
• eaddario: For the extremely diverse imatrix-calibration dataset, which was crucial in building the custom, multilingual, and code-heavy iMatrix.
• Unsloth: For identifying the formatting bugs in the original model and providing the optimized, bug-free chat template and publishing the base BF16 MTP-ready model used on this project.
• ikawrakow: For the ik_llama.cpp fork, whose advanced graph-splitting and speculative decoding capabilities made running this extreme context on dual GPUs a reality.
• The Qwen Team: For researching and releasing the exceptional Qwen3.6 architecture.