Tess-4-27B-Mixed-Q5

Overview

This repository provides a highly optimized, custom-quantized GGUF model of Tess-4-27B, specifically engineered for local deployment on Dual RTX 3090 setups. The primary research objective of this quantization is to achieve an extreme context length (full 262K tokens in F16 KV Cache) while maximizing inference speed through adapted BPW and Multi-Token Prediction (MTP / Self-Speculative Decoding) and retaining most of the original model's capacities. To achieve this, the base network was quantized to Q5_0 and Q8_0 using a custom iMatrix, while the critical NextN layers and embeddings were strictly preserved in Q8_0. The base model used for this requantization is migtissera/Tess-4-27B.

Research & Methodology

Selective precision Quantization for high-speed inference

Most standard quantization pipelines compress the entire model, which severely degrades the quality, the inference speeds and the NextN layer responsible for Multi-Token Prediction (which is, sometimes, completely suppress it).

To maintain capacities while optimizing for a Dual RTX 3090 (NVLink) setup, I have implemented a Selective Precision Mapping strategy. By carefully partitioning the model into Q8_0 (High Precision) and Q5_0 (Balanced Efficiency) tensors, it preserves the critical activation flows, specifically the Multi-Token Prediction (NextN) layer, without sacrificing the throughput necessary for processing massive contexts (full 262K tokens).

Strategic Layer Mapping

The following quantization scheme was applied :

--tensor-type 'token_embd\.weight=q8_0'
--tensor-type 'output\.weight=q8_0'
--tensor-type 'blk\.64\..*=q8_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.attn_qkv\.weight=q8_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.attn_q\.weight=q8_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.attn_k\.weight=q8_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.attn_v\.weight=q8_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.attn_output\.weight=q8_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.attn_gate\.weight=q5_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.ssm_alpha\.weight=q5_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.ssm_beta\.weight=q5_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.ssm_out\.weight=q8_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.ffn_up\.weight=q5_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.ffn_down\.weight=q5_0'
--tensor-type 'blk\.([0-9]|[1-5][0-9]|6[0-3])\.ffn_gate\.weight=q5_0'

This scheme keeps the most important layers at high precision (Q8_0) and lowers the GGUF size on disk, enabling full offload on dual RTX 3090 setups. I chose to go for Q5_0 and Q8_0 as it both retains many capacities while lowering the needs for complex mathematical kernel calculations.

iMatrix Calibration

The model was calibrated using a custom, shuffled iMatrix to ensure high fidelity across coding, instruction-following, and bilingual tasks (English/French).

Accuracy Analysis

I compared the perplexity on Wiki-Text-raw to evaluate the precision loss after the mixed-quantization scheme:

Metric	Value
Mean PPL (Q)	6.717543 ± 0.043015
Mean PPL (Base)	6.686070 ± 0.042694
Correlation (ln PPL)	99.84%
Mean ln(PPL(Q)/PPL(Base))	0.004696 ± 0.000367
Mean PPL Ratio (Q/Base)	1.004707 ± 0.000368
Mean ΔPPL (Q − Base)	0.031473 ± 0.002475

For deeper understanding of the true capacities of this quant, I also tested the KL-Divergence :

Metric	Value
Mean KL Divergence	0.006461 ± 0.000262
Maximum	19.347248
99.9th percentile	0.339803
99th percentile	0.054434
95th percentile	0.015984
90th percentile	0.009486
Median	0.002099
10th percentile	0.000056
5th percentile	0.000015
1st percentile	0.000002
0.1th percentile	-0.000003
Minimum	-0.000089

Key Findings:

• Near-Lossless: The perplexity degradation is minimal at +0.0204, indicating that this mixed precision layout preserves the original model's reasoning capabilities.

Recommended Usage

To replicate the optimal performance (200K context, F16 Cache, Multi-GPU) using llama.cpp, use the following llama-server command. Note the specific use of --split-mode tensor and --tensor-split 1,1 for optimal PCIe bandwidth management across dual RTX 3090s. This command appeared to be the best one I could come across using an NVLink.

/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/Tess-4-27B-Mixed-Q5\
    --mmproj /path/to/mmproj-F16.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k f16 \
    --cache-type-v f16 \
    --flash-attn on \
    -b 2048 -ub 2048 \
    --spec-type draft-mtp \
    -sps 0.70 \
    --image-min-tokens 1024 \
    --alias Qwen3.6-27b \
    --jinja

If I may add, I also developed a proxy to enable users to select thinking or non-thinking behaviors and applied the recommended sampling parameters AND the "Preserve Thinking" option. You may find it on my GitHub.

Hardware Requirements

• Target VRAM: 48 GB (Tested on 2x NVIDIA RTX 3090 24GB).
• RAM: Minimum 32GB system RAM (Prompt caching and system overhead).
• Context limit: The command above loads ~13GB of KV cache across the two GPUs. If you experience OOM (Out of Memory) errors, consider reducing --ctx-size or using 8-bit cache (--cache-type-k q8_0 --cache-type-v q8_0).

Acknowledgments

This project was made possible thanks to the outstanding tools and contributions from the open-source AI community. Special thanks to:

• migtissera: For this amazing model they managed to create, going even further than both Alibaba and Jackrong manage to get.
• llama.cpp: Using the new Tensor split mode, it finally achieves extremelly high performances on dual-GPU setups.
• froggeric: For the immense work done on fixing Qwen3.5 and 3.6 chat_template.
• The Qwen Team: For researching and releasing the exceptional Qwen3.6 architecture.