Qwythos-9B-v2-PerfectSplit-GGUF

A mixed-precision GGUF quantization of Qwythos-9B-v2 using a "Perfect Split" strategy. The model's 427 tensors are selectively quantized: critical reasoning pathways (attention, DeltaNet/SSM, embeddings, output head) stay at full precision, while the redundant MLP layers are compressed.

🧬 Architecture Context

Qwythos-9B-v2 is a hybrid architecture combining:

• Standard attention layers (Q/K/V/O projections with QK-norm)
• DeltaNet/SSM recurrence layers (alpha, beta, dt, conv1d, ssm_out gates)
• Standard MLP blocks (gate, up, down projections with 3x expansion ratio)

This hybrid design means different tensor classes have different sensitivity to quantization:

• Attention & SSM gates control information routing and long-range memory. Errors here compound across the 1M-token context window.
• MLP layers contain massive mathematical redundancy due to the 3x expansion ratio. They tolerate aggressive quantization with minimal quality impact.

🎯 The "Perfect Split" Strategy

Tensor Class	Precision	Count	Rationale
Token Embeddings (token_embd.weight)	F16	1	First point of contact with input; errors propagate everywhere
Output Head (output.weight)	F16	1	Maps hidden states → vocabulary logits; critical for token selection
Output Norm (output_norm.weight)	F32	1	Final normalization before logits
Attention Projections (attn_q, attn_k, attn_v, attn_output, attn_qkv, attn_gate)	F16	~96	Prevents attention score drift at long context
Attention Norms (attn_norm, attn_q_norm, attn_k_norm, post_attention_norm)	F32	~64	Activation scaling stability
DeltaNet/SSM (ssm_alpha, ssm_beta, ssm_dt, ssm_conv1d, ssm_out, ssm_a)	F16	~96	The recurrence S_t = α·S_{t-1} + β·(k⊗v) runs millions of steps; errors accumulate multiplicatively
SSM Norms (ssm_norm)	F32	~32	Recurrence stability
MLP Layers (ffn_gate, ffn_up, ffn_down)	Q5_K_M	96	Compressed — high redundancy, low sensitivity

Final Statistics

Metric	Value
Original Size (BF16)	16.69 GB
Final Size	10.78 GB
Compression	35.4% smaller
Average BPW	10.34
F16 Tensors	331
Q5_K_M Tensors	96
Total Tensors	427
Layers	32 (0 to 31) — all intact

Per-Tensor Compression Example (from quantization log)

blk.29.attn_q.weight : bf16 → f16 64.00 MiB → 64.00 MiB (preserved) blk.29.ssm_out.weight : bf16 → f16 32.00 MiB → 32.00 MiB (preserved) blk.29.ffn_gate.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%) blk.29.ffn_up.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%) blk.29.ffn_down.weight : bf16 → q5_K 96.00 MiB → 33.00 MiB (compressed 65%)

📊 Expected Quality Profile

Because the quantization is surgical (only compressing MLP layers), the quality profile differs from uniform quants:

Capability	Expected Quality	Why
Chain-of-Thought reasoning	Excellent	Attention + SSM pathways at F16 preserve reasoning flow
Long-context recall (128K+)	Excellent	DeltaNet recurrence at F16 prevents drift
Code generation	Very Good	MLP compression has minimal impact on coding patterns
Factual knowledge recall	Very Good	Slight loss on obscure facts (MLP stores knowledge)
Tool-call / JSON output	Very Good	Output head at F16 preserves format fidelity
Overall perplexity	~0.05 above BF16	vs. ~0.18 above BF16 for uniform Q5_K_M

Trade-off: This model is 4 GB larger than a uniform Q5_K_M (6.5 GB), but delivers noticeably better quality on reasoning-heavy tasks.

🚀 Usage

With llama.cpp

./llama-cli \
  -m Qwythos-9B-v2-PerfectSplit.gguf \
  --prompt "You are an expert Python developer. Solve this step by step inside  tags.\n\nTask: Implement a thread-safe sliding-window rate limiter." \
  --n-predict 2000 \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.05 \
  --repeat-penalty 1.05 \
  --ctx-size 8192 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn
With LM Studio
Import the GGUF file
Recommended settings:
Temperature: 0.6
Top-K: 20
Top-P: 0.95
Min-P: 0.05
Repeat Penalty: 1.05
Context Size: 8192 (or higher if RAM allows)
KV Cache Type: q8_0 (important — keeps total VRAM usage reasonable)
With Ollama
Create a Modelfile:
FROM ./Qwythos-9B-v2-PerfectSplit.gguf
PARAMETER temperature 0.6
PARAMETER top_k 20
PARAMETER top_p 0.95
PARAMETER min_p 0.05
PARAMETER repeat_penalty 1.05
PARAMETER num_ctx 8192

ollama create qwythos-perfect -f Modelfile
ollama run qwythos-perfect