audreyt/Hy3-Heretic-GGUF

Tencent-Hy3-Heretic 80-Layer Mixed GGUF

This is a mixed-quantization GGUF of the trohrbaugh/Hy3-heretic model, optimized to fit in a 128 GB RAM machine while keeping the last six routed-expert layers at higher precision.

This variant is a decensored/abliterated version of the Tencent-Hy3 80-layer model, created using the Heretic v1.2.0+custom method.

File

• hy3-heretic-80layer-Layers74-79Q4KExperts-mixed.gguf (~89.8 GiB, 96,456,123,360 bytes)

Quantization recipe

• Layers 1–73: IQ2_XXS × 2 + Q2_K × 1
• Layers 74–79: Q4_K × 3

This mirrors the "Layers37-42Q4KExperts" mixed-quant style used for DeepSeek-V4-Flash, but applied to Hy3’s 80 layers with the last 6 routed MoE layers boosted to Q4_K.

Generation

The GGUF is built for the ds4 runtime. The matching source-code changes are in PR antirez/ds4#523 (branch audreyt/ds4:hy3).

On an M5 Max 128 GB with --ctx 2048:

• Decode speed: ~12.4 tokens/s
• Token execute time: ~81–84 ms

Set DS4_METAL_DISABLE_HY3_SINGLE_TOKEN_DECODE=1 to fall back to the legacy batch path.

Notes

• The model is the 80-layer Hy3 architecture, not DeepSeek-V4-Flash.
• It does not contain mtp/draft tensors, so speculative decoding is not available.
• Absolute speed parity with the shallower 43-layer Flash model is not expected because of the deeper stack and larger active weight per token.

How it was made

• Base: hy3-heretic-80layer.gguf (IQ2_XXS/Q2_K)
• Donor layers: hy3-heretic-80layer-q4k.gguf (Q4_K)
• Splice: gguf-tools/mixed/splice_mixed_expert_layers_gguf.py selecting layers 74–79