KikoCis/DeepHat-V1-7B-GGUF

┌────────────────────────────────────────────────────────────────────┐
│  DeepHat-V1-7B · GGUF                                               │
│  ───────────────────────────────────────────────────────────────   │
│  base   DeepHat/DeepHat-V1-7B   arch  qwen2 (Qwen2.5-Coder-7B ft)   │
│  domain cybersecurity · devops · code                              │
│  ladder Q3_K_M → Q8_0 + IQ4_XS   imatrix  ✓ (code+general)          │
│  fidelity  Q8 KLD 0.0019 · top-1 99.8% vs BF16                      │
└────────────────────────────────────────────────────────────────────┘

DeepHat-V1-7B — GGUF quant ladder

A full, imatrix-calibrated GGUF ladder of DeepHat/DeepHat-V1-7B — a Qwen2.5-Coder-7B fine-tune focused on cybersecurity, devops and code. This is a faithful re-quantization (all credit for the model goes to the DeepHat team); what's added here is the imatrix ladder, an objective KLD fidelity table vs the BF16 reference, and an honest evaluation.

First-mover note: at pack time no GGUF of this model existed. Quants Q3_K_M → Q8_0 + IQ4_XS, each with an importance matrix so the low-bit tiers stay sharp.

✅ Recommended files

Use case	File	Why
Best all-round (16 GB RAM)	deephat-v1-7b-Q4_K_M.gguf	safe default, 98.97% top-1 vs BF16
Quality-first (24 GB+)	deephat-v1-7b-Q6_K.gguf	near-lossless, 99.55% top-1
Smallest usable (8–12 GB)	deephat-v1-7b-Q3_K_M.gguf	still 98.0% top-1
Compact + sharp	deephat-v1-7b-IQ4_XS.gguf	imatrix IQ, 4.2 GB
Archival / eval	deephat-v1-7b-Q8_0.gguf	effectively lossless (KLD 0.0019)

📦 Files

Quant	Bits (BPW)	Size
Q3_K_M	3-bit K	3.81 GB
IQ4_XS	4-bit IQ	4.22 GB
Q4_K_M	4-bit K	4.68 GB
Q5_K_M	5-bit K	5.44 GB
Q6_K	6-bit K	6.25 GB
Q8_0	8.50	8.10 GB

📊 Metrics — fidelity vs BF16 reference

Every tier measured against the unquantized BF16 GGUF (KL-divergence + PPL ratio + Top-1 agreement, general-English eval, c=2048).

Quant	PPL(Q)/PPL(bf16)	Max KLD	KLD p99	Top-1 match
Q8_0	1.000004	0.0019	0.0005	99.82%
Q6_K	1.000340	0.0340	0.0079	99.55%
Q5_K_M	1.000881	0.0624	0.018	99.50%
Q4_K_M	0.998457	0.2078	0.030	98.97%
IQ4_XS	0.996500	0.4858	0.041	98.74%
Q3_K_M	1.002918	0.9454	0.109	98.02%

The quantization is clean top-to-bottom — Q8 is effectively lossless and even Q3_K_M keeps 98% argmax agreement with BF16.

🧮 Will it fit?

RAM / VRAM	Comfortable pick
8 GB	Q3_K_M (short context)
12 GB	Q4_K_M
16 GB	Q5_K_M / Q6_K
24 GB+	Q8_0, long context

7B at Q4–Q6 runs fast on a laptop GPU (Apple Silicon / consumer NVIDIA) or CPU.

🚀 How to run it

llama.cpp

llama-cli -m deephat-v1-7b-Q4_K_M.gguf -ngl 99 -c 8192 \
  -p "<|im_start|>user\nExplain how a SQL injection works and how to prevent it.<|im_end|>\n<|im_start|>assistant\n"

Ollama (a ready Modelfile with configurable context ships in this repo):

ollama create deephat -f deephat-8192ctx.Modelfile
ollama run deephat "Write a bash script that scans a subnet for open port 22."

Sampling: ChatML template, temperature 0.1–0.7, top_p 0.9, top_k 20, stop <|im_end|>.

⚠️ Good to know

• Chat template = upstream original (tool-calling preserved). These GGUFs embed DeepHat's original Qwen tool-calling template (the XLAM/Qwen tools format), so native function-calling works just like the source model. That template uses the Jinja tojson filter, which older ollama / llama.cpp builds cannot parse (they error "Unknown (built-in) filter 'tojson'"). If you hit that: update your runtime (recent ollama/llama.cpp handle it), or override the template with the plain-ChatML scripts/chatml.jinja shipped here (--chat-template in llama.cpp) — that variant loads everywhere but drops native tool-calling.
• Domain model. DeepHat is tuned for security / devops / code, not general chit-chat or general software engineering. See the eval below.

🧪 Evaluation methodology

• Fidelity gate (passed): the KLD/PPL/Top-1 table above — every tier is faithful to BF16 (Q8 lossless).
• Agentic probe (reported honestly): swe-mix — 30 SWE-bench Verified instances (6 continuity anchors + 24 discriminating, 10 repos; spec in swe_mix.json), run through Claude Code + agent-bridge.js → Ollama serving Q6_K, tool-call format openhands, temperature 0.1, in Docker. Result: resolved = 0 / 30. The model did emit tool calls on 22/30 instances but ran very short episodes (2–8 messages) — it doesn't sustain the long multi-step agentic loop a SWE-bench solve needs. Honest caveat: (1) this is a 7B — small models routinely score 0–2 on SWE-bench Verified in a local agentic harness; (2) SWE-bench is general-repo software engineering (Django/astropy/sympy), which is not DeepHat's domain (cybersecurity/devops). This number measures general agentic SWE ability, not the model's security/code strengths, and it is not comparable 1:1 to the official leaderboard. Date: 2026-07-12. Small/hard probe, relative signal only.

This repo is fidelity-gated (like a non-general-SWE model release): the quant quality is proven; the SWE number is published transparently rather than hidden.

🔁 Provenance & reproducibility

• scripts/reproduce.sh — exact convert → template-fix → imatrix → quantize commands.
• scripts/chatml.jinja — the clean chat template that replaced the upstream tojson one.
• imatrix corpus: ~793 KB, general text + Python/Rust/C code (386 chunks, c=512).
• reports/artifact-sha256sums.txt — SHA-256 of every GGUF.
• metrics/quant-summary-with-kld.{json,csv} — the table above, machine-readable.
• swe_mix.json — the 30-instance probe spec.

🗒️ Changelog

• 2026-07-12 — initial release. Ladder Q3_K_M→Q8_0 + IQ4_XS, imatrix, KLD sweep, cleaned chat template, honest SWE-mix eval.

Credit

Model: DeepHat/DeepHat-V1-7B (Apache-2.0). This repo only provides GGUF conversions + fidelity measurements. All model capability is the DeepHat team's work.