Andgihat/Nanbeige4.2-3B-GGUF

Nanbeige4.2-3B — GGUF (imatrix + KLD-guided dynamic quants)

GGUF quantizations of Nanbeige/Nanbeige4.2-3B — a Looped Transformer (22 physical layers reused num_loops=2 → 44 effective depth; ~4B total / ~3B non-embedding).

These are not naive quants. Each is imatrix-calibrated and uses a per-tensor mixed-precision recipe derived from a measured KL-divergence sensitivity map of this specific model (the same idea as Unsloth Dynamic, built and verified here from scratch).

⚠️ This is the only GGUF of Nanbeige4.2 that exists, and it will not run on stock llama.cpp / Ollama / LM Studio yet — see Compatibility.

Compatibility

Nanbeige4.2 uses a looped architecture not yet in upstream llama.cpp. The GGUF carries general.architecture = nanbeige, understood only by:

• The authors' branch: Nanbeige/llama.cpp @ nanbeige42
• Our fork with the arch ported in + prebuilt Windows CUDA (sm_120) binaries: BeeLlama.cpp @ nanbeige-arch (release) — includes KVarN, so the native 256K context fits on 16 GB (-ctk kvarn3 -ctv kvarn3).

Unmodified upstream fails with unknown architecture 'nanbeige'.

Quantizations

Quality measured as KL-divergence vs the BF16 model and Same-top-p (how often the quant's argmax token matches BF16) on wiki.test.raw. Lower KLD / higher top-p = closer to full precision.

File	Size	KL-div ↓	Same top-p ↑	Pick if
Nanbeige4.2-3B-UD-Q4_K_XL.gguf	2.62 GB	0.137	84.4 %	smallest, VRAM-tight
Nanbeige4.2-3B-UD-Q6_K.gguf	3.32 GB	0.048	90.4 %	balanced
Nanbeige4.2-3B-UD-Q6_K_XL.gguf	3.67 GB	0.024	92.9 %	near-lossless
Nanbeige4.2-3B-Q8_0.gguf	4.43 GB	0.009	95.4 %	max fidelity

Note: absolute KLD is small but the tiers are genuinely separated — plain perplexity hides this. For an agent/reasoning model where exact tokens matter (tool-calls, code, math), prefer UD-Q6_K or higher.

Benchmark — GSM8K recovery vs BF16

100 GSM8K test questions, greedy, reasoning enabled. Recovery = accuracy ÷ BF16 accuracy. Agreement = share of questions where the quant's final answer matches the BF16 model's.

Model	Accuracy	Recovery	Agreement w/ BF16
BF16 (reference)	93 %	100 %	100 %
UD-Q4_K_XL	92 %	98.9 %	88 %
UD-Q6_K	90 %	96.8 %	91 %
Q8_0	89 %	95.7 %	94 %

All quants recover ~96–99 % of BF16 accuracy — near-lossless in practice; even the smallest UD-Q4_K_XL loses 1 question in 100. At n=100 the absolute-accuracy differences are within statistical noise (±3 pts) and do not rank the quants. The Agreement column is the meaningful signal: it is monotonic (Q8 > Q6 > Q4), matching the KLD ranking — higher precision deviates less from full precision.

Methodology

1. imatrix via llama-imatrix over bartowski calibration_datav3 (Dampf & Kalomaze). Included as nanbeige42-imatrix-v3.dat so you can roll your own quants. (A larger multilingual corpus was tested and gave no KLD improvement — weight importance here is largely language-independent.)
2. KLD sensitivity sweep: starting from a uniform Q4_K base, each tensor category was bumped to Q6_K one at a time and the KLD gain per MB measured. Key findings for this looped model:
   • attn_v / attn_k are near-free wins (tiny under GQA — 8 KV heads — yet large KLD drop). Always bumped.
   • attn_q is a trap (large tensor, negligible gain) — left at Q4 in the leaner quants.
   • token_embd / output are sensitive (unusual — driven by the concatenated n-gram embeddings) and dominate at the high-precision end (hence the _XL = Q8 embed/output).
3. Bits are then allocated by measured efficiency, not by a fixed template. _XL marks embed/output kept at higher precision (same convention as Unsloth UD).

Running (BeeLlama fork)

llama-server -m Nanbeige4.2-3B-UD-Q6_K.gguf -ngl 99 -c 32768 --jinja

Chat template is embedded. Authors' sampling: agent temp 1.0, reasoning temp 0.6 with long context.

Credits & license

• Base model Nanbeige/Nanbeige4.2-3B, Apache 2.0 — all rights to the original authors.
• Calibration data: bartowski / Dampf / Kalomaze.
• These GGUFs are a quantized derivative, released under the same Apache 2.0. Changes vs original: GGUF conversion + quantization only; weights otherwise unmodified.