Myric/Spark-X2.5-4B-GGUF

Spark-X2.5-4B — GGUF

Full-precision GGUF conversion of XHToken/Spark-X2.5-4B.

⚠️ Requires a patched llama.cpp — upstream will NOT load this

spark2_5 is not in upstream llama.cpp. Loading this file with a stock build fails at the architecture check. Build from the branch that adds it:

git clone -b model/Spark2_5 https://github.com/brywil/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j
./build/bin/llama-cli -m Spark-X2.5-4B-BF16.gguf --ctx-size 32768 -ngl 999 -p "..."

Based on upstream 6d0549831. The branch adds the arch, a converter, and a new BPE pre-tokenizer type; nothing else is changed.

Always pass --ctx-size. The model declares max_position_embeddings: 1048576, and llama.cpp defaults to the trained context — that will try to allocate the full 1 M window.

What's here

file	size	bpw	tensor types
Spark-X2.5-4B-BF16.gguf	7.67 GiB	16.0	reference
Spark-X2.5-4B-Q8_0.gguf	4.08 GiB	8.52	F32 109 · Q8_0 253
Spark-X2.5-4B-Q6_K.gguf	3.15 GiB	6.58	F32 109 · Q6_K 253
Spark-X2.5-4B-Q5_K_M.gguf	2.75 GiB	5.74	F32 109 · Q6_K 37 · Q5_K 216
Spark-X2.5-4B-Q4_K_M.gguf	2.37 GiB	4.95	F32 109 · Q6_K 37 · Q4_K 216
spark-x25-4b.imatrix	4.1 MB	—	importance matrix used for the K-quants

All 362 tensors present in every file, attn_gate F32 in all of them, and zero block-32 substitutions — verified by reading the type histogram out of the shipped bytes rather than trusting the filename.

The bf16 is published deliberately: the arch needed a fork to run at all, so it is the reference artifact to check a quant against.

Which one

Q5_K_M is the default recommendation — 2.75 GiB fits an 8 GB card with room for real context, and 128k of KV costs only another 4.5 GiB. Q4_K_M if you are tight on VRAM. Q8_0 if you want a near-reference artifact without carrying 7.67 GiB.

The imatrix was built from a code-inclusive mixed corpus, 1,000 chunks × 512 tokens = 512k tokens, chosen over a prose-only set because the intended workload is agentic coding.

The model

4.11 B dense, 36 layers, apache-2.0. Interleaved sliding-window attention — 27 sliding (window 512) and 9 full, in a strict sssF pattern. 16 query heads, 4 KV heads, head_dim 256, hidden 2560, GELU FFN, 131,072 vocab, tied embeddings. It is a reasoning model and emits a thinking block.

Long context is cheap here, by design

Only the 9 full-attention layers scale with context; the 27 sliding layers are bounded by their 512 window.

9 layers x 4 KV heads x 256 head_dim x 2 (K+V) x 2 bytes = 36 KiB/token

context	KV (f16)
32k	1.1 GiB
128k	4.5 GiB
256k	9.0 GiB
1M	36 GiB

So 128k of context costs less KV than the weights do. Note this is a memory calculation, not a retrieval claim — long-context quality is unmeasured here.

Notes for anyone quantizing this

• Every dimension is divisible by 256 (hidden 2560, intermediate 10240, vocab 131072, KV dim 1024), so all QK_K types are legal. A file labelled Q4_K_M really will be Q4_K_M — unlike archs with an awkward dimension, where llama-quantize silently substitutes block-32 types.

• Pin attn_gate to F32. g_proj is Linear(2560, 16) — 40,960 params per layer, 1.47 M total, about 5.9 MB. Its sigmoid output multiplies the attention result, so error there compounds along the sequence instead of adding to one layer's output, which is exactly the case where per-tensor weight error under-reports the true cost. At this size the bits are free. The converter on the branch above emits it as F32 — but llama-quantize will still quantize it unasked, because ne0 is 2560 and therefore 256-divisible, so Q5_K is perfectly legal on [2560,16]. Legal is not correct here. Pass it explicitly:

  ./build/bin/llama-quantize --imatrix spark-x25-4b.imatrix --tensor-type attn_gate=f32 \
      Spark-X2.5-4B-BF16.gguf out.gguf Q5_K_M
  

  That is how every quant in this repo was built. Cost: 5.9 MB on a 2.7 GiB file.

Verification

Greedy generation from this file is byte-identical to the HF reference (transformers 4.57.1) over 24 tokens on "The capital of France is Paris. The capital of Japan is" — a span crossing six full-attention and eighteen sliding layers, so both rope configurations and the attention gate are exercised. That check matters more than usual on this arch: it uses two different rope setups selected per layer type (sliding layers rotate 256 of 256 dims at theta 10,000; full-attention layers rotate only 64 of 256 at theta 5,000,000), and applying one rope to every layer yields a model that loads and produces fluent, wrong output with no error anywhere.

Measured

Throughput (GB10 / DGX Spark, unified memory, ~123 GB/s achieved)

arm	decode	prefill	conditions
Q5_K_M	48.3 t/s (p50)	4,482 t/s (p50)	single slot, n=970 decode samples
BF16	18.6 t/s (p50)	2,498 t/s (p50)	4 concurrent slots, per-slot, n=25,724

Do not compute a ratio from those two rows — they were served under different slot counts, and the bf16 figure is per-slot under 4-way batching (~74 t/s aggregate). The clean statement is that decode here is bandwidth-bound: 2.75 GiB of weights at ~123 GB/s predicts ~45 t/s, and Q5_K_M measures 48.3.

Agentic coding, 42 tasks (bf16)

Driven through a real opencode agentic loop with tool calling, graded by a fresh pytest per task — not a multiple-choice benchmark.

suite	tasks	valid	passed	mean output tokens
general	25	22	22/22	6,120
hard	5	3	3/3	6,638
frontier	9	4	4/4	22,340
tier4a	3	3	3/3	5,915

32 of 32 gradeable tasks passed. The 10 excluded tasks were harness limits, not wrong answers: 8 were cut off by a 32,768-token output cap and 2 by the wall clock, with no degeneration detected in any of them. They are reported as void rather than failed because the model was still working when the harness stopped it. Treat 32/32 as a floor on capability, not a measurement of it.

The output-token gradient (6,120 → 6,638 → 22,340) is the more informative number: cost stays flat from general to hard, then rises 3.6× at frontier, which locates this model's ceiling between those two tiers.

Do not use wikitext perplexity to pick a quant on this model

A full ladder on one binary, corpus and config (wiki.test.raw, 137 chunks) came out non-monotonic, with three quants scoring "better" than bf16:

arm	PPL
BF16	24.8258
Q8_0	24.6496
Q6_K	24.1758
Q5_K_M	25.8216
Q4_K_M	24.3353
BF16 (repeat)	24.8258

The repeat pass is identical to four decimals, so the noise floor is zero and those differences are real measurements — they simply are not measuring quantization damage. Two structural reasons: the corpus is English prose against a model whose chat template and 131k vocab indicate Chinese-primary training, and 27 of the 36 layers see only a 512-token window, so 2048-token chunks exercise the long-range path in 9 layers and nothing else in the other 27. An English-prose PPL on this arch measures roughly a quarter of it.

Use KL divergence against the bf16 logits instead — it asks "how far did quantization move the model from its own full-precision self", which is the question that matters:

arm	max KLD	RMS Δp	same top-1
Q8_0	7.25	1.656%	97.18%
Q6_K	10.41	3.300%	93.64%

same top-1 % is the figure worth quoting. Q5_K_M and Q4_K_M are pending and will be added.

Provenance

Quantized with llama-quantize from the model/Spark2_5 branch at 498cbc7e0 (upstream base 6d0549831), x86_64 CUDA build verified to contain the arch before use. Every file was loaded and generated from before publication — the PPL ladder above is that evidence — and Q5_K_M has additionally been driven through the full 42-task agentic suite.

Credit

Model by XHToken. This repo is a format conversion only — no weights were retrained or altered.