OsaurusAI/Raptor-0.6.1-preview-GGUF

Raptor 0.6.1 preview — GGUF

imatrix-calibrated GGUF builds of Raptor 0.6.1 preview, a lightly tuned Spark-X2.5-4B.

file	size	mean KLD vs bf16 ↓	same top-1 ↑
Raptor-0.6.1-preview-Q6_K.gguf	3.38 GB	0.00469 ± 0.00033	98.27 ± 0.08 %
Raptor-0.6.1-preview-Q4_K_M.gguf	2.60 GB	0.03994 ± 0.00126	94.92 ± 0.14 %
raptor-0.6.1-preview.imatrix.gguf	3.6 MB	the importance matrix, for re-quantizing yourself

Both quants were produced with the importance matrix applied. Measured against the model's own bf16 weights on a held-out corpus (100 chunks x 512 tokens) that is disjoint from the imatrix corpus — 9,166 overlapping rows were dropped and the eval asserts the result is non-empty rather than trusting the paths.

Reference perplexity of the bf16 source on that held-out text: 2.1935 ± 0.0129.

Requires a llama.cpp with spark2_5 support

spark2_5 is a new architecture. It is not in upstream llama.cpp — there is no LLM_ARCH_SPARK2_5 in any upstream release — so stock llama.cpp, Ollama and LM Studio will refuse these files until they carry the architecture. The graph needs the fused q_k_v_proj, the per-head sigmoid attention output gate (g_proj), a gelu MLP, plain RMSNorm with no +1 shift, 27 sliding / 9 full attention at window 512, and a different rope per layer type (full: theta 5e6 over 64 of 256 dims; sliding: theta 1e4 over all 256).

This is not an Osaurus bundle. Osaurus serves MLX, not GGUF. For Osaurus use OsaurusAI/Raptor-0.6.1-preview-JANG_6M instead. These files are for llama.cpp-family runtimes.

Serving contract — verified on the shipped Q4_K_M, not assumed

Run llama-server with --jinja so the embedded chat template drives tool and reasoning parsing.

• Sampling defaults ship inside the GGUF: general.sampling.temp = 1.0, general.sampling.top_p = 0.95, general.sampling.top_k = -1. These match the source generation_config.json exactly. No repetition penalty.
• Token ids: bos 0, eos 1, unk 5, pad 2; add_bos_token and add_eos_token both false, because the template emits the sentence markers itself.
• Reasoning is ON by default — the template force-opens the rail, so the generation prompt ends with <|Bot|><think>. <think> and </think> are control tokens (ids 3 and 4). With --jinja, llama.cpp separates the trace into reasoning_content, leaving content clean:

reasoning_content: 'We are asked: "What is 84 * 3 / 2?" ...'
content:           'The expression \(84 \times 3 \div 2\) is evaluated left to right...'

Pass "chat_template_kwargs": {"enable_thinking": false} to close the rail; reasoning_content then comes back empty. A reasoning parser that waits for a literal <think> in the output stream will never see one — the opener is in the prompt.

• Tool calls parse to standard OpenAI JSON. The model emits the Spark XML dialect (<tool_call>NAME<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>) and llama.cpp converts it:

{"finish_reason":"tool_calls",
 "message":{"tool_calls":[{"type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Seoul\"}"}}]}}

• Tools render into the first system block. Adding or removing a tool rewrites the system prefix, so a tool-set change is a full re-prefill, not a suffix append — prefix-cache accounting must treat it that way.
• Reasoning can need more than 8192 output tokens on hard prompts. Budget real input plus output inside the context window.

What the calibration did, and what it did not

The imatrix was captured over 600 chunks x 512 tokens of code, agentic/tool-call transcripts, academic multiple-choice, general chat, Chinese, long-context, science and security text.

Three arms were built at each width and measured against the same bf16 reference, so the effect of each treatment is a controlled comparison rather than an assumption:

arm	Q6_K mean KLD	Q4_K_M mean KLD
plain + imatrix (shipped)	0.004691 ± 0.00033	0.039944 ± 0.00126
plain, no imatrix	0.004745 ± 0.00025	0.043528 ± 0.00137
AWQ-folded + imatrix	0.004985 ± 0.00039	0.042827 ± 0.00143

• imatrix earns its place at Q4_K_M (0.0399 vs 0.0435, an 8% reduction) and is within noise at Q6_K, where the format is already near-lossless.
• An AWQ fold was built, measured and dropped. Activation-aware scaling was captured on GPU (1,500,201 tokens, 72/72 fold sites, verified function-preserving to 3.6e-07 and to top-1 35/35 on real logits) and folded into the bf16 before conversion. It made the k-quants worse at both widths — it is redundant on top of a k-quant that already does per-super-block scaling with importance weighting, and its channel rescale widens the dynamic range inside a block. Recorded here because a negative result measured with a control is worth more than an untested claim on the card.

Known limits

Raptor 0.6.1 preview is a practical checkpoint with documented rough edges — long-context recall is not guaranteed, ordinary agentic mistakes remain, and identity responses are not enforced. The full list, plus the mid-conversation tool-call measurement, is on the JANG_6M card and applies to these weights unchanged.

Provenance

Quantized by Jinho Jang (eric@osaurus.ai). Base model XHToken/Spark-X2.5-4B, apache-2.0. Built from the same merged bf16 checkpoint as the JANG_6M bundle.