OsaurusAI/Diplodocus-preview-GGUF

Diplodocus preview 4, the Osaurus assistant for finance, documents, spreadsheets and tool-assisted work, built on IFM/K2-Horizon-7B and trained on Osaurus' own workspace tools (workbooks and documents edited in place, the built-in calculate tool, reasoning on and off). This is the GGUF build for llama.cpp in three calibrated sizes:

File	Size	Use
Diplodocus-preview-Q4_K_M.gguf	6.33 GB	smallest; 8 GB GPUs / tight RAM
Diplodocus-preview-Q6_K.gguf	7.76 GB	recommended balance
Diplodocus-preview-Q8_0.gguf	9.57 GB	closest to BF16

Using Osaurus on a Mac? Use the MLX bundle OsaurusAI/Diplodocus-preview-JANG_6M instead. Same weights, same template, same reasoning toggle.

Requires llama.cpp with K2 Horizon support (ggml-org/llama.cpp#29535, merged 2026-10-06). Older builds, and apps that bundle an older llama.cpp, will not load the k2-horizon architecture.

What preview 4 adds over preview 3

Trained rounds R8–R11 on top of preview 3: the Osaurus calculate tool (when to call it, chained money arithmetic, recovery from engine errors), workbooks and documents through Osaurus' in-place file_edit operations (bounded file_read windows, mode:"structure", set_cells with live formulas, never regenerating an existing workbook), PDF forms and signed-PDF handling, and a loop fix for reasoning-off answers. The source's own held-out evaluation moved end-to-end workbook/document tasks from 59/51 (on/off, of 100) to 70/70 and calculator-executed long-document money answers from 31 to 43 of 60 with reasoning on.

Quantization

Component	Q4_K_M	Q6_K	Q8_0
Attention projections (q, k, v, o; all 36 layers)	Q8_0	Q8_0	Q8_0
MLP gate / up (all 36 layers)	Q4_K, importance-weighted	Q6_K, importance-weighted	Q8_0
MLP down (all 36 layers)	Q6_K in 18 layers, Q4_K in 18, importance-weighted	Q6_K, importance-weighted	Q8_0
Embeddings	Q4_K	Q6_K	Q8_0
Untied output head	Q6_K	Q6_K	Q8_0
Norms	F32	F32	F32

File	Bytes	sha256
Diplodocus-preview-Q4_K_M.gguf	6,327,738,208	32c4d81cacb4af48c61e852e8bca2d974548900aca337088c14c2869e15311e4
Diplodocus-preview-Q6_K.gguf	7,760,248,672	c81c80cc2f3d5f9fec0eb06d9a8732404ac37516fde2d8692eb49f6e1d39cfd9
Diplodocus-preview-Q8_0.gguf	9,573,973,632	8d39a2d47b56b308d60d79efa194a2f0796571b439b14499cd4847ccacdf8a16

The importance matrix (Diplodocus-preview.imatrix.gguf, included) was computed on the BF16 model over 1,653,067 tokens of Osaurus tool conversations, reasoning-on/off traces and calculator chains, with chat and tool special tokens parsed as the model sees them.

The layouts were chosen by measurement, not by default. Every candidate was scored against the BF16 source on a held-out set disjoint from calibration: 64 rows, 206k tokens, reasoning on and off, 41 with tool calls. Lower KL divergence is better; "top-1 same" is how often the quant picks the same next token as BF16.

Candidate	Size	Mean KLD	Median KLD	Top-1 same
Q8_0 (shipped)	9.57 GB	0.0285	0.0019	95.5%
Q6_K, standard	7.39 GB	0.1325	0.0109	89.7%
Q6_K + importance matrix	7.39 GB	0.1020	0.0076	91.3%
Q6_K + importance matrix + Q8_0 attention (shipped)	7.76 GB	0.0608	0.0044	93.1%
Q6_K shipped + Q8_0 output head	8.01 GB	0.0603	0.0039	93.5%
Q4_K_M + importance matrix	5.59 GB	0.2055	0.0311	85.3%
Q4_K_M + importance matrix + Q8_0 attention (shipped)	6.33 GB	0.1246	0.0182	88.4%
Q4_K_M shipped + Q6_K MLP down	6.56 GB	0.1110	0.0149	89.3%

At both Q6 and Q4, moving the attention projections to 8-bit removed roughly 40% of the divergence for +0.37 / +0.74 GB. This matches the JANG bundle's finding that preview 4's fine-tuning lives mostly in attention. The next steps up (8-bit output head for Q6, 6-bit MLP down for Q4) bought much less per byte and would crowd the next size tier, so they were not shipped. For scale, this model is noise-sensitive: the JANG card measured BF16 with one-ULP embedding noise at 96.2% top-1 on its set, and Q8_0 sits at 95.5% here. These are quantization diagnostics on training-distribution data, not an independent task-accuracy benchmark.

Reasoning, tools and sampling

• High: reasoning on (reasoning_effort: "high", the default). Low: reasoning off, using the embedded template's empty prefilled reasoning block. Do not send medium.
• The embedded chat template is the same template shipped with the JANG bundle. Run llama-server with --jinja so it is used.
• Default sampling is temperature 1.0, top_p 0.95, stamped in the GGUF metadata.
• Tool calls use the native k2_horizon envelope. Upstream llama-server --jinja parses them into OpenAI tool_calls and puts reasoning in reasoning_content.
• For money and exact arithmetic, use reasoning on and let the model drive the calculate tool.

llama-server -hf OsaurusAI/Diplodocus-preview-GGUF:Q6_K --jinja -ngl 99 -c 32768   # or :Q4_K_M / :Q8_0

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "What is 1234.56 * 0.0825, rounded to cents?"}],
  "tools": [{"type": "function", "function": {"name": "calculate", "description": "Evaluate an arithmetic expression exactly.",
             "parameters": {"type": "object", "properties": {"expression": {"type": "string"}}, "required": ["expression"]}}}],
  "chat_template_kwargs": {"reasoning_effort": "high"}
}'

Send "chat_template_kwargs": {"reasoning_effort": "low"} to turn reasoning off.

Validation

Every shipped file was checked in upstream llama-server (llama.cpp a0e6dc738) on an NVIDIA GB10:

• Template and tokens: the rendered prompt carries exactly one BOS; reasoning-off prompts end in the empty prefilled block. Generation stops at <|ifm|im_end|>; <|ifm|endoftext|> also stops.
• Reasoning tags: reasoning-off outputs contain no reasoning tags; reasoning-on outputs close with exactly one </ifm|think>.
• OpenAI API: reasoning on returns reasoning_content plus clean content; reasoning off returns clean content with no reasoning. A calculate request returns finish_reason: tool_calls with parsed arguments, and after the tool result the final answer is the correct $101.85.
• Tool use under the default sampling (temperature 1.0, top_p 0.95, reasoning on, 40 runs over four money prompts; think-tag integrity 40/40 for every file):

	BF16	Q8_0	Q6_K	Q4_K_M
Calculator called	39/40	38/40	38/40	40/40

The differences are within sampling noise.

Known limits

From the source's own evaluation: new workbooks with live formulas sometimes end up with text formulas; change requests on signed PDFs with reasoning on are handled in 3 of 10 cases; money arithmetic with reasoning off is unreliable, so use reasoning on (in our checks, reasoning-off answered a calculator question from memory, and both BF16 and Q6_K got it wrong). The model may identify itself as K2 and may attempt a tool from older conversation history. Use current tool definitions and validate consequential outputs.

한국어 안내

Diplodocus 프리뷰 4의 llama.cpp용 GGUF 모델입니다. Q4_K_M(6.33 GB), Q6_K(7.76 GB, 권장), Q8_0(9.57 GB) 세 가지로 제공되며, Q4_K_M과 Q6_K는 어텐션을 Q8_0으로 두고 나머지를 중요도 행렬로 보정했습니다. K2 Horizon을 지원하는 llama.cpp(#29535 이후)가 필요하며 --jinja로 실행하세요. 추론은 reasoning_effort를 high로 켜거나 low로 끌 수 있고, 기본 샘플링은 temperature 1.0, top_p 0.95입니다. 금액 계산에는 추론을 켜고 calculate 도구를 사용하세요. Mac의 Osaurus에서는 Diplodocus-preview-JANG_6M을 사용하세요.

Quantized and validated by Jinho Jang — eric@osaurus.ai