Baekpica/Inkling-Small-Mixed-Quant-GGUF

Inkling-Small — Mixed Quant GGUF

Two independently audited mixed-precision variants of thinkingmachines/Inkling-Small, source revision 8cc5877b44d343f88b92086aa1fb72897950f06a. Choose one variant directory to download. Each contains all of its shards, checksums, tokenizer/processor/config assets, model card and validation reports.

Multimodal serving (2026-09-10) and prefill optimization (2026-09-11): The MQ85GB variant serves text, PNG/JPEG image and 16 kHz WAV audio input with text output through ds4-dfm-rs on one DGX Spark: image requests pass on OpenAI Chat Completions, OpenAI Responses and Anthropic Messages, audio input on Chat Completions, and the separate eight-layer MTP-BF16 sidecar is checked on and off over HTTP. After three further prefill rounds, 8,192-token prefill reaches 312.88 tok/s (+15.4% over 271.21) and 2,048-token prefill 327.86 tok/s (+15.7% over 283.41) against fresh controls for the prior merged runtime: medians of three fresh processes at the default 512-token chunk, 64 greedy output tokens, MTP off. Each comparison checks 1,200,348 logits with zero differences and identical generated IDs; decode medians differ by at most 0.1%. Runtime: ds4-dfm-rs@b189bf6, PR #29.

MQ85GB runtime: ds4-dfm-rs integration and checks cover serial CUDA serving on DGX Spark with text/image/audio input and text output, plus the separate eight-layer MTP-BF16 sidecar. Short HTTP checks use context 1,024; the guide records exact API/media limits and MTP on/off evidence. The MQ85GB performance report records 8192- and 2048-token prefill and 64-token decode with MTP off, fresh-process samples and exact full-logit/token parity against the retained ds4 control paths. MQ89, llama.cpp compatibility, long-context serving and MTP speedup remain unqualified by these checks.

Variant directory	Bytes	Decimal GB	GiB	Shards
MQ89	95,446,595,200	95.446595	88.891569	6
MQ85GB	85,732,587,168	85.732587	79.844694	6

GB = 1,000,000,000 bytes; GiB = 1,073,741,824 bytes. MQ89 addresses the original 84–89 GiB weight budget. MQ85GB addresses the additional 84–86 GB budget, saving 9.714008 GB for additional runtime headroom. MTP is separate and excluded.

Support my work

I work on making large language models practical on hardware they were never really designed to fit on — through mixed quantization, inference optimization, custom kernels, and serving experiments.

While much of the development happens on local hardware, calibration, profiling, and large-scale validation often require expensive on-demand GPUs.

Contributions help pay for that compute, storage, and testing infrastructure, so I can keep experimenting and publishing the results openly.

Download one variant

# MQ89
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ89/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ89 && sha256sum -c SHA256SUMS)

# MQ85GB
hf download Baekpica/Inkling-Small-Mixed-Quant-GGUF --include 'MQ85GB/*' --local-dir ./Inkling-Small
(cd Inkling-Small/MQ85GB && sha256sum -c SHA256SUMS)

Both retain all 888 main tensors, all 42 decoder layers, 256 experts in every sparse layer, and complete native BF16 vision/audio ingress and attention. general.architecture=inkling, inkling.tensor_layout=source-interleaved-v1, GGUF quantization version 2. Gate/up interleaving and original tensor names/shapes are unchanged. Read INKLING-CONTRACT.md before native work.

Precision differences

Group	MQ89	MQ85GB
Attention, vision/audio	Original BF16	Original BF16
L2 routed experts	BF16	Q8_0
Embedding/head, dense L0–1, shared experts	Q8_0	Q8_0
L3–39 routed gate/up	IQ2_XXS	IQ2_XXS
L3–39 routed down	Q2_K	IQ2_XS
L40 routed gate/up	Q4_K	Q3_K
L40 down and all L41 routed matrices	Q4_K	Q4_K
Router/norm/bias/scale/convolution/relative controls	Native	Native

MQ85GB changes 40 precisions and preserves the other 848 tensor payloads bit-exactly. Its L2 Q8 payloads also match the separately published Q8 reference. All native payloads match original BF16/F32 source bytes. L2 Q8 is the deliberate large capacity tradeoff; this variant does not retain that official NVFP4 BF16 exclusion. Detailed audits and results are in each variant directory.

Calibration and fidelity

Both use the same measured multimodal imatrix: 1,663 exact processed inputs / 1,578,255 valid decoder tokens. The balanced primary pass has importance weight 1.0; a real structured reasoning/tool coverage supplement has weight 0.1. Effective text/code/image/audio shares are 45.215/21.433/23.817/9.535 percent. The public calibration dataset contains the exact inputs and complete source lineage/weighting details.

All L3–39 target experts were observed. L41 expert 123 remained unobserved; its ordinary Q4_K no-imatrix payloads are identical in both variants. No missing importance vector is fabricated. Shared calibration/ contains NPZ/DAT importance and coverage. reproduction/ and provenance/ record code, compiler/backend context and precise pins.

Offline fidelity suite	MQ89 mean KL vs BF16	MQ85GB mean KL vs BF16	MQ89 top-1 agreement	MQ85GB top-1 agreement
short	0.309541	0.382427	0.824	0.784
extended	0.133307	0.146076	0.918	0.920

These are small teacher-forced diagnostics (including real document/speech), not task benchmarks or guarantees of equivalent quality. Individual probes, direct variant comparisons and 32-token-budget generation checks are included. Extended media were excluded from calibration; text rows were excluded from the structured supplement, but overlap with historical primary text pools cannot be fully ruled out. Offline GGUF weights were dequantized to BF16 on B300 GPUs, so those timings/memory figures are not native quantized serving data.

The weight budget leaves room for caches and activations but does not prove that a particular context fits on GB10. Seven global-attention layers alone need about 28 GiB BF16 KV at 1,048,576 tokens, before local state, activations and OS memory. Actual admission and serving measurements remain necessary.

Related artifacts

• Q8 reference and optional MTP-BF16.
• Exact calibration dataset.

The earlier MQ89 root-level paths remain available at historical revision f091d3672dc69a1af77cad92b63cdeddc145a039; current main uses variant directories. The MQ89 weight bytes and hashes did not change during this organization. Source model is Apache-2.0; dataset sources retain their respective licenses.