Inkling-975B-Alis-MLX-Dynamic-2.7bpw

The capacity / size-optimal tier of the Inkling · Alis MLX Dynamic family — siblings: the ~3.71 bpw quality build (single 512 GB box, golden spot) and the ~6.6 bpw two-box Q6 performance build. Full family: Inkling 975B · Alis MLX Dynamic collection.

Apple Silicon (MLX) mixed-precision quantization of thinkingmachines/Inkling — a 975B-class multimodal Mixture-of-Experts model (66 hybrid decoder layers, 256 routed experts (top-6) + 2 shared per MoE layer, hidden 6144, sliding-window attention + short-convolution hybrid, vision + audio front-ends, 201K vocab).

This build answers a different question than its quality sibling: how small can 975B get while every design-critical path keeps real precision? ~2.72 bits/weight, 299 GiB on disk — a 975B-class multimodal MoE that fits a 512 GB Mac with ~150 GiB to spare for KV cache and activations, with quantization scales certified by the same fully receipt-sealed, layer-local ALIS-DWQ pass as the quality tier.

---

Capacity-first recipe

A 2-bit floor for the expert bulk, with targeted promotions exactly where damage concentrates: late layers, early down-projections, shared experts, and every control path. Nothing here is uniform-2-bit — the recipe spends its bits dynamically.

Component	Precision	Why
Routed experts w13/w2, layers 3–55	2-bit affine, g128	the parameter bulk at the floor
Routed experts, layers 56–65	3-bit, g128	late layers shape the output distribution
Early down-proj w2, layers 3–7	3-bit, g128	down-projections compound early errors
Shared experts (all layers)	4-bit, g64	on every token, fine-grained groups
Attention (all 5 proj., hybrid + sconv)	6-bit, g128	every token's critical path, kept high
Dense MLP (layers 0–1)	6-bit, g128	early-layer protection
Embeddings · LM head	8-bit, g128	distribution-sensitive
Layer 2 (entirely) · router/gates · norms	BF16	control flow stays exact
Vision + audio towers	BF16	modality front-ends kept exact

MTP head dropped. The base model's multi-token-prediction head (model.mtp.*, 5.26B params) is a speculative-decoding accessory the MLX runtime does not use; it is dropped at conversion. That is why all three tiers report 947.1B logical params from a 952.4B source — the tiers quantize an identical parameter set.

Base model	thinkingmachines/Inkling (975B-class; 952.4B source / 947.1B logical params)
Bits/weight	2.72 effective (321.4 GB total)
On-disk size	321.4 GB (299.3 GiB), 108 shards
Format	MLX safetensors (inkling_mm_model)
Modalities	text + image + audio inputs
Attention	sliding-window (512) hybrid + short conv — KV stays small at long context

---

What makes this build different: a sealed, audited DWQ pass

Most public quants ship converted weights. This one ships converted weights plus a machine-checkable certification trail. The quantization scales/biases were passed through layer-local ALIS-DWQ (alis-dwq) against exact BF16 teacher activations, under a guard-and-receipt harness built for reproducible two-box runs:

• BF16 teacher boundaries, not logits-only: the full-precision teacher ran as a distributed pipeline across two 512 GB Macs (layers 0–32 / 33–65), dumping per-layer input/target activations for 72 calibration batches across text, image, and audio — every decoder layer trains against its own exact teacher boundary, per modality.
• Layer-local, memory-bounded training: each of the 66 layers strict-loaded alone and its affine scales/biases tuned with stop-gradient teacher boundaries and valid-token NMSE, using the expert-group × token-block streamed backward (fused-equivalent gradients at a ~23 GiB peak instead of ~150 GiB), inside a watchdog envelope of min 90% system RAM free, zero swap growth.
• Do-no-harm acceptance, per layer: a layer's tuned scales are kept only if held-out boundary NMSE does not regress at all (allowed regression: 0.0); otherwise the layer rolls back to baseline, byte-exact.
• Seven layers committed genuine improvements: 0, 4, 6, 9, 10, 11, 12. At a 2-bit floor there is more room to recover than at 3-bit — the DWQ pass found seven early/mid layers where tuned scales strictly improved held-out boundary error, versus one such layer in the 3.7 bpw sibling. The remaining 59 layers certified neutral and stayed byte-identical to baseline.
• Every step is evidence: each layer ran as a sealed process chain (frozen runtime bundle → launch pin → guarded watchdog → atomic no-clobber receipts); the 66 terminal receipts form a hash chain closed by an advanced-completion receipt (status: pass), and the merged shards were rebuilt with per-tensor byte verification (patched tensors match their sealed patch payloads; all others match baseline exactly).

The result is conservative by construction: provably-not-worse-than-baseline weights with certified improvements where they were found — not a quant that traded unknown regressions for a benchmark bump. (A higher-lr DWQ tier on the same evidence rails is planned as a separate revision.)

---

Sibling builds

Build	bpw	Size	For
quality / golden spot	3.71	409 GiB	single 512 GB Mac, best quality in one box
this — capacity / size-optimal	2.72	299 GiB	512 GB Mac with generous headroom; 320–384 GB boxes
Q6 teacher (two-box)	6.60	728 GiB	maximum fidelity, 2 × 512 GB pipeline serving

---

Usage

The inkling_mm_model architecture is served by the project's MLX port (hybrid sliding-window + short-conv attention, 256-expert gather MoE, multimodal towers); upstream mlx-lm support is not yet merged. Until the port lands upstream, treat this repo as weights + provenance, loadable with the Inkling MLX runtime from the alis pipeline:

# Inkling MLX runtime (alis pipeline port)
from inkling_mlx.model import load_model
model = load_model("avlp12/Inkling-975B-Alis-MLX-Dynamic-2.7bpw")

Sliding-window attention keeps the KV cache small, so long contexts are memory-cheap relative to dense-attention peers. At 299 GiB of weights, a 512 GB box runs this with ~150 GiB of true headroom — and 384 GB machines fit it comfortably.

Hardware

Built for Apple Silicon at 320 GB+ unified memory (M2/M3 Ultra class). On 512 GB machines, consider the 3.7 bpw quality build for maximum single-box quality.

Validation status

The DWQ schedule is complete and certified (66/66 sealed layer receipts + advanced-completion pass; 7 accepted improvements, 59 certified-neutral).

Runtime verification (measured, this exact artifact):

• Strict load passes on a 512 GB M3 Ultra: exact-set index check over all 108 shards, mixed-quantization config applied, load_weights(strict=True) — no missing/unexpected tensors.
• Greedy generation smoke passes end-to-end (sliding-window hybrid attention + 256-expert gather path).
• End-to-end text NLL, candidate vs. baseline conversion, teacher-forced on all 32 held-out calibration text batches (5,926 scored tokens): candidate 3.2035 vs baseline 3.2027 — parity within noise (Δ +0.0008 nats, ≈0.03%), candidate better on 19/32 batches. Exactly what the do-no-harm contract predicts: the certified layer-local improvements do not regress end-to-end behavior.

The remaining project gates — two-box candidate held-out evaluation against the Q6 teacher and final quality validation — are in progress; this card will be updated with those measurements as they land. Until then, treat quality claims as certified-not-worse than the baseline conversion rather than benchmarked. At 2.72 bpw, expect visible quality loss versus the 3.7 bpw tier on hard tasks — this tier trades fidelity for footprint by design.

Credits

• Base model: Thinking Machines — Inkling (Apache-2.0).
• MLX: Apple ml-explore.
• Two-box BF16 boundary pipeline, layer-local streamed-backward DWQ, sealed evidence harness, and the MLX inkling_mm_model port: Alis (avlp12).

Citation

Alis (avlp12) (2026). Inkling-975B-Alis-MLX-Dynamic-2.7bpw — 2.7 bpw certified layer-local DWQ MLX quantization of Inkling. https://huggingface.co/avlp12/Inkling-975B-Alis-MLX-Dynamic-2.7bpw