Jakevin/d3-nano-ternary-3bit-mlx

d3-nano ternary MLX (~3 bit)

Unofficial MLX post-training quantization of vllm-sr/d3-nano (bf16, Apache-2.0). The source is a 24-layer Qwen3.5 text stack plus a vision tower and a 255-way decision readout. This package is not an official vLLM Semantic Router release. d3-nano is the source checkpoint's name. The bit allocation below is not a method named D3.

Suggested repository id: Jakevin/d3-nano-ternary-3bit-mlx.

Method

The original bfloat16 checkpoint is the reference. Each language-model projection is one unit: attention Q/K/V/O, DeltaNet in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj, and MLP gate/up/down. Layer kinds are read from text_config.layer_types (24 layers: 18 DeltaNet, 6 full attention). Full attention is non-causal. DeltaNet stays causal.

For each unit, one candidate replaces that projection alone and the KL of the decision readout's option distribution is measured against the source bf16 model. Candidates are ternary (one scale per group of 64, values -s, 0, +s) and MLX affine 2-bit, 3-bit and 4-bit with group size 64. Ternary is stored in MLX's built-in 2-bit affine tensors (code 0 → -s, code 1 → 0, code 2 → +s) so stock quantized matmul can run it. No custom Metal kernel is included. At a storage width of 2 bits, ternary is kept only when its KL is lower than affine 2-bit. Ternary still counts as 2 storage bits, because that is the container it uses.

Bits start from the assignment that minimizes the sum of those one-at-a-time KL values with the average storage width of the language projections between 2.90 and 3.00 bits. That additive minimum sums to 3.562783. Later swaps each raise one projection by one bit and lower another by one bit. A swap is kept only when the joint KL of the fully quantized model falls on 20 calibration questions and the average stays inside 2.90–3.00. Joint KL on those questions went from 1.509303 to 0.293697 across 12 swaps. The shipped assignment's one-at-a-time sum is 4.069842.

The decision readout is a separate matrix, last_hidden @ readout.T. It does not use the tied lm_head, so the token embedding is input-only and is stored at 8-bit affine (group 64). The decision readout, RMSNorms, DeltaNet convolution / A_log / dt_bias, and the entire vision tower stay at the original precision. The vision tower is not compressed.

Calibration: 64 questions from the author's decision-v7 development split (seed 20261011, one question from each drawn record) plus the three text questions in the source model card. 67 questions total. The joint check uses 20 of them. hard-v1 was not used.

Size

Measured from the files in this folder. model.safetensors contains the quantized text stack (including the 8-bit embedding) and the uncompressed vision tower.

	bytes
model.safetensors	1,804,522,963
vision tensors inside that file	662,833,152
embedding tensors inside that file	540,344,320
other tensors inside that file (quantized projections, norms, DeltaNet state)	601,222,784
safetensors header and alignment padding	122,707
readout.safetensors (fp32 decision head, unchanged)	2,089,040
this folder	1,829,658,644
source safetensors (bf16 weights + readout)	4,428,644,200

Language-projection storage mix. Ternary counts as 2 bits. Average 2.997446 storage bits over 1,372,717,056 projection parameters in 186 units. The 8-bit embedding is not in that average. The vision tower is not in that average.

assignment	units	unit share	parameters	parameter share	storage bits
ternary	4	2.15%	33,554,432	2.44%	2
2	47	25.27%	478,216,192	34.84%	2
3	47	25.27%	352,681,984	25.69%	3
4	88	47.31%	508,264,448	37.03%	4
total	186	100%	1,372,717,056	100%	2.9974 avg

Ternary is 4 of 186 units (2.15% of units) and 33,554,432 of 1,372,717,056 parameters (2.44% of projection parameters).

ternary unit	parameters
language_model.model.layers.0.mlp.gate_proj	12,582,912
language_model.model.layers.18.linear_attn.out_proj	4,194,304
language_model.model.layers.19.mlp.down_proj	12,582,912
language_model.model.layers.22.linear_attn.out_proj	4,194,304

Check against the source bf16 model

Same 150 questions as the d3-edge gate: 50 holdout questions, 50 decision-v7 questions and 50 transfer-v9 questions (seed 20261013, disjoint from the 67 calibration questions). Holdout and decision-v7 share 49 ids, so the merged 150 counts those twice, matching the list that was specified. One run, no confidence interval. Agreement is argmax against the source bf16 model loaded through this same MLX path. Accuracy is argmax against the question's label. KL is KL(bf16 || quantized).

split	agree	quantized correct	bf16 correct	mean KL	median KL
holdout	47/50	39/50	38/50	0.049618	0.012510
decision-v7	47/50	40/50	39/50	0.045930	0.011232
transfer-v9	37/50	32/50	34/50	0.168804	0.037441
merged 150	131/150	111/150	111/150	0.088117	0.019553

Agreement is 131/150 (87.3%). Quantized accuracy is 111/150, the same count as the bf16 run (111/150). Mean KL is 0.088117. Median KL is 0.019553.

This did not meet our own 90% agreement bar. The package is published as-is: one run, no confidence interval, under that self-set bar.

This was not a formal benchmark. The Jev Decision Index, the vision board and Perception Test were not run on these weights.

The vision tower was not quantized. Image inputs and video inputs were not run.

Scores quoted from the source model card

The source card for vllm-sr/d3-nano reports, on its own evaluation and not on this package:

• Jev Decision Index 0.3, public suite: 42.22 (highlights; official 0.3 kit, 140,178 public requests).
• Jev Decision Index 0.3.1 table: index 37.2, public 42.2, same-skill tests 37.6, new-domain tasks 31.8.
• Jev Decision Index vision board 0.3.1: 58.1 (public 60.4, private 55.8).
• Perception Test, all 19,140 validation questions: 63.5.

Those numbers are copied from that card (Highlights and Evaluation). They were not remeasured here.

Scores quoted from FluidInference/d3-coreml

FluidInference/d3-coreml publishes a Core ML conversion of the same source. The nano folder card describes text graphs as fp16 compute with int8 weights, and a vision tower left in fp32. It reports answer flips against the upstream PyTorch runtime (bf16, MPS), same requests:

• Text (typed-decisions TEST, 20 requests): 0 / 100
• Images (receipt checks): 0 / 24
• Video (4-second street clips, 2 fps): 1 / 28

Those three rows are copied from that card. They were not remeasured here. They are not on the 150-question list above, so they are not a comparison on the same items.

Use

from pathlib import Path
from transformers import AutoTokenizer
import decide
import d3_mlx

root = Path(".")
model, config, decision, readout = d3_mlx.load_model(root)
tokenizer = AutoTokenizer.from_pretrained(root)
answers = decide.predict(model, readout, tokenizer, decision["codes"], state, questions)

From a shell, in this directory, python decide.py runs the three text questions from the source model card. python decide.py request.json reads a JSON object with state and questions.

mlx_lm can load the language tower because config.json sets model_file to d3_mlx.py, which builds layers from layer_types and uses stock MLX quantized linears. Stock Qwen3.5 code that assumes a fixed full-attention interval will not match this checkpoint. The vision weights use mlx-vlm's vision_tower.* names and stay at the original floating-point precision. Text decisions use decide.py. Image and video inputs are not wired up or checked in this package.

License

Apache-2.0. Source: vllm-sr/d3-nano, itself built on Qwen/Qwen3.5-2B. See LICENSE and NOTICE.

繁體中文摘要

這是 vllm-sr/d3-nano 的非官方 MLX 量化包，授權 Apache-2.0。d3-nano 是來源模型的名字，不是這次位元分配方法的名字。語言層每個投影各自對原 bf16 決策分布量 KL，在平均約 3 bit 的預算（實測 2.9974，範圍 2.90–3.00，三值以 2 bit 容器計）內分配三值或 2/3/4 bit，再用 12 次對調做聯合 KL 修正（20 題，聯合 KL 1.5093 → 0.2937）。三值 4/186 個單位（2.15%）、33,554,432/1,372,717,056 個參數（2.44%），借 MLX 內建 2-bit affine 儲存。嵌入 8 bit。決策頭、norm 與視覺塔維持原精度。視覺塔沒有壓縮，也沒有驗證圖片或影片輸入。沒有做正式基準評測。150 題（holdout 50 + decision-v7 50 + transfer-v9 50）單次、無信賴區間，與 bf16 一致 131/150（87.3%），未達我們自訂的 90% 一致率門檻，照實公開；正確率 111/150，與 bf16 的 111/150 相同；平均 KL 0.088117，中位數 KL 0.019553。model.safetensors 1,804,522,963 bytes（含 8-bit 嵌入與未壓縮視覺塔），整個資料夾 1,829,658,644 bytes。FluidInference/d3-coreml 的 nano 卡寫 Core ML 文字圖為 int8 權重，文字 0/100、圖片 0/24、影片 1/28 次答案翻轉，數字引自該卡，這裡沒有重跑。來源模型卡的 Jev Decision Index 公開套件 42.22、0.3.1 指數 37.2、視覺指數 58.1、Perception Test 63.5 也是原文數字。

Related release

A ~3.5-bit build of the same method that clears our 90% agreement bar (141/150) is at Jakevin/d3-nano-ternary-mlx.