d3-mini mixed-bit MLX
Unofficial MLX post-training quantization of vllm-sr/d3-mini (bf16, Apache-2.0). The source is a 32-layer Qwen3.5 text stack plus a vision tower and a 255-way decision readout. This package is not an official vLLM Semantic Router release.
Suggested repository id: Jakevin/d3-mini-ternary-mlx.
Method
The original bfloat16 checkpoint is the reference. For each language-model projection
(attention Q/K/V/O, DeltaNet in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj,
and MLP gate/up/down) one candidate replaces that projection alone and the KL of the decision
readout's option distribution is measured. Candidates are ternary (one scale per group of 64,
values -s, 0, +s), and MLX affine 2-bit, 3-bit and 4-bit with group size 64. Ternary is stored
in MLX's built-in 2-bit affine tensors so mlx-lm / mlx-vlm quantized matmul can run it.
No custom Metal kernel is included.
Layer kinds are read from text_config.layer_types: 24 DeltaNet layers and 8
full-attention layers. Full attention is non-causal, as in the source checkpoint. DeltaNet stays causal.
Bits start from the assignment that minimizes the sum of those one-at-a-time KL values with the
average storage width of the language projections about 3 bits. Ternary counts as 2 storage bits
because that is the container it uses; it is selected only when its KL is lower than affine 2-bit.
That additive minimum sums to 1.5411.
4 later swaps each raise one projection by one bit and lower another by one bit.
A swap is kept only when the joint KL of the fully quantized model falls on 20
calibration questions and the average stays inside the same budget. Joint KL on those questions went from
0.2733 to 0.2351.
The shipped assignment's one-at-a-time sum is 1.5414.
The decision readout is a separate matrix, last_hidden @ readout.T. It does not use the tied
lm_head, so the token embedding is input-only and stays 8-bit affine (group 64).
The decision readout, RMSNorms, DeltaNet convolution / A_log / dt_bias, and the entire vision
tower stay at the original precision.
Calibration: 64 questions from the author's decision-v7 development split
(seed 20261011, one question from each drawn record) plus the three text questions in the
source model card. 67 questions total. hard-v1 was not used.
Size
| bytes | |
|---|---|
model.safetensors (quantized text + full-precision vision tower) |
2,928,233,188 |
readout.safetensors (fp32 decision head, unchanged) |
2,611,280 |
| this folder | 2,953,926,604 |
| source bf16 shards + readout | 9,081,227,360 |
| tied embedding tensors, 8-bit (used) | 675,430,400 |
Language-projection storage mix (ternary counted as 2 bits). Average 3.050 bits
over 3,569,090,560 projection parameters. The tied embedding is 8-bit and is not in that average.
The vision tower inside model.safetensors is uncompressed bf16.
| assignment | parameters | share | storage bits |
|---|---|---|---|
| ternary | 186,286,080 | 5.22% | 2 |
| 2 | 1,048,821,760 | 29.39% | 2 |
| 3 | 921,108,480 | 25.81% | 3 |
| 4 | 1,412,874,240 | 39.59% | 4 |
Evaluation
Single run, no confidence interval. 150 questions: 50 holdout from decision-v7 (seed 20261013,
disjoint from calibration and the 8 smoke questions) plus 50 decision-v7 and 50 transfer-v9
(same seed, disjoint from calibration). Agreement is argmax vs the source bf16 model on the same
MLX decision path. Accuracy is argmax vs the dataset label.
Overall agreement 140/150 (93.3%). Mean KL(bf16 || quantized) 0.0261, median 0.0077. Compressed accuracy 117/150, bf16 accuracy 119/150.
| suite | compressed correct | bf16 correct | agree | mean KL | median KL |
|---|---|---|---|---|---|
| holdout | 40/50 | 41/50 | 47/50 | 0.0167 | 0.0068 |
| decision-v7 | 41/50 | 42/50 | 47/50 | 0.0162 | 0.0063 |
| transfer-v9 | 36/50 | 36/50 | 46/50 | 0.0455 | 0.0099 |
Reload check: eight other questions scored from the in-memory quantized model and again after
loading out/release/ from disk. Agreement with memory 8/8.
The vision tower was not quantized and was not part of this evaluation.
Official source-card scores, quoted from vllm-sr/d3-mini (Jev Decision Index 0.3.1, source evaluation, not reproduced here): Decision Index 54.2, public suite 54.9, same-skill tests 52.7, new-domain tasks 48.6. Vision Index 65.5. Perception Test (internal) 70.3.
Use
from pathlib import Path
from transformers import AutoTokenizer
import decide
import d3_mlx
root = Path(".")
model, config, decision, readout = d3_mlx.load_model(root)
tokenizer = AutoTokenizer.from_pretrained(root)
answers = decide.predict(model, readout, tokenizer, decision["codes"], state, questions)
From a shell, in this directory:
python decide.py
That runs the three text questions from the source model card. python decide.py request.json
reads {"state": ..., "questions": ...}.
mlx_lm can load the language tower because config.json sets model_file to d3_mlx.py,
which builds layers from layer_types and uses stock MLX quantized linears. The vision weights
use mlx-vlm's vision_tower.* names and stay bf16. Text decisions use decide.py.
License
Apache-2.0. Source: vllm-sr/d3-mini,
itself built on Qwen/Qwen3.5-4B.
See LICENSE and NOTICE.
繁體中文摘要
這是 vllm-sr/d3-mini 的非官方 MLX 量化包。
語言層的每個投影各自試過三值、2、3、4 bit，先用一次換一層的 KL 在平均約 3.05 bit 的預算內分配，
再做 4 次對調。三值借 MLX 內建 2-bit affine 儲存。決策頭是獨立 readout，不經過 tied 的輸出頭，
所以嵌入維持 8 bit。決策頭、norm 與視覺塔維持原精度，視覺部分未壓縮。
model.safetensors 為 2928.2 MB，整個資料夾 2953.9 MB。
150 題（單次、無信賴區間）與 bf16 一致 140/150，
壓縮版正確 117/150，平均 KL 0.0261。
- Downloads last month
- 29
4-bit