Hy3-Alis-MLX-Dynamic

Mixed-precision MLX quantizations of tencent/Hy3 (295B total / 21B active MoE) for Apple Silicon, sized for 128 / 256 / 512 GiB unified-memory Macs. One repo, one card — each quantization lives on its own branch.

branch	eff. bpw	size	target Mac	decode*	KL vs 8-bit (top-1 flip)	status
main	4.568	168.5 GB	256 GiB	28.7 tok/s	0.094 (8.9%)	✅ daily driver
T512	6.561	242.0 GB	512 GiB	24.0 tok/s	0.020 (4.6%)	✅ flagship, near-lossless
T512REF	8.502	313.5 GB	512 GiB	22.2 tok/s	(reference)	✅ max quality
T128	2.375	87.6 GB	128 GiB	38.2 tok/s	0.802 (26.2%)	⚠️ experimental, DWQ-distilled
mtp-bf16	bf16	7.5 GB	—	—	—	MTP layer sidecar

* single-stream decode on an M3 Ultra, short prompt; longer contexts decode slower.

Which build?

• 512 GiB Mac → T512. Faster than the 8-bit and statistically near-lossless (KL 0.020 nats). 128K-token needle retrieval verified at 285 GB peak; 256K fp16 KV fits in the default wired limit (242 + 80 = 322 GB < ~384 GB).
• 256 GiB Mac → main. 64K-token needle verified at 190 GB peak — just inside the default ~192 GB wired limit. For 128K, use int8 KV (--kv-bits 8, generate path) or raise iogpu.wired_limit_mb.
• 128 GiB Mac → T128, with eyes open: see the experimental warning below. 16K fp16 verified at 93.8 GB peak; 32K with int8 KV peaked at 109 GB, which requires raising the wired limit on a 128 GB machine.
• KL ground truth / archival → T512REF (uniform 8-bit).

Usage

Hy3 (hy_v3) support is not yet in an mlx-lm release (ml-explore/mlx-lm#1211 is open). Until it merges, install from the pinned branch (PR #1211 plus an fp32-router correctness patch, commit 14f7837):

pip install git+https://github.com/avlp12/mlx-lm@hy3-support

from mlx_lm import load, generate

# main = the 256 GiB build; pass revision= for other tiers
model, tokenizer = load("avlp12/Hy3-Alis-MLX-Dynamic")
model, tokenizer = load("avlp12/Hy3-Alis-MLX-Dynamic", revision="T512")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain MoE routing in two sentences."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

CLI / server:

mlx_lm.generate --model avlp12/Hy3-Alis-MLX-Dynamic --prompt "..." -m 512
hf download avlp12/Hy3-Alis-MLX-Dynamic --revision T512 --local-dir Hy3-T512
mlx_lm.server --model ./Hy3-T512 --port 8080

Long context on the smaller tiers: quantize the KV cache in the generate path (--kv-bits 8 --kv-group-size 64). Note mlx_lm.server currently serves fp16 KV only.

Recipes (verified from each build's config.json, not from intent)

97.0% of Hy3's parameters are the 192 routed experts, so expert bit-width sets the size; everything else is cheap to keep at high precision. Common to all tiers:

• mlp.router.gate — never quantized (bf16). The sigmoid+bias top-8 routing is a discrete control path; 62M params total, so full precision is free.
• expert_bias (e-score correction) kept fp32, verified in the output shards.
• Attention bits = min(expert bits + 2, 8).

tier	routed experts	expert down_proj	attn / shared / dense	embed / lm_head
T128	2-bit g128	3-bit g128 on 12 early/late layers	4-bit g64	6-bit g64
main (T256)	4-bit g64	4-bit g64	6-bit g64	8-bit g64
T512	6-bit g64	6-bit g64	8-bit g64	8-bit g64
T512REF	8-bit g64	8-bit g64	8-bit g64	8-bit g64

T128 additionally went through a DWQ pass (distilled weight quantization: its quantization scales/biases were trained against the 8-bit build's logits over 384×1024-token samples) — same size and recipe, KL −15% overall (−21% EN, −27% code, −8% ZH) vs the raw quantization.

Quality

KL divergence & top-1 flip rate vs T512REF — measured on a fixed 3,072-token slice (English / code / Chinese thirds). Disclosure: the reference is the 8-bit build, not bf16 — the bf16 model (598 GB) does not fit in 512 GB of RAM.

tier	overall KL	EN	code	ZH	overall flip
T512	0.020	0.017	0.011	0.033	4.6%
main	0.094	0.058	0.054	0.169	8.9%
T128	0.802	0.641	0.336	1.430	26.2%

Perplexity (strided, ctx 2048 / stride 1024, fixed corpora):

tier	wikitext	code	Chinese
T512	3.85	1.93	3.54
main	3.97	1.98	3.73
T128	6.25	2.53	9.01

lm-eval (hellaswag / piqa / winogrande, 0-shot, limit 500, same harness across tiers — absolute numbers are base-style loglikelihood, useful only for tier-vs-tier comparison): all three tiers are statistically indistinguishable (±2σ), e.g. piqa acc 0.75–0.77 across every tier. Loglikelihood tasks are insensitive at this scale; the KL table above is the discriminating metric.

Long-context needle retrieval (needle at 25% depth, greedy): every listed claim was tested, not extrapolated —

tier	context tested	result	peak memory
T128	16K fp16 · 32K int8-KV	PASS · PASS	93.8 · 109.2 GB
main	16K · 64K fp16	PASS · PASS	174.6 · 190.3 GB
T512	64K · 128K fp16	PASS · PASS	263.8 · 285.2 GB

⚠️ T128 is experimental

2.375 bpw on a 295B MoE with 18.9M-parameter experts is aggressive. It loads and generates fluent English/Korean at 38 tok/s in under 88 GB, and its lm-eval scores match the bigger tiers, but the distribution damage is real and concentrated in Chinese (KL 1.43, 42.2% top-1 flip, PPL +154% vs T512) even after the DWQ pass. Recommended for English/code use on 128 GiB machines; not recommended for Chinese. A Chinese-heavy DWQ data mix is the remaining recovery lever.

How it compares to other Hy3 MLX ports

Every public Hy3(-preview) MLX quantization we could find, measured on the same harness, same machine (M3 Ultra 512GB, wired-limit set, quiet disk): identical 3,072-token KL slice vs our 8-bit reference, identical PPL corpora, same prompt for decode speed. Sorted by size:

model	size	KL vs 8-bit† (flip)	PPL wiki / code / ZH	decode	fits default wired limit of
ours T128	87.6 GB	0.802 (26.2%)	6.25 / 2.53 / 9.01	38.2 tok/s	128 GB Mac (93.8 GB peak @16K)
ox-ox w2q3exp	112.6 GB	0.686 (24.1%)	5.85 / 2.45 / 7.36	8.9 tok/s	256 GB Mac (112.3 GB peak @0 ctx — over a 128 GB Mac's ~96 GB)
unigilby MXFP4-imatrix	161.2 GB	0.172 (11.3%)	4.09 / 2.00 / 3.93	28.1 tok/s	256 GB Mac
mlx-community 4bit‡	166.0 GB	0.319 (18.0%)‡	3.90 / 2.10 / 3.75	32.5 tok/s	256 GB Mac
ours main (T256)	168.5 GB	0.094 (8.9%)	3.97 / 1.98 / 3.73	28.7 tok/s	256 GB Mac (190.3 GB peak @64K)
ours T512	242.0 GB	0.020 (4.6%)	3.85 / 1.93 / 3.54	24.0 tok/s	512 GB Mac
ours T512REF	313.5 GB	(reference)	—	22.2 tok/s	512 GB Mac
inferencerlabs Q9	322.2 GB	—	—	—	does not load in stock mlx-lm (custom config: no bits in its quantization block)

† KL(8-bit ref ‖ model) on the fixed EN/code/ZH slice — lower is better. ‡ mlx-community's build quantizes Hy3-preview, a different base checkpoint than the final Hy3; its KL vs our final-Hy3 reference includes base-model drift, so compare it on PPL. Its uniform 4-bit g64 recipe (8-bit routers) decodes faster than mixed recipes because its attention path carries fewer bytes per token.

Reading the table:

• 4-bit class (~161–169 GB): at essentially the same size, main has the lowest KL of any final-Hy3 build (0.094 vs 0.172 for the imatrix MXFP4 build — ~45% lower) and the best code/ZH perplexity, at equal speed.
• low-bit class: ox-ox's 112.6 GB build is somewhat closer to 8-bit (KL 0.686 vs 0.802 after T128's DWQ pass) — it spends 25 GB more, mostly on 8-bit attention. But it exceeds a 128 GB Mac's default wired limit before the first token, and decoded at 8.9 tok/s on our machine vs T128's 38.2. If your Mac has 256 GB, use main instead; if it has 128 GB, T128 is the build that actually fits.
• Routers: ours keep the sigmoid+bias router in bf16 (ox-ox and mlx-community quantize it to 8-bit; unigilby's config applies its 4-bit default to the router). With 192 experts and near-tie top-8 margins, full-precision routing is the cheapest correctness insurance in the recipe (~124 MB total).

mtp-bf16 branch — MTP self-speculative decoding (measured: 1.13×, lossless)

Hy3 ships a Multi-Token-Prediction layer (model.layers.80.*, 3.75B params, DeepSeek-V3-style eh_proj/enorm/hnorm + MoE block) that mlx-lm strips at conversion. The mtp-bf16 branch preserves it verbatim in bf16 (7.5 GB) plus a runnable self-speculative-decode harness (mtp_run.py + module + step generators).

Measured on the 8-bit build (M3 Ultra, greedy, 200 tokens, output verified token-identical to plain decode):

config	tok/s	speedup	accept-len
plain decode	21.4	1.00×	—
MTP k=1 (bf16 draft)	24.1	1.13×	1.65
MTP k=2	22.9	1.07×	2.02
MTP k=3	19.7	0.92×	2.13

Notes: k=1 with the bf16 sidecar is the operating point. A 4-bit draft was measured too (isolated draft call 2.26 → 1.74 ms, −23%): the saving is real but amounts to ~1% of loop time (the 47 ms verify forward dominates), while acceptance drops slightly and consistently across EN/KO/code prompts (accept-len −0.02…−0.06) — net ≈ −1…−4% vs bf16, tie on code. Use the 4-bit draft only if you want its 5.4 GB memory saving. The harness uses the MTP's own KV cache and final-layernorm ("normed hidden") chaining for k>1.

hf download avlp12/Hy3-Alis-MLX-Dynamic --revision mtp-bf16 --local-dir mtp
python mtp/mtp_run.py --model <any-tier> --mtp mtp/Hy3-MTP-bf16.safetensors --k 1

Correctness

• Cross-framework logit parity vs transformers 5.12.1 reference implementation on tiny-random weights: top-1 agreement 100%, max |Δlogit| ≤ 5e-3 (fp32).
• One correctness patch on top of PR #1211: the router matmul runs in fp32, matching the reference F.linear(hidden.float(), weight.float()) — near-tie top-8 selection under a sigmoid+bias router is sensitive to bf16 matmul noise.
• Recipe verification: quantization blocks parsed from each output config.json; router dtype and expert_bias fp32 checked in the shards themselves.

Credits

• Tencent Hunyuan for Hy3 (Apache 2.0) and the AngelSlim compression toolkit — the official quantization path for CUDA deployments (FP8/INT4 for vLLM/SGLang). These MLX builds are an independent, complementary Apple-Silicon path.
• kernelpool for the hy_v3 MLX port (ml-explore/mlx-lm#1211).
• Built with MLX and mlx-lm.

Citation — if you use these builds, please cite Tencent's Hy3 release (tencent/Hy3, 2026) and Apple's MLX framework.