Hy3-4bit (MLX)

A clean 4-bit MLX quantization of tencent/Hy3 (Hunyuan 3.0, Apache-2.0), converted with mlx-lm and verified running on Apple Silicon.

Hy3 is a 295B-parameter Mixture-of-Experts model with 21B active parameters, 80 layers, 192 routed experts (top-8) plus one shared expert, a native MTP (multi-token-prediction) head, and 256K context.

Why this quant is clean

Naive uniform 4-bit quantization degrades MoE models badly, because the router that picks which experts fire is precision-sensitive. This build follows a mixed-precision recipe:

• 4-bit (group size 64, affine) for attention and expert weights
• 8-bit for every MoE router gate (*.mlp.router.gate)
• MTP head preserved

That router protection is the difference between a coherent model and mush, and it matches the recipe used by the reference mlx-community/Hy3-preview-4bit.

Footprint

• Weights: ~166 GB
• Fits: two 128GB Apple Silicon machines (or one 192GB+ Mac), does not fit a single 128GB machine at 4-bit.

How it was made

from mlx_lm.convert import convert

def hy3_predicate(path, module, config=None):
    if path.endswith("mlp.router.gate"):
        return {"group_size": 64, "bits": 8}
    return {"group_size": 64, "bits": 4}

convert(
    hf_path="tencent/Hy3",
    mlx_path="Hy3-4bit",
    quantize=True, q_bits=4, q_group_size=64, q_mode="affine",
    quant_predicate=hy3_predicate,
)

hy_v3 architecture support comes from mlx-lm PR #1211.

Benchmarks (2x M5 Max, Thunderbolt RDMA)

Measured on two M5 Max (128GB each), pipeline-parallel over Thunderbolt with Apple's jaccl RDMA backend (MLX_METAL_FAST_SYNCH=1). Single-stream decode.

Metric	Value
Decode (generation)	36.91 tok/s
Prompt (prefill)	8.6 tok/s
Peak memory / node	84.4 GB
Backend	jaccl (RDMA), pipeline-parallel

Notes: at 4-bit (~166GB) the model does not fit a single 128GB machine, so this is a genuine 2-node cluster run. The ring/TCP backend trips the macOS Metal command-buffer watchdog on the cross-node fence wait; RDMA (jaccl) is required for stable single-stream decode.

Credits

• Base model: Tencent Hunyuan (tencent/Hy3, Apache-2.0)
• Architecture support: mlx-lm PR #1211
• Quantization + Apple Silicon cluster benchmark: bicVanYonk