Hy3-JANG_2K-MTP

Quantized tencent/Hy3 for Apple Silicon MLX / JANG runtimes — a 295B-total / 21B-active text MoE, packed to ~98 GiB with the native Multi-Token-Prediction (MTP) head preserved.

Source	tencent/Hy3
License	other — inherits the upstream Tencent Hunyuan Community License
Architecture	hy_v3 (HYV3ForCausalLM), text-only
Parameters	295B total / 21B active per token
Format	JANG_2K (mixed-affine), routed experts avg 2.33-bit
Bundle size	105.28 GB (98.05 GiB), 23 shards, 2,918 tensor keys
MTP	num_nextn_predict_layers = 1 — speculative-decoding head preserved
Context	262,144 tokens

What this is

Hy3-JANG_2K-MTP is a JANG mixed-affine quantization of Tencent's Hy3 dense-MoE. It targets Apple Silicon runtimes (MLX / vMLX) and keeps the model's native MTP layer in the bundle so a compatible engine can run next-token speculative decoding. The 2K profile spends an extra bit on the routed down_proj (3-bit vs the 2-bit gate/up), which cleans up the sampling tail relative to a uniform 2-bit pack.

Quantization (JANG_2K)

Tensor family	Policy
Routed expert gate_proj / up_proj	affine 2-bit, group size 128
Routed expert down_proj	affine 3-bit, group size 128
Attention q/k/v/o	affine 8-bit
Shared expert	affine 8-bit
Dense layer-0 MLP	affine 8-bit
embed_tokens	affine 6-bit
lm_head	affine 8-bit
RMSNorms, router gate, expert bias	16-bit passthrough
MTP matmuls	affine 8-bit

Routed-expert effective average: 2.33 bit. AWQ scaling is disabled for this bundle (measured negligible on Hy3).

Architecture

Hy3 is a text-only dense-causal-GQA MoE — not MLA, not SSM, not sliding-window, not a VLM.

• 80 decoder layers, hidden_size 4096
• GQA: 64 attention heads / 8 KV heads, head_dim 128, QK-norm
• RoPE default, rope_theta 11,158,840, max_position_embeddings 262,144
• MoE: 192 routed experts, top-8, sigmoid router + expert-correction bias, route_norm, router_scaling_factor 2.826, 1 shared expert, first_k_dense_replace 1
• MTP: 1 next-token-prediction layer preserved (num_nextn_predict_layers = 1)
• vocab_size 120,832

Reasoning & tool use

• Reasoning: <think>…</think> tags, reasoning_effort (no_think / low / high).
• Tool calling: Hunyuan / Tencent XML-style tags (<tool_calls>, <tool_call>, <arg_key>, <arg_value>).
• Hy3's tokenizer uses a :opensource special-token dialect (e.g. <｜hy_eos:opensource｜>, <think:opensource>); the bundled chat_template.jinja is the upstream template. A compatible runtime must resolve these variant-suffixed tokens at the token→text boundary.

Runtime support

• Converted and structurally verified (index complete, 2,918 tensors / 23 shards).
• Runs on the vMLX Python engine with Hy3 support: JANG affine loader, GQA KV cache, <think> reasoning stream, and Hunyuan tool-call parsing.
• The MTP head is preserved and the runtime supports Hy3 speculative decoding. MTP throughput on a 2-bit-routed backbone is workload-dependent; this bundle is provided so a compatible engine can exercise the native head.

Requires a Hy3-aware MLX/JANG runtime. Stock mlx-lm / transformers will not load the JANG mixed-affine layout or the MTP head as-is.

Known limitations

• No published quality benchmark yet for this specific pack.
• MTP speculative-decode speedup is not asserted as a fixed figure — it depends on acceptance rate at the target sampling regime.
• Very loose sampling (top_p 1.0 + temperature 0.9) exposes more of the routed-expert tail; a mild top_p ≤ 0.9 or min_p floor is recommended for long-form generation.

소개 (Korean)

이 번들은 Tencent의 Hy3 (295B 총 파라미터 / 21B 활성 MoE, 텍스트 전용)를 Apple Silicon MLX / JANG 런타임용으로 양자화한 모델입니다. JANG_2K 프로파일은 라우팅 전문가의 down_proj를 3-bit로, gate/up을 2-bit로 양자화하며(평균 2.33-bit), 네이티브 MTP(Multi-Token-Prediction) 헤드를 번들에 보존합니다. Hy3의 GQA 어텐션 상태와 MoE 라우팅, :opensource 특수 토큰 방식을 정확히 구현한 런타임에서만 사용해야 합니다.