Hy3-1M — 4-bit (INT4) quantization of tencent/Hy3 for 1M context

A 4-bit weight-only (W4A16) quantization of tencent/Hy3 (HYV3ForCausalLM, hy_v3) — a 295B-parameter / 21B-active Mixture-of-Experts model. Packaged in the compressed-tensors pack-quantized format so it loads directly in vLLM.

Why this model

• Small. ~146 GB vs ~557 GB for the original BF16 (~3.8× smaller). The whole 295B MoE now fits on a single ≥180 GB GPU (e.g. one B200 192 GB / B300 ~288 GB) with KV-cache headroom — no tensor-parallel sharding required just to load it. (Note: it does not fit a 141 GB H200 without offload/TP.)
• vLLM-native. Loads out of the box with vLLM (recent build with hy_v3 support) using the Marlin INT4 MoE + Linear kernels. Fast tensor-core prefill.
• Long context via YaRN. With YaRN RoPE scaling the context extends from the native 262,144 up to 1,048,576 (1M) tokens (configurable). Dense needle-in-a-haystack retrieval is verified past native (1M, PASS) on a single GPU; see Long context below.

Verified results (single B300, this checkpoint)

Test	Result
HumanEval pass@1 (greedy)	✅ 150/164 = 91.5% — coding ability well-preserved at 4-bit
GSM8K (0-shot CoT, greedy)	✅ 1265/1319 = 95.9% — math reasoning preserved at 4-bit
Chat sanity	✅ correct
Needle-in-a-haystack 4K / 16K / 64K / 137K (in-range)	✅ all PASS
Needle-in-a-haystack 320K/1M dense (YaRN ×4, fp8/int4 KV)	✅ PASS — retrieval works past the native 262,144

Quantization details

Scheme	W4A16 — 4-bit int, symmetric, group_size=128, RTN (round-to-nearest, data-free)
Format	compressed-tensors pack-quantized (quant_method: compressed-tensors)
Quantized	attention q/k/v/o_proj, dense-layer FFN, all 192 routed experts + shared expert (gate/up/down_proj)
Kept in original precision	lm_head, router gate (mlp.router.gate), eh_proj (MTP), all norms, embed_tokens
Base dtype	bf16 (scales stored bf16)

Produced by a direct tensor-by-tensor RTN packer (no calibration dataset). RTN keeps the pipeline simple and lossless-format-correct; for maximum quality at 4-bit, a calibrated GPTQ/AWQ pass would be marginally better.

Running with vLLM

Requires a vLLM build new enough to include the hy_v3 architecture (vLLM main/nightly at time of writing). Example on a single GPU:

vllm serve /path/to/Hy3-1M \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code

NVIDIA Blackwell (sm_100/sm_103, e.g. B200/B300) note: at the time of testing, FlashInfer's runtime JIT could not compile for compute_103a with the bundled CUDA toolkit, which crashed the default sampler/attention. Work around it with the Triton attention backend + native sampler:

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /path/to/Hy3-1M \
  --attention-backend TRITON_ATTN \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code --enforce-eager

--kv-cache-dtype fp8 halves KV memory (recommended for long context). On Hopper/Ada or with a FlashInfer build that supports your GPU, you can drop the two workaround flags.

Inference tuning: MoE top-K (speed vs quality)

The number of routed experts per token (num_experts_per_tok, native 8) can be lowered at inference time (no re-quantization) to trade quality for less expert compute, via vLLM's --hf-overrides '{"num_experts_per_tok": K}'. Measured on this 4-bit checkpoint (greedy):

top-K	HumanEval	GSM8K	routed-expert FLOPs
8 (native)	91.5%	95.9%	100%
6	89.6% (−1.9)	94.8% (−1.1)	~75%
4	86.6% (−4.9)	93.5% (−2.4)	~50%

Degradation is graceful — even top-4 (half the routed-expert compute) stays coherent and usable. top-6 is a sweet spot (~25% less expert compute for ≈1-2 pts). Coding is a bit more sensitive to fewer experts than math. (Default = 8; only lower it if you need the speed/energy.)

Long context (YaRN)

The base model is rope_type: "default" with max_position_embeddings: 262144. To go beyond, enable YaRN in config.json:

"rope_parameters": {
  "rope_theta": 11158840.0,
  "rope_type": "yarn",
  "factor": 4.0,
  "original_max_position_embeddings": 262144
}

and raise --max-model-len (up to 262144 * factor = 1048576).

This shipped config.json already has YaRN factor 4 enabled (context up to 1,048,576). Set rope_type back to "default" if you want the native-only 262,144 behavior.

Memory on a single ~275 GB GPU (146 GB weights):

• fp8 KV (--kv-cache-dtype fp8): comfortably fits ~500K dense tokens; fast tensor-core prefill.
• int4 KV (--kv-cache-dtype int4_per_token_head): fits ~1M dense tokens, but its kernel is compute-bound and much slower for long prefill.
• Full dense 1M is best served with multi-GPU (tensor-parallel) for both memory and speed.

Verified results (single B300, this checkpoint)

Test	Result
HumanEval pass@1 (greedy)	✅ 150/164 = 91.5% — coding ability well-preserved at 4-bit
GSM8K (0-shot CoT, greedy)	✅ 1265/1319 = 95.9% — math reasoning preserved at 4-bit
Chat sanity (11+22+33 → 66; capital of France → Paris; first 5 primes)	✅ correct
Needle-in-a-haystack 4K / 16K / 64K / 137K (in-range)	✅ all PASS
Needle-in-a-haystack 1M dense (YaRN ×4, fp8 KV)	✅ PASS — retrieval works past the native 262,144

How HumanEval was measured (for reproducibility)

• Engine/config: this W4A16 checkpoint served by vLLM on a single B300, as shipped (YaRN factor 4 enabled), --attention-backend TRITON_ATTN --kv-cache-dtype fp8 --enforce-eager, VLLM_USE_FLASHINFER_SAMPLER=0.
• Data: the 164 problems from openai_humaneval (human-eval package).
• Decoding: greedy (temperature=0), max_tokens=768, pass@1 (single sample per problem).
• Prompting: chat template with the instruction *"Complete the following Python function. Return the COMPLETE function in a single python code block. No tests, no explanations."* The first python block is extracted; if it lacks the entry_point def, the original stub is prepended.
• Scoring: each candidate is run as candidate + test + check(entry_point) in a subprocess (15s timeout); exit-code 0 = pass.
• Result: 150/164 = 91.5%. Note this uses a chat+extraction harness (not the canonical raw-completion protocol), so a few of the 14 misses may be extraction artifacts — treat 91.5% as a conservative figure.

GSM8K: full 1319-problem test set, 0-shot chain-of-thought, greedy (temperature=0, max_tokens=512), prompt "Solve step by step… on the last line write 'The answer is '"; the final number is compared to the gold answer (after ####). Result: 1265/1319 = 95.9%.

Caveats & honesty

• This is a community derivative, not affiliated with or endorsed by Tencent.

License

Apache-2.0, inheriting the license of the base model tencent/Hy3 (see LICENSE).