auroter/Hy3-295B-A21B-FP8-Dynamic

Hy3 295B-A21B · FP8 Dynamic

Curated FP8 — measurably better than the official FP8 release.

_{The quality arm · speed sibling: Hy3-295B-A21B-NVFP4A16-GPTQ · 2026-07-16}

---

This is a curated FP8 quantization of tencent/Hy3 (295B-A21B MoE). It was built for one goal: the highest-quality 8-bit build of this model that runs on four RTX PRO 6000 Blackwell cards. That machine — 4× 96 GB — is the starter configuration for serious local inference on models of this class, and this build is designed, measured, and tuned for exactly that box. The mechanism is granularity: everywhere the official FP8 release rounds a whole tensor with one scale, this build carries a scale per output channel; where the official release pins activations to one static scale, this build scales every token as it arrives. The result measures better than the official FP8 on both axes — lower perplexity and faster decode, served identically — with full 262,144-token context.

Highlights

• Strictly better than the official FP8, measured on the same engine — served identically (vLLM 0.25, fp8 KV, same box, same corpus, same hour): perplexity 4.8194 vs 4.9133 (−1.91%), plain decode 103.3 vs 94.3 tok/s (+9.5%), MTP depth 1 110.1 vs 99.7 tok/s (+10.4%). Nicer and faster, in every configuration.
• Finer at every layer — weights FP8 with per-output-channel scales (the official release: one scalar per tensor); activations FP8 scaled per token at runtime (the official release: one static scale fixed at build time).
• What matters most stays 16-bit — the shared expert that fires on every token, the dense first layer, embeddings, output head, router gates, and norms are BF16; the multi-token-prediction (MTP) draft head ships byte-identical.
• Full 262,144-token context on 4× 96 GB — fp8 KV cache, the same scheme Tencent's own config declares for this checkpoint.
• The 16-bit keep-set costs bandwidth and wins anyway — this build reads ~8% more bytes per token than the official FP8, and still decodes faster on vLLM ≥0.25 because per-channel FP8 rides the fast fused-MoE kernels and the byte-identical draft head accepts more drafts than the official's quantized one (1.61 vs 1.56 mean acceptance length). For absolute maximum speed on this machine, the NVFP4 sibling remains the pick.
• Ships the fix for Chinese characters appearing in English output — the same ban kit as the sibling build; the leak is a base-model behavior every quant inherits (receipts under decision 5). Works with any quant of this model.

_{All numbers measured on 4× RTX PRO 6000 Blackwell (SM120), vLLM 0.24.0, TP4, single stream.}

---

The decisions

1 · Routed experts: FP8 with per-channel scales

45,504 matrices — 79 MoE layers × 192 experts × gate/up/down — at FP8-E4M3, one scale per output channel, computed from the weights themselves.

Why 8-bit here. This is the quality arm. The experts hold ~96% of the weights, and at 8-bit the rounding grid is fine enough that data-free scaling is exact: each channel's scale is a pure function of that channel's weights, no calibration estimate involved. The sibling build takes these same experts to 4-bit for speed and needs Hessian calibration to do it safely; at 8-bit that risk class does not exist.

Why per-channel and not per-tensor. A tensor-wide scalar forces every channel onto the dynamic range of the tensor's loudest channel; quiet channels lose resolution to a maximum they never use. Per-channel scaling gives each output row its own grid — 109.9 million scales across the build instead of one per tensor. This is the single largest difference from the official FP8 release, and the perplexity gap follows from it.

2 · Activations: FP8 scaled per token, at runtime

Why dynamic. The official release fixes one static activation scale per tensor at build time — an estimate from calibration data that every future token must live under. This build measures each token's actual range as it arrives and scales exactly to it. No estimate, no calibration-set bias, no clipping when an unusual token exceeds the calibration range. E4M3's exponent absorbs the within-token spread natively; the per-token scale handles the between-token spread.

3 · The every-token 16-bit keep-set

237 shared-expert matrices, the dense layer-0 MLP, embeddings, the output head, router gates, expert biases, and norms — all BF16, byte-identical to source.

Why. The shared expert runs on every token in every MoE layer; the routed experts each see a fraction of the traffic. Error in the every-token path compounds across the whole sequence, so those weights stay at full precision. Router gates and the output head shape logits directly — small tensors, outsized influence, not worth any rounding. Tencent's own curated quantized releases protect exactly these classes; their sensitivity judgment and ours agree.

4 · The MTP draft head: untouched BF16

All 593 tensors of the draft layer, byte-identical to source.

Why keep it whole. A draft head earns its keep through acceptance rate, and acceptance is speed: every accepted draft token is a token the target did not have to decode alone. The official FP8 quantizes its draft head; this one ships at full precision.

Why it wins here, measured. The full-precision head out-accepts the official's quantized one on the same engine: 1.61 vs 1.56 mean acceptance length at depth 1. On vLLM ≥0.25 that acceptance converts directly to speed — 110.1 tok/s vs 103.3 plain on this build. Serve MTP depth 1 on vLLM 0.25 or later.

One engine-version fact. On vLLM 0.24.x, speculation measures net-negative on this build (64.1 vs 83.0 plain) despite the same healthy acceptance: 0.24's spec-decode path drops this build's per-channel MoE onto an unoptimized Triton kernel, making one draft-plus-verify cycle cost roughly two plain steps. 0.25's kernels close the gap entirely. On 0.24, run plain; on ≥0.25, run depth 1.

5 · Stop Chinese characters from appearing in the output — and ship the fix

Why a fix is needed at all. The base checkpoint leaks Chinese synonyms into pure-English prose, at every precision. At semantic slots it places the Chinese word at rank 2–3 of the next-token distribution with double-digit probability. The leak survives Tencent's own FP8 — it is in the checkpoint's training, not in anyone's rounding — and a rank-2 token at 37% cannot be filtered by any top_p, top_k, or min_p setting. Full forced-token measurements are on the sibling build's card.

The sampler can do what the weights cannot: ban every token containing a Chinese, Japanese, or Korean (CJK) script character, passed as allowed_token_ids. Control tokens are exempt — their names carry fullwidth-bar punctuation from the CJK block, and banning them removes end-of-sequence; the ban governs text, grammar tokens stay emittable.

The kit, in cjk-ban/: generate-allowed-ids.py (model-agnostic) and the pregenerated list for this model (47,295 banned, 73,523 allowed, of 120,818 — identical tokenizer to the sibling build). Verified live on this build: 50,700 tokens at temperature 0.9 / top_p 1.0 produced one whole-word substitution unbanned; under the ban the class is unemittable, and generation terminates normally — finish_reason stop, verified.

import json
from openai import OpenAI

allowed = json.load(open("cjk-ban/hy3-allowed-token-ids.json"))["allowed_token_ids"]
client = OpenAI(base_url="http://localhost:30000/v1", api_key="unused")
out = client.chat.completions.create(
    model="hy3-fp8-dynamic",
    messages=[{"role": "user", "content": "Write a scene where he denies it."}],
    temperature=0.9,
    extra_body={"allowed_token_ids": allowed},
)

---

Measured results

Both arms served identically: vLLM 0.25.0, TP4, fp8 KV, utilization 0.95, single-stream decode, the same 48-chunk wikitext-2 corpus and generation harness, the same 4× RTX PRO 6000 Blackwell box, measured the same day. Single-stream is deliberate: this build is for a personal workstation, and a workstation serves one person at a time.

Measure	Official FP8	This build
Perplexity, wikitext-2 (66,055 tokens)	4.9133	4.8194 (−1.91%)
Plain decode (tok/s)	94.3	103.3 (+9.5%)
MTP depth 1 (tok/s)	99.7	110.1 (+10.4%)
MTP mean acceptance length	1.56	1.61
Weight scales	1 per tensor	1 per output channel
Activation scales	static, fixed at build	per token, at runtime
MTP draft head	quantized	BF16, byte-identical
Stray Chinese-character events at temp 0.9 / top_p 1.0	0 / 50,598 tokens	1 / 50,700 tokens¹
Mid-word corruption / replacement chars	0	0
Full 262,144-token context	yes	yes — 337,600-token KV pool, 1.29× full-context concurrency
Verbatim recall probes at 162K prompt tokens (10% and 90% depth)	—	2/2 exact

_{¹ One whole-word substitution of the class documented under decision 5 — a base-model behavior, statistically inseparable between arms at these sample sizes; the official FP8 measures a stronger leak propensity at the forced-token trigger slots. The ban kit makes the rate exactly zero.}

The bandwidth story, stated plainly: the 16-bit keep-set adds ~8% to the bytes each token reads, and this build is faster anyway — per-channel FP8 rides vLLM 0.25's fused-MoE kernels, and the full-precision draft head out-accepts the official's quantized one. On vLLM 0.24.x the kernel coverage is older and this build measures 83.0 tok/s plain (perplexity 4.7989 on BF16 KV) — still nicer than the official, no longer faster; serve ≥0.25. The NVFP4 sibling remains the maximum-speed arm of the pair (117.7 tok/s measured on 0.24).

Capability benchmarks are not re-run here; see Tencent's model card.

---

Serving

This build is designed and tested on 4× RTX PRO 6000 Blackwell (96 GB each) — the machine class it exists for.

Recommended: vLLM 0.25 or later (fast per-channel MoE kernels, profitable speculation, native reasoning parsing — all measured above).

vllm serve /path/to/Hy3-FP8-Dynamic \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.95 \
  --reasoning-parser hy_v3 \
  --speculative-config '{"method": "hy_v3_mtp", "num_speculative_tokens": 1, "draft_tensor_parallel_size": 1}'

• ~71 GiB of weights per GPU at TP4. The KV cache must be fp8 for full context: BF16 KV costs ~320 KiB/token — 21 GiB per card for one 256K stream, which does not fit beside the weights on a 96 GB card at any setting. fp8 KV halves that and is the scheme Tencent's own config declares for this checkpoint; the KV vectors it stores are produced by a compute path that is already 8-bit on either side. Measured at this exact configuration: 337,600-token KV pool — one full 262,144-token stream with 29% headroom — and exact verbatim recall from 162K-token prompts at both shallow and deep needle positions.
• First boot loads 285 GB and includes torch.compile — raise the readiness timeout (VLLM_ENGINE_READY_TIMEOUT_S=1800) so the engine is not killed mid-load.
• On vLLM 0.24.x: serve plain (drop the speculative config — speculation is net-negative there, see decision 4) and drop the reasoning parser (0.24's predates the namespaced tags; split client-side on the last </think:…> closer). From 0.25.0, vllm-project/vllm#47192 resolves the pair natively — parsed thinking arrives in the message's reasoning field.

---

Versus the official FP8 release

Both builds store FP8-E4M3 weights and serve full context on this machine. Every difference is a granularity or precision decision:

	Official FP8	This build
Weight scale granularity	per tensor (one scalar)	per output channel
Activation scaling	static, from calibration	dynamic, per token
Shared experts / dense-0	FP8	BF16
MTP draft head	quantized	BF16, byte-identical
Output head, embeds, router	mixed	BF16
Wikitext-2 perplexity, served identically	4.9133	4.8194
Plain / MTP decode, served identically (tok/s)	94.3 / 99.7	103.3 / 110.1
Size on disk	~300 GB	285 GB

---

Quantization details

Full build recipe and integrity gates

• llmcompressor 0.12, one oneshot pass, sequential pipeline, single modifier: FP8_DYNAMIC on attention (q/k/v/o) and routed experts; shared experts, dense-0 MLP, lm_head, embeddings, router gates, expert biases, and norms excluded.
• Weight scales are per-output-channel symmetric E4M3, computed directly from the weights (RTN — at 8-bit, rounding-to-nearest on a per-channel grid is exact enough that Hessian calibration measures no gain; the calibration corpus drives the memory-safe layer-by-layer pipeline, not the scales).
• Activation quantization is declared dynamic per-token in the config; no activation scales are stored — the fingerprint of this scheme is the absence of input_scale tensors, verifiable in the safetensors index.
• Integrity gates, all passed against the BF16 source: 45,824 FP8 weight/scale pairs with exact shapes; 1,314 tensors passed through byte-identical; zero missing, extra, or mismatched; 109,871,104 scale elements all finite and strictly positive; zero input_scale/k_scale/v_scale/weight_global_scale tensors anywhere; all 593 MTP-layer tensors byte-identical to source.
• 285 GB on disk, down from 557 GB BF16.

---

License and provenance

Apache-2.0. Tencent's LICENSE ships verbatim in this repository. The weights here are a modified version of tencent/Hy3: quantized as described above; they are not the original weights and are not affiliated with or endorsed by Tencent. "Tencent" and "Hy3" are used descriptively to identify the source model.

_{Quantization, forensics, and documentation by Auroter, with AI assistance (Claude); all measurements taken and verified on the hardware named above.}