avtc/Hy3-GPTQ-RTN-4bit-tp8

Tencent Hy3 GPTQ RTN 4bit tp8 (MPT layer removed)

This repository contains a 4-bit RTN-quantized version of the Tencent Hy3 Mixture-of-Experts model.

Hy3 is a 295B-parameter MoE model with 21B active parameters, 192 experts (top-8 routed), and a 256K context window. Quantizing it to 4-bit brings the weight footprint down to ~150 GB, making it possible to serve the full 256K context on a single 8x24 GB node with TurboQuant KV cache compression.

⚠️ Required vLLM Branch

This model requires a patched vLLM build. Stock vLLM cannot load it due to several bugs in weight-name remapping, dynamic GPTQ MoE group-size handling, and TurboQuant KV cache shaping:

➡️ avtc/vllm @ fix/load-quantized-hy3

The branch includes these fixes:

Commit	Fix
a1ca5aff6	Remap e_score_correction_bias→expert_bias for transformers 5.x checkpoint renames
80437c75e	Respect dynamic group_size=64 override for GPTQ MoE Marlin support + WNA16 fallback
c2905928f	Fix shared_mlp weight prefix so shared-expert weights load correctly
4e3126f5f	Use WeightsMapper for HY3 transformers weight renames + quant detection
325a5ddd7	Disable TurboQuant boundary skip layers (mixed-backend KV cache allocation crash)
33a482b71	Fix layer_cache_dtype_str for TurboQuant in KV cache reshape (TQFullAttentionSpec has kv_quant_mode=NONE)

Build from source:

git clone -b fix/load-quantized-hy3 https://github.com/avtc/vllm.git
cd vllm
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
TORCH_CUDA_ARCH_LIST="8.6" uv pip install --editable . --torch-backend=auto

Set TORCH_CUDA_ARCH_LIST to your GPU's compute capability to skip unused architectures and speed up the build (e.g. 8.6 for RTX 3090, 8.9 for RTX 4090, 9.0 for H100).

Quantization Details

Quantized with GPTQModel (7.2.0) using the RTN (Round-To-Nearest) fallback strategy.

Property	Value
Quant method	GPTQ (RTN fallback)
Bits	4
Symmetric	true
desc_act	false
Global group size	128 (dense layers, attention projections)
Expert / shared-MLP group size	64 (dynamic override)
Fallback threshold	0.5%
Pack dtype	int32

Verified Hardware & Performance

This model is verified to run with Tensor Parallel on 8x NVIDIA RTX 3090 (24 GB each) GPUs with the full 256K (262,144) context window, using TurboQuant 4-bit KV cache compression.

• Tensor parallel: 8 (no expert parallel — expert group size 64 divides cleanly across 8 GPUs)
• KV cache: turboquant_4bit_nc
• Max model len: 262,144
• Model weight footprint: ~150 GB
• Reasoning (<think>) and tool-calling work.

⚠️ TurboQuant requires FlashAttention 2 (it is incompatible with FlashAttention 3).

Quick Start

export VLLM_SLEEP_WHEN_IDLE=1
export TORCH_CUDA_ARCH_LIST="8.6"
export CUDA_VISIBLE_DEVICES=4,5,6,7,0,1,2,3
export RAY_memory_monitor_refresh_ms=0
export NCCL_CUMEM_ENABLE=0
export VLLM_ENABLE_CUDAGRAPH_GC=1

export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export VLLM_FLOAT32_MATMUL_PRECISION=high
export OMP_NUM_THREADS=1

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve /path/to/tencent-Hy3-RTN \
  -tp 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3 \
  --port 8000 \
  --host 0.0.0.0 \
  --uvicorn-log-level info \
  --trust-remote-code \
  --max-num-seqs 4 \
  --seed 1234 \
  --max-model-len 262144 \
  --skip-mm-profiling \
  -O3 \
  --no-use-tqdm-on-load \
  --default-chat-template-kwargs '{"interleaved_thinking": true}' \
  --performance-mode balanced \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048 \
  --enable-prefix-caching \
  --kv-cache-dtype turboquant_4bit_nc \
  --gpu-memory-utilization 0.935

Note: with fp8 kv cache inference is twice faster on 8x3090, but max context window is smaller:

    --max-model-len 143000 \
    --gpu-memory-utilization 0.945 \
    --kv-cache-dtype fp8_e4m3 \

Then call the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy3",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    # reasoning_effort: "no_think" (default, direct response),
    #                   "low", "high" (deep chain-of-thought)
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

Recommended sampling parameters: temperature=0.9, top_p=1.0.

Reasoning mode: Set reasoning_effort to "high" for complex tasks (math, coding, reasoning) or "no_think" for direct responses.

Verified Functionality

• ✅ Functional 2D HTML aquarium generation (without reasoning enabled) — working artifact.
• ✅ Reasoning mode (<think>) engages and produces output.

Original Model

For full details on the architecture, benchmarks, and capabilities of the base model, see the original model card: ➡️ tencent/Hy3

Property	Value
Architecture	Mixture-of-Experts (MoE)
Total Parameters	295B
Activated Parameters	21B
Number of Layers	80 (MTP layer was removed during quantization)
Attention Heads	64 (GQA, 8 KV heads, head dim 128)
Hidden Size	4096
Intermediate Size	13312 (dense), 1536 (expert)
Context Length	256K
Vocabulary Size	120,832
Number of Experts	192, top-8 activated
License	Apache 2.0

Acknowledgments

• Tencent Hunyuan Team for the original Hy3 model.
• GPTQModel team for the quantization toolkit.
• vLLM team for the inference engine.

License

This model inherits the Apache License 2.0 of the base model. See the original license.