Hy4-NVFP4

Model Overview

  • Model Architecture: HYV4VLForConditionalGeneration
    • Input: Text, Image, Video
    • Output: Text
  • Model Optimizations:
    • Weight quantization: FP4
    • Activation quantization: FP4
  • Format: compressed-tensors
  • Base model: tencent/Hy4

This is an NVFP4 checkpoint derived from tencent/Hy4. It is optimized for Blackwell inference with NVFP4 weights and activations while retaining higher precision where needed for model quality.

Model Optimizations

This model uses NVFP4 FP4 (E2M1) weights and activations with two-level scaling (FP8 E4M3 scales per 16 elements plus a per-tensor FP32 global scale) for the routed experts of all 77 MoE layers. Layers outside of routed experts are left unquantized in BF16: attention and the DSA indexer, the dense first layer, shared experts, routers, the vision encoder and adapter, lm_head, and the DFly2 speculative-decoding drafter. This reduces the checkpoint from about 1.55 TB to about 447 GB.

The checkpoint uses LLM Compressor's compressed-tensors NVFP4 format and keeps the tensor names and config of the original checkpoint.

Deployment

vLLM Serving

vllm serve RedHatAI/Hy4-NVFP4 \
  --served-model-name hy4 \
  --tensor-parallel-size 4 \
  --dtype bfloat16 \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v4 \
  --tool-call-parser hy_v4 \
  --attention-backend FLASHMLA_SPARSE

Recommended sampling parameters are temperature=0.9, top_p=1.0. Reasoning effort accepts "no_think" and "high" (default) through chat_template_kwargs.

The BF16 DFly2 drafter weights are included in the checkpoint. vLLM currently enables --speculative-config '{"method":"dfly",...}' only for FP8 and MXFP4 Hy4 checkpoints.

Creation

The source model was compressed with LLM Compressor using the NVFP4 scheme on the routed experts, calibrated on 512 samples of mlabonne/open-perfectblend with a maximum sequence length of 2048, distributed over 4 GPUs. Expert weights use the nvfp4_expanded_imatrix observer.

Transformers' reverse weight-name conversion for hy_v4_vl is lossy, so the compressed model was saved with Transformers' internal names and then converted back to the original checkpoint layout (tensor names, config.json, tokenizer and processor files). The conversion was verified tensor by tensor: every non-expert tensor is bitwise identical to the base checkpoint. The DFly2 drafter, which Transformers does not model, was copied unchanged from the base checkpoint.

Creation script
import torch
from transformers import AutoProcessor, HYV4VLForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context

MODEL_ID = "tencent/Hy4"

# launched with torchrun --nproc-per-node=4; the default process group is initialized
# with a long timeout because rank 0 loads the full checkpoint before sharing it
with load_context(HYV4VLForConditionalGeneration):
    model = HYV4VLForConditionalGeneration.from_pretrained(
        MODEL_ID,
        dtype=torch.bfloat16,
        device_map="auto_offload",
        max_memory={"cpu": "700GiB"},
        offload_folder="offload_folder",
    )
processor = AutoProcessor.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
    scheme="NVFP4",
    ignore=["lm_head"],
)

oneshot(
    model=model,
    processor=processor.tokenizer,
    dataset="perfectblend",
    splits="train[:512]",
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
    batch_size=4,
    shuffle_calibration_samples=False,
)

model.save_pretrained("Hy4-NVFP4-raw", save_compressed=True, save_original_format=False)

Evaluation

This model was evaluated on GSM8K Platinum and GPQA Diamond using lighteval. Evaluation was run on a vLLM server hosted on 4xB300 with tensor parallelism 4, the hy_v4 reasoning parser and default (high) reasoning effort. Generation used temperature 0.9, top_p 1.0, one seed. NVFP4 and the tencent/Hy4-FP8 baseline used the same serving stack and harness. Recovery is relative to the FP8 baseline.

Accuracy

Category Benchmark RedHatAI/Hy4-NVFP4 tencent/Hy4-FP8 Recovery
Reasoning GSM8K Platinum (strict-match) 96.28% 96.77% 99.5%
GPQA Diamond (pass@1) 89.90% 91.41% 98.3%

Standard errors are about 0.5 points for GSM8K Platinum and 2.1 points for GPQA Diamond (198 questions, one seed), so the GPQA difference is within run-to-run noise. Most GSM8K Platinum misses for both models are answers that omit the requested ANSWER: line while stating the correct value.

Downloads last month
61
Safetensors
Model size
774B params
Tensor type
F32
·
BF16
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for RedHatAI/Hy4-NVFP4

Base model

tencent/Hy4
Quantized
(7)
this model