Instructions to use RedHatAI/Hy4-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RedHatAI/Hy4-NVFP4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="RedHatAI/Hy4-NVFP4") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("RedHatAI/Hy4-NVFP4") model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/Hy4-NVFP4", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use RedHatAI/Hy4-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "RedHatAI/Hy4-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/Hy4-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/RedHatAI/Hy4-NVFP4
- SGLang
How to use RedHatAI/Hy4-NVFP4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "RedHatAI/Hy4-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/Hy4-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "RedHatAI/Hy4-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/Hy4-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use RedHatAI/Hy4-NVFP4 with Docker Model Runner:
docker model run hf.co/RedHatAI/Hy4-NVFP4
Hy4-NVFP4
Model Overview
- Model Architecture: HYV4VLForConditionalGeneration
- Input: Text, Image, Video
- Output: Text
- Model Optimizations:
- Weight quantization: FP4
- Activation quantization: FP4
- Format: compressed-tensors
- Base model: tencent/Hy4
This is an NVFP4 checkpoint derived from tencent/Hy4. It is optimized for Blackwell inference with NVFP4 weights and activations while retaining higher precision where needed for model quality.
Model Optimizations
This model uses NVFP4 FP4 (E2M1) weights and activations with two-level scaling (FP8 E4M3 scales
per 16 elements plus a per-tensor FP32 global scale) for the routed experts of all 77 MoE layers.
Layers outside of routed experts are left unquantized in BF16: attention and the DSA indexer, the
dense first layer, shared experts, routers, the vision encoder and adapter, lm_head, and the
DFly2 speculative-decoding drafter. This reduces the checkpoint from about 1.55 TB to about
447 GB.
The checkpoint uses LLM Compressor's compressed-tensors NVFP4 format and keeps the tensor names and config of the original checkpoint.
Deployment
vLLM Serving
vllm serve RedHatAI/Hy4-NVFP4 \
--served-model-name hy4 \
--tensor-parallel-size 4 \
--dtype bfloat16 \
--limit-mm-per-prompt '{"image":1,"video":0}' \
--enable-auto-tool-choice \
--reasoning-parser hy_v4 \
--tool-call-parser hy_v4 \
--attention-backend FLASHMLA_SPARSE
Recommended sampling parameters are temperature=0.9, top_p=1.0. Reasoning effort accepts
"no_think" and "high" (default) through chat_template_kwargs.
The BF16 DFly2 drafter weights are included in the checkpoint. vLLM currently enables
--speculative-config '{"method":"dfly",...}' only for FP8 and MXFP4 Hy4 checkpoints.
Creation
The source model was compressed with LLM Compressor
using the NVFP4 scheme on the routed experts, calibrated on 512 samples of
mlabonne/open-perfectblend
with a maximum sequence length of 2048, distributed over 4 GPUs. Expert weights use the
nvfp4_expanded_imatrix observer.
Transformers' reverse weight-name conversion for hy_v4_vl is lossy, so the compressed model was
saved with Transformers' internal names and then converted back to the original checkpoint layout
(tensor names, config.json, tokenizer and processor files). The conversion was verified tensor by
tensor: every non-expert tensor is bitwise identical to the base checkpoint. The DFly2 drafter, which
Transformers does not model, was copied unchanged from the base checkpoint.
Creation script
import torch
from transformers import AutoProcessor, HYV4VLForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context
MODEL_ID = "tencent/Hy4"
# launched with torchrun --nproc-per-node=4; the default process group is initialized
# with a long timeout because rank 0 loads the full checkpoint before sharing it
with load_context(HYV4VLForConditionalGeneration):
model = HYV4VLForConditionalGeneration.from_pretrained(
MODEL_ID,
dtype=torch.bfloat16,
device_map="auto_offload",
max_memory={"cpu": "700GiB"},
offload_folder="offload_folder",
)
processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = QuantizationModifier(
targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
scheme="NVFP4",
ignore=["lm_head"],
)
oneshot(
model=model,
processor=processor.tokenizer,
dataset="perfectblend",
splits="train[:512]",
recipe=recipe,
max_seq_length=2048,
num_calibration_samples=512,
batch_size=4,
shuffle_calibration_samples=False,
)
model.save_pretrained("Hy4-NVFP4-raw", save_compressed=True, save_original_format=False)
Evaluation
This model was evaluated on GSM8K Platinum and GPQA Diamond using
lighteval. Evaluation was run on a vLLM server hosted
on 4xB300 with tensor parallelism 4, the hy_v4 reasoning parser and default (high) reasoning
effort. Generation used temperature 0.9, top_p 1.0, one seed. NVFP4 and the
tencent/Hy4-FP8 baseline used the same serving stack and
harness. Recovery is relative to the FP8 baseline.
Accuracy
| Category | Benchmark | RedHatAI/Hy4-NVFP4 | tencent/Hy4-FP8 | Recovery |
|---|---|---|---|---|
| Reasoning | GSM8K Platinum (strict-match) | 96.28% | 96.77% | 99.5% |
| GPQA Diamond (pass@1) | 89.90% | 91.41% | 98.3% |
Standard errors are about 0.5 points for GSM8K Platinum and 2.1 points for GPQA Diamond (198
questions, one seed), so the GPQA difference is within run-to-run noise. Most GSM8K Platinum misses
for both models are answers that omit the requested ANSWER: line while stating the correct value.
- Downloads last month
- 61
Model tree for RedHatAI/Hy4-NVFP4
Base model
tencent/Hy4