Instructions to use RedHatAI/gemma-4-31B-it-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/gemma-4-31B-it-FP8-dynamic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("RedHatAI/gemma-4-31B-it-FP8-dynamic")
model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/gemma-4-31B-it-FP8-dynamic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/gemma-4-31B-it-FP8-dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/gemma-4-31B-it-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/gemma-4-31B-it-FP8-dynamic

SGLang

How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/gemma-4-31B-it-FP8-dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/gemma-4-31B-it-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/gemma-4-31B-it-FP8-dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/gemma-4-31B-it-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/gemma-4-31B-it-FP8-dynamic with Docker Model Runner:
```
docker model run hf.co/RedHatAI/gemma-4-31B-it-FP8-dynamic
```

Can you add more details to the model card?

by bullerwins - opened Apr 11

Discussion

bullerwins

Apr 11

Like creation code, like in the FP8-Block version
https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-block

bdellabe

Red Hat AI org Apr 13

Hi @bullerwins , added to bottom of model card. it's the same pathway but with different scheme

qenme

Jun 3

•

edited Jun 3

Which is more accurate, this or the block quant?

bdellabe

Red Hat AI org Jun 3

Hi @qenme , the best choice between FP8 block and this is really dependent on your hardware. The benchmarks are preliminary, but you can see it achieves near-recovery with the dense model for the eval task at hand. The NVFP4 checkpoint is targeting NVIDIA GPUs with the Blackwell architecture

qenme

Jun 4

•

edited Jun 4

Hi @qenme , the best choice between FP8 block and this is really dependent on your hardware. The benchmarks are preliminary, but you can see it achieves near-recovery with the dense model for the eval task at hand. The NVFP4 checkpoint is targeting NVIDIA GPUs with the Blackwell architecture

Thanks for the response, that makes sense.

Prefacing with a sorry for the long wall of text incoming.

Second question, a little bit outside of the scope but while I have your attention... I've benchmarked many Gemma 4 31B variants trying to identify which ones have best KL divergence and same top p agreement %, but I am consistently noticing this model does not like KV cache or model quant. All of my tests were limited to llama-cpp because I don't know how to get these statistics from VLLM. I compared BF16 model + BF16 KV cache to every other variant mixing in 8 bit kv cache or 8 bit model, and they all have very bad results. Which is why I am surprised when I see accuracy benchmarks show that it's able to meet or get close to BF16 accuracy. I've seen others online also showing Gemma 4 is very sensitive to KV cache and model weight quant, so I don't think the issue is llama-cpp's statistics tools. I've even encountered strange outputs in many 8 bit quants which leads me to believe it's a real issue. I really like the model at BF16 but I can't run it at a big KV cache size. Unfortunately, this is very much outside my realm of expertise, so I am wondering if you've encountered this or know how to test for this in vLLM? Thanks again for your time and quanting the models!

bdellabe

Red Hat AI org 25 days ago

Hi @qenme , if you are running via llama-cpp support will be limited. vllm benchmark provides a lot of statistics, but you'd have to look into the docs to find what you're looking for, or alternatively https://github.com/vllm-project/guidellm

qenme

22 days ago

Hi @qenme , if you are running via llama-cpp support will be limited. vllm benchmark provides a lot of statistics, but you'd have to look into the docs to find what you're looking for, or alternatively https://github.com/vllm-project/guidellm

Thank you :)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment