Instructions to use RiverRider/gemma-4-31B-it-nf4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RiverRider/gemma-4-31B-it-nf4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RiverRider/gemma-4-31B-it-nf4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("RiverRider/gemma-4-31B-it-nf4")
model = AutoModelForMultimodalLM.from_pretrained("RiverRider/gemma-4-31B-it-nf4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RiverRider/gemma-4-31B-it-nf4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RiverRider/gemma-4-31B-it-nf4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RiverRider/gemma-4-31B-it-nf4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RiverRider/gemma-4-31B-it-nf4

SGLang

How to use RiverRider/gemma-4-31B-it-nf4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RiverRider/gemma-4-31B-it-nf4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RiverRider/gemma-4-31B-it-nf4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RiverRider/gemma-4-31B-it-nf4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RiverRider/gemma-4-31B-it-nf4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RiverRider/gemma-4-31B-it-nf4 with Docker Model Runner:
```
docker model run hf.co/RiverRider/gemma-4-31B-it-nf4
```

gemma-4-31B-it, NF4

google/gemma-4-31B-it quantised to 4-bit NF4 with bitsandbytes, double quantisation on, bfloat16 compute. Nothing else is changed. Use of this model is governed by the Gemma Terms of Use.

It exists so that a Space can load the host in about 17 GB instead of pulling 62 GB and quantising at startup.

Why this exact quantiser

We read the layer-47 hidden state out of this model and hand it to a separate 0.6B decoder and a linear retrieval head, both of which were fitted on bf16 states. So the question is whether quantisation moves the state out of the distribution they expect. We measured it on 100 COCO val2017 images.

Anisotropy on this backbone is 0.9874, so raw cosine is not usable: matched bf16/NF4 pairs read 0.9986 while mismatched pairs, different pictures across the two precisions, read 0.9867. Centered on the pool mean, matched pairs are 0.9631 (worst 0.9056) against a mismatched floor of 0.0237.

	bf16	NF4
distance to the reader's training mean	13.43	14.51
self-retrieval into the 2000-image bf16 index, r@1	1.000	1.000
top-10 neighbour agreement with bf16	1.000	0.704

Retrieval survives: an NF4 query lands on its own image at rank 1 in 100 of 100 cases. Deeper ranking does move, so the two indices are not interchangeable. Verbalisation is content-stable but not token-stable: 8 of 8 sampled pairs name the same subject and 2 of 8 are identical strings, so any quoted example sentence should say which precision produced it.

AWQ and GGUF builds of this model exist and are probably fine, but they were not the ones we measured.

Downloads last month: -

Safetensors

Model size

31B params

Tensor type

BF16

Model tree for RiverRider/gemma-4-31B-it-nf4

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Quantized

(307)

this model

RiverRider
/

gemma-4-31B-it-nf4

gemma-4-31B-it, NF4

Why this exact quantiser

Model tree for RiverRider/gemma-4-31B-it-nf4

Space using RiverRider/gemma-4-31B-it-nf4 1