Instructions to use LilaRest/gemma-4-31B-it-NVFP4-turbo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LilaRest/gemma-4-31B-it-NVFP4-turbo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained("LilaRest/gemma-4-31B-it-NVFP4-turbo")
model = AutoModelForCausalLM.from_pretrained("LilaRest/gemma-4-31B-it-NVFP4-turbo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LilaRest/gemma-4-31B-it-NVFP4-turbo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LilaRest/gemma-4-31B-it-NVFP4-turbo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LilaRest/gemma-4-31B-it-NVFP4-turbo

SGLang

How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LilaRest/gemma-4-31B-it-NVFP4-turbo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LilaRest/gemma-4-31B-it-NVFP4-turbo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LilaRest/gemma-4-31B-it-NVFP4-turbo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LilaRest/gemma-4-31B-it-NVFP4-turbo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with Docker Model Runner:
```
docker model run hf.co/LilaRest/gemma-4-31B-it-NVFP4-turbo
```

model with opencode & claude code

by soslowboy100 - opened Apr 10

Discussion

soslowboy100

Apr 10

Thanks a lot for your effort with this model, it is exactly what i was looking for.
I have a 5090 and using vllm/vllm-openai:cu130-nightly as you mentioned. I am running archlinux.

I am using it with the latest opencode. Unfortunately, while it starts fine and it is at
least 20% faster than llama , very fast it gets stuck where the opencode showing it is
working but vllm does nothing. Trying to stop and restarting it does nothing.
I also tried claude code and it shows similar behavior so it must be something with vllm and the model.

Do you have any ideas on what might be going wrong? Also, i run the vllm with
--max-model-len 131072 --max-num-seqs 4, without seeing any memory issues (no monitor is plugged in the 5090).

JohnUser

Apr 10

Doesn't happen to me. I'm using --max-model-len auto:

vllm serve LilaRest/gemma-4-31B-it-NVFP4-turbo
--quantization modelopt
--kv-cache-dtype fp8
--gpu-memory-utilization 0.95
--max-model-len auto
--max-num-seqs 128
--max-num-batched-tokens 8192
--enable-prefix-caching
--reasoning-parser gemma4
--trust-remote-code
--tool-call-parser gemma4
--enable-auto-tool-choice

soslowboy100

Apr 11

Hi JohnUser,
Are you using it like you mentioned with a 5090? When i run it like you mentioned i get a torch.OutOfMemoryError
Have to decrease the max-num-seqs to 2 to run, and set the --gpu-memory-utilization to 0.85 so that [autotuner] doesn't complain with OOM. Still getting the hangs without any log or warning.
Thanks

LilaRest

Owner Apr 11

@soslowboy100 Thanks for trying it! Your --max-model-len 131072 is likely the issue. On a 5090 (32GB), context tops out around ~25K for this model. At 131K the KV cache will overcommit and vLLM can silently hang instead of OOMing cleanly. Try --max-model-len 16384 (or 25000 max) and see if the stalling goes away. Let me know how it goes.

JohnUser

Apr 11

•

edited Apr 11 by

LilaRest

Are you using it like you mentioned with a 5090? When i run it like you mentioned i get a torch.OutOfMemoryError

Yep, my GPU is ZOTAC GAMING GeForce RTX 5090 Solid OC White Edition, 32768 MB GDDR7
I am running it on a fresh dedicated Ubuntu Server machine, with a Threadripper CPU and Asus TRX50 motherboard, 64 gb of ram.

soslowboy100

Apr 12

@soslowboy100 Thanks for trying it! Your --max-model-len 131072 is likely the issue. On a 5090 (32GB), context tops out around ~25K for this model. At 131K the KV cache will overcommit and vLLM can silently hang instead of OOMing cleanly. Try --max-model-len 16384 (or 25000 max) and see if the stalling goes away. Let me know how it goes.

I am pretty sure it will work fine when using 16384, and honesly my question is more related to vllm tuning than your model.
When i used --max-model-len auto with a low --max-num-seqs 2, vllm was utilizing quite high context >100k. The 131k i mentioned was just to have a concrete example.

For my use case, single user using opencode, what would be the best parameters to maximize the available context?
I was hoping that the 25k you mention in the model card is due to the high number of parallel requests you want to support.
Thus, if you lower the parallel requests to 1-4, you could have way higher context.
For reference, with llama.cpp and UD-Q5_K_XL which is 2.5G larger, i can easily fit more than 100k context, thus i was assuming i
could do something similar with vllm.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment