Instructions to use LilaRest/gemma-4-31B-it-NVFP4-turbo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LilaRest/gemma-4-31B-it-NVFP4-turbo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained("LilaRest/gemma-4-31B-it-NVFP4-turbo")
model = AutoModelForCausalLM.from_pretrained("LilaRest/gemma-4-31B-it-NVFP4-turbo", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LilaRest/gemma-4-31B-it-NVFP4-turbo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LilaRest/gemma-4-31B-it-NVFP4-turbo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LilaRest/gemma-4-31B-it-NVFP4-turbo

SGLang

How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LilaRest/gemma-4-31B-it-NVFP4-turbo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LilaRest/gemma-4-31B-it-NVFP4-turbo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LilaRest/gemma-4-31B-it-NVFP4-turbo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LilaRest/gemma-4-31B-it-NVFP4-turbo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use LilaRest/gemma-4-31B-it-NVFP4-turbo with Docker Model Runner:
```
docker model run hf.co/LilaRest/gemma-4-31B-it-NVFP4-turbo
```

Is this quant support image recognition?

by alexcardo - opened Apr 9

Discussion

alexcardo

Apr 9

"This variant is text-only, video/audio weights and encoders have been stripped. If you need video/audio support -> open an issue or PR."

What about images? Nobody cares about audio/video. Images are most important.

Thnx.

LilaRest

Owner Apr 9

Hi @alexcardo ! Let's see if more people care about that, and I'll consider spending time to ship an image and/or audio variants.

vimasamo

Apr 11

Hi! I'm interested in images only for OCR capabilities, is it possible? Or at least text/image to text.

How can I help?

TYIA

LilaRest

Owner Apr 11

This comment has been hidden (marked as Resolved)

Fajkel

Apr 11

Hey! First off, great work on the Turbo variant — I've been running it on an RTX PRO 6000 and it's noticeably faster than NVIDIA's original checkpoint (~40 tok/s vs ~32 tok/s for the same prompt), and the quality difference is barely noticeable. However, being able to send images alongside text is a big deal for my use case, and I'd rather not fall back to the full 31 GiB NVIDIA checkpoint just for that.

Would it be feasible to keep the vision encoder while still maintaining the smaller footprint and speed gains? Even if it adds a few GiB, the tradeoff would be worth it.

Thanks again for putting this together!

LilaRest

Owner Apr 11

Thanks @Fajkel ! the primary use case for turbo was maximizing text throughput (most use cases need mainly text). I'm extra busy right now, I'll wait a few more days, and if enough people show interest i’ll ship a multimodal variant

Narutoouz

Apr 12

•

edited Apr 12

People who want multimodal variant, show your interest and like to this developer's comment. Show him you that many people want multimodal variant, thankyou

Thanks @Fajkel ! the primary use case for turbo was maximizing text throughput (most use cases need mainly text). I'm extra busy right now, I'll wait a few more days, and if enough people show interest i’ll ship a multimodal variant

pastoriomarco

Apr 13

I'd be interested too, currently looking for a model with multimodal capabilities for robotics, I'd be running it on Jetson Thor

QueryQuery

Apr 13

+1 for image analysis. Thanks for everything!

lalit-dumka

Apr 14

Thanks @Fajkel ! the primary use case for turbo was maximizing text throughput (most use cases need mainly text). I'm extra busy right now, I'll wait a few more days, and if enough people show interest i’ll ship a multimodal variant

Yes. I am also looking forward for a video-text-text as well to be run on Jetson Thor. Thanks!

necroyancer

Apr 16

Thanks for the lovely quant! Vision support would be really appreciated <3

tftranslate

May 15

Thank you ever so much for releasing this model. I achieve 10.8 to 11 tok/s on a DGX Spark OEM with GB10. That makes the dense model work for me (before, Q8_UD_K_X_L was crawling at 6 tok/s), and I am really grateful for that, because for my task its capabilities far exceed the 26b a4e version.
Here is a vote: I would also like to have a mmproj file for image processing (generating image descriptions in a docling workflow, ocr, etc.).

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment