Instructions to use Inferact/Qwen3.8-27B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Inferact/Qwen3.8-27B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Inferact/Qwen3.8-27B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Inferact/Qwen3.8-27B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("Inferact/Qwen3.8-27B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Inferact/Qwen3.8-27B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Inferact/Qwen3.8-27B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Inferact/Qwen3.8-27B-NVFP4

SGLang

How to use Inferact/Qwen3.8-27B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Inferact/Qwen3.8-27B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Inferact/Qwen3.8-27B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Inferact/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Inferact/Qwen3.8-27B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/Inferact/Qwen3.8-27B-NVFP4
```

Runaway whitespace under JSON-schema structured output (vLLM/xgrammar) — not seen with the official FP8 checkpoint

by KoliaNik - opened 8 days ago

Discussion

KoliaNik

8 days ago

Heads-up rather than a bug report — this may be useful to other users of the checkpoint and possibly to whoever re-runs the calibration.

Serving Inferact/Qwen3.8-27B-NVFP4 (revision 6128240e) on vLLM 0.28.0 (2× RTX 5090, TP=2, default xgrammar structured-outputs backend, MTP speculative decoding), we see the following with response_format={"type": "json_schema", ...}: roughly a third of requests produce a well-formed JSON prefix, stop right after a key ("user_problems": in our schema), and then emit only spaces / tabs / newlines until max_tokens is exhausted. The response is HTTP 200 with finish_reason="length", so it looks like a successful answer to the client. Without the grammar (JSON by instruction, validated afterwards) the same prompts are fine 15/15.

What we could isolate over two maintenance windows:

It is not the engine flags: without speculative decoding 11/30, without async scheduling 7/30, baseline 12/30.
It is checkpoint-specific: the official Qwen/Qwen3.8-27B-FP8 (revision 017b9c7a) on the same engine, same flags, same schema and prompts did not do this once in 60 requests over two independent launches, with arrays populated.
The only server-side mitigation in vLLM (disable_any_whitespace) stops the runaway but makes this checkpoint emit empty arrays (29/30).

Full numbers and environment: https://github.com/vllm-project/vllm/issues/38696#issuecomment-5522658891 (the underlying engine issue is that xgrammar's JSON grammar allows an unbounded whitespace run, so a checkpoint whose whitespace-token probabilities are slightly shifted can stay "grammatical" forever).

Two questions, if you have a moment:

Did the calibration set include any JSON / code-like data? A plausible mechanism is that whitespace tokens in value positions got their probabilities shifted during NVFP4 calibration.
Have you seen this with other Qwen3.8 NVFP4 builds? We plan to test a couple more quants and can report back here.

Thanks for publishing the checkpoint — apart from this, it has been our production model since August.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment