Instructions to use LibertAIDAI/Qwen3.8-27B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LibertAIDAI/Qwen3.8-27B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="LibertAIDAI/Qwen3.8-27B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("LibertAIDAI/Qwen3.8-27B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("LibertAIDAI/Qwen3.8-27B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LibertAIDAI/Qwen3.8-27B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LibertAIDAI/Qwen3.8-27B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LibertAIDAI/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/LibertAIDAI/Qwen3.8-27B-NVFP4

SGLang

How to use LibertAIDAI/Qwen3.8-27B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LibertAIDAI/Qwen3.8-27B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LibertAIDAI/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LibertAIDAI/Qwen3.8-27B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LibertAIDAI/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use LibertAIDAI/Qwen3.8-27B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/LibertAIDAI/Qwen3.8-27B-NVFP4
```

Qwen3.8-27B-NVFP4

NVFP4 post-training quantization of Qwen/Qwen3.8-27B, produced with NVIDIA ModelOpt. This is the safetensors source checkpoint. If you want ready-to-run llama.cpp files, use Qwen3.8-27B-NVFP4-GGUF or the MTP variant, both converted from this checkpoint.

We publish it because no recipe-transparent NVFP4 source existed for this model. The exact recipe is below so the result can be audited or reproduced.

What is quantized

NVFP4, group size 16, weights and activations, on the language-model MLP linears only:

model.language_model.layers.{0..63}.mlp.{gate,up,down}_proj

That is 192 quantized linears, 64 layers by three projections. This follows NVIDIA's recipe for dense models rather than quantizing everything reachable.

Kept in BF16:

model.language_model.layers.*.self_attn.*, the 16 full-attention layers
model.language_model.layers.*.linear_attn.*, the 48 Gated-DeltaNet layers
model.visual.*, the entire vision tower
mtp.*, the multi-token-prediction block
lm_head.weight and model.language_model.embed_tokens.weight
all norms and the Mamba conv1d weights

KV cache quantization is not enabled, so this checkpoint carries no k_scale or v_scale calibration artifacts.

Recipe

Tool: NVIDIA ModelOpt 0.45.0
Stack: transformers 5.10.1, torch 2.11.0+cu128
Calibration: 512 samples from abisee/cnn_dailymail 3.0.0, sequence length 512, max algorithm
Config: mtq.NVFP4_DEFAULT_CFG with disable rules appended for *self_attn*, *linear_attn*, *visual*, *vision*, *embed*, *lm_head* and *mtp*
Hardware: single RTX 5090 (32 GB, sm_120) with accelerate CPU offload, since the BF16 source is 55.6 GB
Wall clock: 18.7 minutes, of which 17 minutes was calibration

The MTP block

transformers declares _keys_to_ignore_on_load_unexpected = [r"^mtp.*"] for the qwen3_5 architecture, so the MTP weights are dropped silently when the model is loaded and any straightforward export omits them entirely, without raising an error.

The 15 mtp.* tensors here were re-injected unquantized in BF16 after export, written to model-mtp.safetensors and referenced from model.safetensors.index.json. Keep that file alongside the main shards. If it goes missing, downstream tooling will load the model without speculative decoding and give no warning.

Caveats

Calibration was English news text, matching NVIDIA's published choice. Multilingual and code-heavy workloads were not measured against a held-out set.
The vision tower is BF16 and was not separately evaluated. Vision quality should track the upstream BF16 release closely.
This checkpoint was validated by converting it to GGUF and benchmarking there. It has not been tested under vLLM or TensorRT-LLM.

License and attribution

Inherits Apache 2.0 from Qwen/Qwen3.8-27B. All rights, responsibilities and acceptable-use policies of the upstream license apply.

Quantization performed by LibertAI.

Downloads last month: -

Safetensors

Model size

19B params

Tensor type

BF16

Model tree for LibertAIDAI/Qwen3.8-27B-NVFP4

Base model

Qwen/Qwen3.8-27B

Quantized

(691)

this model