Instructions to use coolbho3k/GLM-5.3-Flash-NVFP4-Optimized with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use coolbho3k/GLM-5.3-Flash-NVFP4-Optimized with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="coolbho3k/GLM-5.3-Flash-NVFP4-Optimized")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("coolbho3k/GLM-5.3-Flash-NVFP4-Optimized")
model = AutoModelForMultimodalLM.from_pretrained("coolbho3k/GLM-5.3-Flash-NVFP4-Optimized", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use coolbho3k/GLM-5.3-Flash-NVFP4-Optimized with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "coolbho3k/GLM-5.3-Flash-NVFP4-Optimized"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "coolbho3k/GLM-5.3-Flash-NVFP4-Optimized",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/coolbho3k/GLM-5.3-Flash-NVFP4-Optimized

SGLang

How to use coolbho3k/GLM-5.3-Flash-NVFP4-Optimized with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "coolbho3k/GLM-5.3-Flash-NVFP4-Optimized" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "coolbho3k/GLM-5.3-Flash-NVFP4-Optimized",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "coolbho3k/GLM-5.3-Flash-NVFP4-Optimized" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "coolbho3k/GLM-5.3-Flash-NVFP4-Optimized",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use coolbho3k/GLM-5.3-Flash-NVFP4-Optimized with Docker Model Runner:
```
docker model run hf.co/coolbho3k/GLM-5.3-Flash-NVFP4-Optimized
```

GLM-5.3-Flash NVFP4 Optimized

An experimental mixed-precision quantization of zai-org/GLM-5.3-Flash, derived from and compared against RedHatAI/GLM-5.3-Flash-NVFP4.

What changed

NVFP4 routed-expert weights with repaired shared W1/W3 scale handling.
Reconstruction-optimized NVFP4 scaling.
Selected dense, shared-expert, and attention weights retained in FP8.
Tensors unnecessarily converted to BF16 in the earlier export were restored to their native checkpoint representation.
Tokenizer, processor, chat template, and generation configuration are included.

These changes improve weight reconstruction error relative to the starting NVFP4 export.

Quantization methodology

We started with the Red Hat NVFP4 export and used Z.ai's FP8 checkpoint as the reference. For each routed-expert group, we tested nearby valid NVFP4 scales and kept the one with the lowest reconstruction error. On a representative sample of 72 matrices, mean relative weight RMSE improved from 9.1114% to 7.7185%.

We also restored passthrough tensors that had been expanded to BF16 back to their source FP8 values. W1 and W3 keep shared scales because Marlin serves them as one fused projection; the extra per-matrix scale tuning is applied to W2 only.

Serving

The tested two-node DGX Spark/SM121 recipe is available here:

https://github.com/coolbho3k/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark/tree/feature/exl3-ab-current

The recipe supports selectable NVFP4 or EXL3 weights, FP8 or native-FP4 KV cache, DCP2, and optional DFlash2 speculative decoding.

Limitations

The published serving results are specific to the repository's patched vLLM stack on two DGX Sparks.
Independent downstream benchmark evaluation is still recommended.
This is a community derivative and is not an official Z.ai or Red Hat release.

Acknowledgements

Thanks to Z.ai for GLM-5.3-Flash and Red Hat AI for the original NVFP4 checkpoint and quantization work.

Downloads last month: 60

Safetensors

Model size

162B params

Tensor type

F32

BF16

F8_E4M3

Model tree for coolbho3k/GLM-5.3-Flash-NVFP4-Optimized

Base model

zai-org/GLM-5.3-Flash

Quantized

(97)

this model