Instructions to use Infatoshi/GLM-5.3-Flash-NVFP4-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Infatoshi/GLM-5.3-Flash-NVFP4-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Infatoshi/GLM-5.3-Flash-NVFP4-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Infatoshi/GLM-5.3-Flash-NVFP4-FP8")
model = AutoModelForMultimodalLM.from_pretrained("Infatoshi/GLM-5.3-Flash-NVFP4-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Infatoshi/GLM-5.3-Flash-NVFP4-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Infatoshi/GLM-5.3-Flash-NVFP4-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Infatoshi/GLM-5.3-Flash-NVFP4-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Infatoshi/GLM-5.3-Flash-NVFP4-FP8

SGLang

How to use Infatoshi/GLM-5.3-Flash-NVFP4-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Infatoshi/GLM-5.3-Flash-NVFP4-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Infatoshi/GLM-5.3-Flash-NVFP4-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Infatoshi/GLM-5.3-Flash-NVFP4-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Infatoshi/GLM-5.3-Flash-NVFP4-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Infatoshi/GLM-5.3-Flash-NVFP4-FP8 with Docker Model Runner:
```
docker model run hf.co/Infatoshi/GLM-5.3-Flash-NVFP4-FP8
```

GLM-5.3-Flash W4A16 NVFP4-FP8

This checkpoint is a mixed-precision quantization of zai-org/GLM-5.3-Flash. It uses MSE-calibrated NVFP4 weights with BF16 inputs for routed experts in MoE layers 6 through 41. It uses 128x128 block-scaled FP8 weights for routed experts in edge MoE layers 3 through 5 and 42 through 44. Other quantized linear layers also use block-scaled FP8 weights. The checkpoint keeps their activation inputs unquantized. NVIDIA Model Optimizer produced the checkpoint.

The project release gate passed. Each test compares this checkpoint with the exact upstream FP8 revision in the same Transformers implementation. The saved-artifact test dequantizes the stored FP8 and NVFP4 weights to BF16 before reload. The gate is a project rule. It is not a published standard.

Source and format

Item	Value
Upstream revision	`04c4e9e95c5da8862dced7e5056455116f83a7e0`
Upstream weight format	native FP8
Interior routed expert weights	NVFP4 E2M1, static FP8 scales, group size 16
Edge routed expert weights	128x128 block-scaled FP8 E4M3
NVFP4 routed expert inputs	BF16
FP8 routed expert inputs	Source precision
Other quantized linear weights	128x128 block-scaled FP8 E4M3
Other quantized linear inputs	Source precision
KV cache	Source precision
Export quantization method	ModelOpt `MIXED_PRECISION`
ModelOpt build	`0.0.1.dev1+g5db268251.d20260828`
ModelOpt base commit	`5db268251945b579d642bbd7449c63006cfa829f`
PTQ container	`nvcr.io/nvidia/pytorch:26.08-py3`
PTQ container image ID	`sha256:3becd068f49bd2ad38f90db5f9a4803019a76933a24e63d821376c44e7a9200a`
PTQ Transformers	`5.16.1`
NVFP4 routed expert layer entries	36
FP8 routed expert layer entries	6
FP8 layer entries	468
Weight shards	21
Weight bytes	201,823,044,552
Total file bytes before this card	201,872,724,688
Chat template SHA-256	`34d5ee66b12fa6446cdae131c352b8f68cd85369e0e6fda115583805fada3891`

The embedding, LM head, routers, recurrent state controls, MTP block, and visual modules remain unquantized. The visual path was not evaluated.

Calibration

Item	Value
Algorithm	MSE with FP8 scale sweep for static NVFP4 interior expert weights
Data mix quotas	OpenCodeReasoning 384, OpenMathReasoning 384, Nemotron-Science-v1 256
Dataset revisions	`20a1ca19`, `d3d08664`, `82e1af46`
Packed rows	1,024
Tokens per row	1,024
Total calibration tokens	1,048,576
Batch size	4
Hardware	4 NVIDIA B200 GPUs
Peak calibration memory	GPU 0: 119,350 MiB, GPU 1: 137,878 MiB, GPU 2: 137,938 MiB, GPU 3: 83,602 MiB

The loader collects 16 times each source quota, shuffles the combined source records with seed 0, then packs 1,024 rows. The code and math records use their source text fields. The science records use the upstream GLM chat template. Packing inserts EOS separators and fills each row to 1,024 tokens. The recipe directory contains the exact recipe, calibration manifest, ModelOpt base commit, local ModelOpt patch, PTQ command, and environment versions.

FP8 agreement

The reference uses the exact upstream FP8 checkpoint. The in-memory test and the saved-artifact reload use the same Transformers implementation and the same sequential device-map loader. The reload test reads the published packed weights, dequantizes them to BF16, and then runs the model. The temporary BF16 view is not part of this repository.

The sweep uses eight WikiText-2 samples, eight HumanEval samples, and eight GSM8K samples. It scores 6,507 held-out next-token positions. It also compares eight 48-token greedy generations. Sampling is disabled.

Runtime	Top-1 token agreement	Median first divergence	Temp-0 repeats
In-memory ModelOpt fake quant	95.42%	49	3 / 3 identical: true
Saved checkpoint reload	95.24%	35	3 / 3 identical: true

Held-out source	Tokens	Reload agreements	Agreement rate
WikiText-2	3,064	2,829	92.33%
GSM8K	1,226	1,171	95.51%
HumanEval	2,217	2,197	99.10%

Upstream FP8 margin	Tokens	Reload disagreements	Disagreement rate
margin < 0.5	368	153	41.58%
margin 0.5 to < 2	847	136	16.06%
margin 2 to < 5	1,294	21	1.62%
margin >= 5	3,998	0	0.00%

The median first-divergence value is censored at 49 when a generation matches all 48 compared tokens. The observed comparison positions have median 35.

The release gate requires at least 92% overall agreement, at most 2% disagreement when the FP8 margin is 2 to 5, at most 0.5% disagreement when the margin is at least 5, median first divergence at or after token 24, and three identical temperature-0 repeats. Both the in-memory and saved-artifact paths must pass.

The in-memory to saved-artifact diagnostic is 97.40% across 6,507 positions.

Saved-artifact reload

Item	Value
GPUs	4 NVIDIA B200, 183,359 MiB each
Runtime	Transformers 5.16.1 with ModelOpt source loader
Device map	sequential
GPU memory fraction	0.95
Packed-weight reload	NVFP4 and block-FP8 dequantized to BF16
Model load time	2905.47 seconds
Agreement scoring time	20.59 seconds
Eight-generation time	89.21 seconds
Peak GPU memory	GPU 0: 175,350 MiB, GPU 1: 179,764 MiB, GPU 2: 182,122 MiB, GPU 3: 145,748 MiB

The reload uses runtime/dequantize-modelopt-mixed-for-validation.py. It is an artifact integrity and quality test. It is not the recommended serving path.

Serving status

The pinned vLLM development build loaded the checkpoint on four B200 GPUs with pipeline split 15/9/9/12 and concurrency 1. The run was deterministic. Its cross-runtime top-1 agreement with the working Transformers FP8 control was 54.89%. The upstream vLLM control also produced invalid repeated token 1023 output. Therefore this card does not claim a validated vLLM serving recipe. The storage format and quality gate remain valid. A later vLLM GLM-5.3 integration can use the included runtime evidence.

Limits

The reference is upstream FP8, not BF16.
Interior routed expert weights are NVFP4. Six edge routed expert layers use block-scaled FP8. The saved-artifact gate dequantizes both formats to BF16 at load. It does not measure native FP4 kernel speed.
The agreement sweep is a stability test. It is not a task-accuracy score.
No HumanEval pass@1, agentic coding, long-context, or multimodal score is claimed.
The vLLM diagnostic uses context 1,024 and concurrency 1. It is not a passed serving gate.

License

The upstream checkpoint uses the MIT license. The upstream LICENSE file is included with this derivative.

Downloads last month: 27

Safetensors

Model size

183B params

Tensor type

F32

BF16

F8_E4M3

Model tree for Infatoshi/GLM-5.3-Flash-NVFP4-FP8

Base model

zai-org/GLM-5.3-Flash

Quantized

(70)

this model