Instructions to use JANGQ-AI/GLM-5.3-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JANGQ-AI/GLM-5.3-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="JANGQ-AI/GLM-5.3-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("JANGQ-AI/GLM-5.3-FP8")
model = AutoModelForCausalLM.from_pretrained("JANGQ-AI/GLM-5.3-FP8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JANGQ-AI/GLM-5.3-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JANGQ-AI/GLM-5.3-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JANGQ-AI/GLM-5.3-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/JANGQ-AI/GLM-5.3-FP8

SGLang

How to use JANGQ-AI/GLM-5.3-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JANGQ-AI/GLM-5.3-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JANGQ-AI/GLM-5.3-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JANGQ-AI/GLM-5.3-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JANGQ-AI/GLM-5.3-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use JANGQ-AI/GLM-5.3-FP8 with Docker Model Runner:
```
docker model run hf.co/JANGQ-AI/GLM-5.3-FP8
```

GLM-5.3-FP8

GLM-5.3 — ~756 GB on disk (down from the ~1.5 TB BF16 source) — FP8 E4M3 block-128 quantization, weight-only with dynamic activations. Runs natively on Hopper (H100 / H200) in vLLM via the DeepSeek-V3-style block-FP8 kernels — no Blackwell required.

Source: zai-org/GLM-5.3 (GLM-MoE-DSA architecture, 78 layers, 256 routed experts top-8, 1 shared expert, MLA + DeepSeek Sparse Attention, MTP head, 1M context)
Quantization: FP8 E4M3, weight-only static with per-128×128-block scales (weight_scale_inv, fp32); dynamic activation quantization at runtime — bit-for-bit the same scheme as the official zai-org/GLM-5.3 FP8 release and the DeepSeek-V3 block-FP8 format
Coverage: the exact quantized-tensor set of the official FP8 release (59,044 tensors). Verified 1:1 tensor-name match against both the BF16 and official-FP8 indexes
MTP included: the multi-token-prediction layer (model.layers.78.*, incl. eh_proj / enorm / hnorm / shared_head) is present and quantized identically to the official release
Bundle size: ~756 GB across 282 shards
Runs on: 8× H200 (or H100) with vLLM tensor-parallel

What's in the bundle

Module	Source dtype	Bundle dtype
Routed experts (256 × 3 mats × sparse layers)	BF16	FP8 E4M3 + F32 `weight_scale_inv` (block 128×128)
Attention (MLA q/kv proj, o_proj)	BF16	FP8 E4M3 + F32 block scales
Dense MLP (first 3 layers)	BF16	FP8 E4M3 + F32 block scales
MTP layer 78 (linear weights)	BF16	FP8 E4M3 + F32 block scales
RMSNorms, router gates, `e_score_correction_bias`	BF16 / F32	passthrough (BF16 / F32)
DSA indexer norms / proj, shared-expert gate	BF16	passthrough
`embed_tokens`, `lm_head`	BF16	passthrough

Dequant per block: w = weight_fp8 * weight_scale_inv, scale = amax / 448.0.

Usage (vLLM)

vllm serve JANGQ-AI/GLM-5.3-FP8 \
  --tensor-parallel-size 8 \
  --trust-remote-code

Enable MTP speculative decoding per vLLM's GLM / DeepSeek MTP docs for a decode speedup (the MTP weights are shipped here).

Method

Converted from the BF16 release with a streaming shard-by-shard quantizer (one tensor in memory at a time), so no GPU and no large host RAM are needed. The set of tensors to quantize is taken directly from the official FP8 index rather than re-derived heuristically, and the output config.json (quantization_config with modules_to_not_convert) is a verbatim copy of the official FP8 config — so vLLM treats this identically to the upstream release. The scheme is data-free (dynamic activation scales), so no calibration set is involved.

Quantized by JANGQ-AI.

Downloads last month: 33

Safetensors

Model size

753B params

Tensor type

BF16

F8_E4M3

F32

Model tree for JANGQ-AI/GLM-5.3-FP8

Base model

zai-org/GLM-5.3

Quantized

(28)

this model