Instructions to use axteam/GLM-4-9B-0414-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/GLM-4-9B-0414-FP8-dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="axteam/GLM-4-9B-0414-FP8-dynamic")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("axteam/GLM-4-9B-0414-FP8-dynamic")
model = AutoModelForCausalLM.from_pretrained("axteam/GLM-4-9B-0414-FP8-dynamic", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/GLM-4-9B-0414-FP8-dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/GLM-4-9B-0414-FP8-dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4-9B-0414-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/GLM-4-9B-0414-FP8-dynamic

SGLang

How to use axteam/GLM-4-9B-0414-FP8-dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/GLM-4-9B-0414-FP8-dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4-9B-0414-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/GLM-4-9B-0414-FP8-dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4-9B-0414-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use axteam/GLM-4-9B-0414-FP8-dynamic with Docker Model Runner:
```
docker model run hf.co/axteam/GLM-4-9B-0414-FP8-dynamic
```

GLM-4-9B-0414 · FP8 (dynamic)

8-bit quantization of zai-org/GLM-4-9B-0414 for vLLM serving. Quantized by axteam.


Base model	`zai-org/GLM-4-9B-0414`
Method	FP8 (dynamic)
Format	compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`)
Weights / activations	W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations
Architecture	Dense (GLM-4 · Glm4ForCausalLM) · 9.4B · text-only
Size on disk	~10.6 GB
Calibration	None (data-free)
License	mit

Quantization details

Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors float-quantized (scheme FP8_DYNAMIC).
Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized.
Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic (this repo)	NVFP4A16
MMLU (5-shot)	73.99	74.01 (100.0%)	73.23 (99.0%)
GSM8K (5-shot, strict)	80.21	80.89 (100.9%)	78.17 (97.4%)
ARC-Challenge (0-shot)	60.15	59.64 (99.1%)	58.36 (97.0%)
Winogrande (5-shot)	78.85	79.24 (100.5%)	77.82 (98.7%)
KMMLU (5-shot, direct)	48.34	48.26 (99.8%)	48.05 (99.4%)
HAE-RAE (0-shot)	55.36	55.18 (99.7%)	56.10 (101.3%)
Average	66.15	66.21 (100.1%)	65.29 (98.7%)

Requirements


Runtime	vLLM — the architecture is in vLLM ≥ 0.8.4; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. GLM-4-0414 runs in bf16 only (float16 is disabled for this architecture), so Ampere (sm80) or newer in practice. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 10.6 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU.
Transformers (without vLLM)	`transformers>=4.56.0` + `compressed-tensors>=0.15.0` (the architecture exists from 4.51.3, but these compressed-tensors checkpoints cannot load before 4.56).
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/GLM-4-9B-0414-FP8-dynamic --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/GLM-4-9B-0414-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/GLM-4-9B-0414-FP8-dynamic", max_model_len=32768)
params = SamplingParams(max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/GLM-4-9B-0414-FP8-dynamic --kv-cache-dtype fp8

License

Quantized derivative of zai-org/GLM-4-9B-0414 (mit).

한국어 요약

zai-org/GLM-4-9B-0414의 FP8 (W8A8) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	`zai-org/GLM-4-9B-0414`
양자화 방식	FP8 동적 양자화 (W8A8)
저장 형식	compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`)
모델 구조	Dense (GLM-4, `Glm4ForCausalLM`) · 9.4B · 텍스트 전용
용량	약 10.6 GB
보정 데이터	없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산)

양자화 상세

가중치: FP8(E4M3), 출력 채널별 스케일
활성화: 실행 시 토큰별 동적 FP8
양자화 절차: PTQ(학습 후 양자화, 재학습 없음), 가중치 스케일은 미리 계산하고 활성화 스케일은 실행 시 토큰마다 계산
BF16 유지: 토큰 임베딩, LM 헤드, 모든 RMSNorm
양자화 대상: 어텐션·MLP 선형층(projection)
원본 파일: 토크나이저·chat template은 원본과 동일

품질 (실측)

벤치마크 평균: 66.21 (원본 66.15, 100.1%) — MMLU·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 few-shot·greedy, 모든 빌드를 같은 설정으로 측정
한국어 벤치마크: KMMLU 48.26 (원본 48.34, 99.8%) · HAE-RAE 55.18 (원본 55.36, 99.7%)

요구사항

실행 엔진: vLLM 0.8.4 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인
vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
GPU: NVIDIA Turing(sm75, 예: T4) 이상, GLM-4-0414는 bf16 전용(float16 비활성)이라 실질적으로 Ampere(sm80) 이상, V100(sm70) 미지원 — FP8 연산은 Ada·Hopper·Blackwell(sm89 이상)에서 수행, Turing·Ampere는 vLLM이 가중치 전용 FP8로 자동 전환 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
메모리: 가중치 10.6 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
Transformers로 직접 사용: transformers 4.56.0 이상 + compressed-tensors 0.15.0 이상 (구조 자체는 4.51.3부터 있으나 이 compressed-tensors 체크포인트는 4.56 이상에서만 로드 가능)
trust_remote_code: 불필요

사용법

서버 실행: vllm serve axteam/GLM-4-9B-0414-FP8-dynamic --max-model-len 32768 — OpenAI 호환 API
Python: LLM(model="axteam/GLM-4-9B-0414-FP8-dynamic", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

라이선스

라이선스: MIT — 원본 모델의 라이선스를 그대로 따름
권리: 원본 모델의 권리는 Z.ai에게 있음, 이 리포는 양자화 파생물

Downloads last month: 109

Safetensors

Model size

9B params

Tensor type

BF16

F8_E4M3

Model tree for axteam/GLM-4-9B-0414-FP8-dynamic

Base model

zai-org/GLM-4-9B-0414

Quantized

(28)

this model