axteam/GLM-4.7-Flash-FP8-dynamic

GLM-4.7-Flash · FP8 (dynamic)

8-bit quantization of zai-org/GLM-4.7-Flash for vLLM serving. Quantized by axteam.

Base model	zai-org/GLM-4.7-Flash
Method	FP8 (dynamic)
Format	compressed-tensors float-quantized (scheme FP8_DYNAMIC)
Weights / activations	W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations
Architecture	MoE (GLM-4.7 Lite · Glm4MoeLite) · 31.2B · 202K context · text-only
Size on disk	~32.2 GB
Calibration	None (data-free)
License	mit

Quantization details

• Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors float-quantized (scheme FP8_DYNAMIC).
• Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized. In addition, the MoE router (mlp.gate, incl. e_score_correction_bias) is kept in BF16. The MTP layer (model.layers.47: eh_proj, shared_head.head) is quantized like the other layers.
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Requirements

Runtime	vLLM — the architecture is in vLLM ≥ 0.15.0; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 32.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU.
Transformers (without vLLM)	transformers>=5.15.0.
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/GLM-4.7-Flash-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic", max_model_len=32768)
params = SamplingParams(temperature=1.0, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Recommended sampling (from the original generation_config.json): temperature=1.0. Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --kv-cache-dtype fp8

License

Quantized derivative of zai-org/GLM-4.7-Flash (mit).

---

한국어 요약

zai-org/GLM-4.7-Flash의 FP8 (W8A8) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	zai-org/GLM-4.7-Flash
양자화 방식	FP8 동적 양자화 (W8A8)
저장 형식	compressed-tensors float-quantized (scheme FP8_DYNAMIC)
모델 구조	MoE (GLM-4.7 Lite, Glm4MoeLite) · 31.2B · 컨텍스트 202K · 텍스트 전용
용량	32.2 GB
보정 데이터	없음 (보정 데이터 불필요)

양자화 상세

• 가중치: FP8(E4M3), 출력 채널별 스케일
• 활성화: 실행 시 토큰별 동적 FP8
• 방식: PTQ(학습 후 양자화, 재학습 없음) · 가중치 스케일은 미리 계산, 활성화 스케일은 실행 시 토큰마다 계산
• BF16 유지: 토큰 임베딩·LM 헤드·모든 RMSNorm·MoE 라우터(mlp.gate, e_score_correction_bias 포함)
• 양자화 대상: 어텐션·MLP projection (MTP 층 model.layers.47도 다른 층과 같이 양자화)
• 원본과 동일: 토크나이저·chat template·generation config

품질 (실측)

• 벤치마크 평균: 66.56 — MMLU·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE, lm-evaluation-harness(vLLM) (원본은 미측정)

요구사항

• 실행 엔진: vLLM 0.15.0 이상 — vLLM 0.32.0에서 실측
• vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드 — CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 (vLLM 고정 버전으로 이 체크포인트 로드 가능, 별도로 올리지 말 것)
• GPU: NVIDIA Turing(sm75, 예: T4) 이상 — FP8 연산은 Ada·Hopper·Blackwell(sm89 이상)에서 수행, 그보다 오래된 GPU는 vLLM이 가중치 전용 FP8로 자동 전환. V100(sm70) 미지원
• 메모리: 가중치 32.2 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph 수 GB — 한 장에 들어가지 않으면 --tensor-parallel-size N
• Transformers로 직접 사용: transformers 5.15.0 이상
• trust_remote_code: 불필요

사용법

• 서버 실행: vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768 — OpenAI 호환 API (기본 포트 8000)
• Python: LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic").chat(...) — 전체 예시는 위 Usage 참고
• 권장 샘플링: temperature=1.0
• thinking 모드: 기본 켜짐 — 끄려면 chat_template_kwargs={"enable_thinking": False}
• KV 캐시 절감(선택): --kv-cache-dtype fp8 — 긴 문맥·대량 요청에서 KV 메모리 약 절반, 가중치 양자화와 별개인 실행 옵션

라이선스

• 라이선스: MIT — 원본 모델의 라이선스를 그대로 따름
• 권리: 원본 가중치·토크나이저의 권리는 Z.ai에 있으며, 이 리포는 양자화 파생물