axteam/MiniCPM-o-4_5-FP8-dynamic

MiniCPM-o-4_5 · FP8 (dynamic)

8-bit quantization of openbmb/MiniCPM-o-4_5 for vLLM serving. Quantized by axteam.

Base model	openbmb/MiniCPM-o-4_5
Method	FP8 (dynamic)
Format	compressed-tensors FP8_DYNAMIC
Weights / activations	W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations
Architecture	Omni (vision + audio + TTS) · multimodal
Size on disk	~10.4 GB
Calibration	None (data-free)
License	Apache-2.0

Quantization details

• Scheme — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors FP8_DYNAMIC.
• Kept in BF16 — the vision, audio (whisper) and TTS modules, token embeddings, the LM head, and all RMSNorms (a small fraction of parameters). Only the language-model backbone's attention & MLP projections are quantized.
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Evaluation

Checked against the BF16 original with perplexity and top-1 agreement (EN/KO).

Usage (vLLM)

vllm serve axteam/MiniCPM-o-4_5-FP8-dynamic

License

Quantized derivative of openbmb/MiniCPM-o-4_5 (Apache-2.0). Original weights & tokenizer belong to OpenBMB.

---

한국어 요약

openbmb/MiniCPM-o-4_5 를 FP8 (dynamic) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

• 방식 — FP8 E4M3, 가중치 채널별 스케일(정적·보정불필요), 활성화 동적 FP8. 형식 compressed-tensors FP8_DYNAMIC.
• BF16 유지 — 비전·오디오(whisper)·TTS 모듈·토큰 임베딩·LM 헤드·모든 norm. 언어모델 백본의 attention·MLP projection 만 양자화.
• PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
• 품질 — BF16 대비 perplexity·top-1 로 검증(영/한).
• 사용 — vllm serve axteam/MiniCPM-o-4_5-FP8-dynamic. 라이선스 Apache-2.0(원본 OpenBMB).