axteam/MiniCPM5-2B-NVFP4A16

MiniCPM5-2B · NVFP4A16 (W4A16, FP4 weights)

4-bit quantization of openbmb/MiniCPM5-2B for vLLM serving. Quantized by axteam.

Base model	openbmb/MiniCPM5-2B
Method	NVFP4A16 (W4A16, FP4 weights)
Format	compressed-tensors nvfp4-pack-quantized
Weights / activations	W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations
Architecture	Dense (Llama-architecture) · text-only
Size on disk	~2.2 GB
Calibration	256 samples
License	Apache-2.0

Quantization details

• Scheme — W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations, stored as compressed-tensors nvfp4-pack-quantized.
• Kept in BF16 — token embeddings, the LM head, and all RMSNorms (about 21% of all parameters in this build).
• AWQ smoothing with the calibration set is applied before FP4 rounding (checked 2026-11-01: the smoothed RMSNorms differ from the original as expected).
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic	AWQ-8bit	NVFP4A16 (this repo)	NVFP4	AWQ-4bit
MMLU (5-shot, generative)	59.74	59.56 (99.7%)	59.86 (100.2%)	56.81 (95.1%)	53.50 (89.5%)	56.31 (94.3%)
GSM8K (5-shot, strict)	73.01	71.27 (97.6%)	73.01 (100.0%)	65.20 (89.3%)	55.04 (75.4%)	65.96 (90.3%)
ARC-Challenge (0-shot)	43.94	44.20 (100.6%)	43.86 (99.8%)	43.43 (98.8%)	40.10 (91.3%)	44.88 (102.1%)
Winogrande (5-shot)	64.40	63.22 (98.2%)	64.40 (100.0%)	62.75 (97.4%)	59.04 (91.7%)	62.12 (96.4%)
KMMLU (5-shot, direct)	37.65	38.38 (101.9%)	38.27 (101.7%)	35.56 (94.4%)	34.06 (90.5%)	30.73 (81.6%)
HAE-RAE (0-shot)	39.05	38.13 (97.7%)	39.14 (100.2%)	35.38 (90.6%)	30.16 (77.2%)	36.94 (94.6%)
Average	52.97	52.46 (99.0%)	53.09 (100.2%)	49.85 (94.1%)	45.32 (85.6%)	49.49 (93.4%)

⚠️ Note: average recovery is 94.1% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.

Evaluation (measured)

Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts, BF16 greedy answers scored under each build).

Metric	This model	BF16 original
Perplexity (lower = closer)	1.8315	1.5988 (+14.6%)
Top-1 next-token agreement	85.0%	98.9%

⚠️ Note: noticeable 4-bit degradation here; prefer the 8-bit (FP8 / AWQ INT8) builds.

Requirements

Runtime	vLLM — the architecture is in vLLM; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75) or newer — weight-only FP4, no FP4 tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 2.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU.
Transformers (without vLLM)	transformers>=4.56.0 + compressed-tensors>=0.15.0 (checked 4.56.2 / 4.57.6 with FP8, AWQ-4bit and NVFP4A16, and 4.56.0 with AWQ-4bit; on transformers 4.x the tokenizer also returns token_type_ids, which the model does not accept, so drop it from the tokenizer output before calling generate, as these checks did).
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/MiniCPM5-2B-NVFP4A16 --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/MiniCPM5-2B-NVFP4A16",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "top_p": 0.95, "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/MiniCPM5-2B-NVFP4A16", max_model_len=32768)
params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Recommended sampling (from the original generation_config.json): temperature=1.0, top_p=0.95. Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/MiniCPM5-2B-NVFP4A16 --kv-cache-dtype fp8

License

Quantized derivative of openbmb/MiniCPM5-2B (Apache-2.0). Original weights & tokenizer belong to OpenBMB.

---

한국어 요약

openbmb/MiniCPM5-2B의 NVFP4A16 (W4A16) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	openbmb/MiniCPM5-2B
양자화 방식	NVFP4A16 (W4A16, FP4 가중치)
저장 형식	compressed-tensors nvfp4-pack-quantized
모델 구조	Dense (Llama 구조) · 텍스트 전용
용량	약 2.2 GB
보정 데이터	샘플 256개

양자화 상세

• 가중치: FP4(E2M1), 그룹 크기 16 + 그룹별 FP8 스케일
• 활성화: BF16 (양자화하지 않음)
• 양자화 절차: PTQ(학습 후 양자화, 재학습 없음), AWQ 방식 — 보정 데이터로 활성화가 큰 가중치 채널을 찾아 채널별 스케일로 보호한 뒤 FP4로 반올림
• BF16 유지: 토큰 임베딩, LM 헤드, 모든 RMSNorm — 전체 파라미터의 약 21%
• 원본 파일: 토크나이저·chat template은 원본과 동일

품질 (실측)

• 원본 대비 충실도: PPL 1.8315 (원본 1.5988, +14.6%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 85.0% (원본 자체 기준값 98.9%) — 내부 영어·한국어 혼합 프롬프트 100개, 원본(BF16)의 greedy 답변을 기준 텍스트로 측정
• 벤치마크 평균: 49.85 (원본 52.97, 94.1%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
• 한국어 벤치마크: KMMLU 35.56 (원본 37.65, 94.4%) · HAE-RAE 35.38 (원본 39.05, 90.6%)
• MMLU 방식: lm-eval 생성형 변형(mmlu_generative) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가
• 주의: ⚠️ 이 정밀도에서 품질 저하가 뚜렷함 (PPL +14.6%, 벤치마크 평균 원본 대비 94.1%) — 품질이 중요하면 8비트(FP8·AWQ-8bit) 빌드 권장

요구사항

• 실행 엔진: vLLM, 이 빌드는 vLLM 0.32.0에서 실행 확인
• vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
• GPU: NVIDIA Turing(sm75) 이상, V100(sm70) 미지원 — 가중치 전용 FP4라 FP4 전용 텐서코어 불필요 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
• 메모리: 가중치 2.2 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
• Transformers로 직접 사용: transformers 4.56.0 이상 + compressed-tensors 0.15.0 이상 (4.56.2·4.57.6에서 FP8·AWQ-4bit·NVFP4A16, 4.56.0에서 AWQ-4bit 확인, transformers 4.x에서는 토크나이저가 token_type_ids도 돌려주는데 모델이 받지 않으므로 generate 호출 전에 토크나이저 출력에서 빼야 함 — 확인 때도 뺌)
• trust_remote_code: 불필요

사용법

• 서버 실행: vllm serve axteam/MiniCPM5-2B-NVFP4A16 --max-model-len 32768 — OpenAI 호환 API
• Python: LLM(model="axteam/MiniCPM5-2B-NVFP4A16", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
• 권장 샘플링: temperature=1.0, top_p=0.95 (원본 generation_config.json 기준)
• thinking 모드: 기본 켜짐 — 끄려면 chat_template_kwargs={"enable_thinking": False} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": false})
• KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

라이선스

• 라이선스: Apache-2.0 — 원본 모델의 라이선스를 그대로 따름
• 권리: 원본 가중치·토크나이저는 OpenBMB 소유, 이 리포는 양자화 파생물