Instructions to use 88plug/MiniCPM-V-4.5-W8A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use 88plug/MiniCPM-V-4.5-W8A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="88plug/MiniCPM-V-4.5-W8A16", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("88plug/MiniCPM-V-4.5-W8A16", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use 88plug/MiniCPM-V-4.5-W8A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "88plug/MiniCPM-V-4.5-W8A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "88plug/MiniCPM-V-4.5-W8A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/88plug/MiniCPM-V-4.5-W8A16

SGLang

How to use 88plug/MiniCPM-V-4.5-W8A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "88plug/MiniCPM-V-4.5-W8A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "88plug/MiniCPM-V-4.5-W8A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "88plug/MiniCPM-V-4.5-W8A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "88plug/MiniCPM-V-4.5-W8A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use 88plug/MiniCPM-V-4.5-W8A16 with Docker Model Runner:
```
docker model run hf.co/88plug/MiniCPM-V-4.5-W8A16
```

Load path (important)

These weights are compressed-tensors (pack-quantized / int-quantized).

Runtime	Supported
vLLM ≥ 0.21	Yes — preferred (auto-detect CT; no `--quantization` flag)
transformers + `compressed-tensors`	Yes for many text models; multimodal may need `trust_remote_code`
Text Generation Inference (TGI)	Not supported for these CT packs
Hugging Face Inference Widget	Often fails — use vLLM locally instead

vllm serve 88plug/MiniCPM-V-4.5-W8A16 --trust-remote-code

Do not deploy via TGI — that backend does not load our CT format.

MiniCPM-V-4.5-W8A16

INT8 post-training quantization of openbmb/MiniCPM-V-4_5 — MiniCPM-V (vision + LLM), not MiniCPM-o (no audio / TTS). Qwen3-8B LLM + SigLIP2-400M vision + unified 3D-resampler (image, multi-image, video). Apache-2.0 base.

This is not the same product as 88plug/MiniCPM-o-4.5-W8A16.

At a Glance

Property	Value
Base model	`openbmb/MiniCPM-V-4_5`
Release tier	Pending-gold (gold path in progress)
Quant method	AutoRound W8A16 iters=200 (LLM Linear; vpm+resampler BF16)
FLAC status	Not measured (T+7d milestone)
Architecture	Qwen3-8B LLM + SigLIP2 vision + 3D-resampler
Quant format	compressed-tensors (native vLLM)
Quantized	LLM Linear layers (`model.llm`)
Kept BF16	vision encoder (vpm) + 3D-resampler
Lab pin	vLLM v0.21.0-cu129 (sidecar 0.28 tracked; pin switch after both hosts IMAGE_OK)

Quick Start

Tested target: vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are compressed-tensors — vLLM detects quantization automatically. No --quantization flag.

docker run --gpus device=0 -p 8080:8080 \
  vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \
  88plug/MiniCPM-V-4.5-W8A16 \
  --trust-remote-code \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Requires vLLM ≥ v0.21.0. Upstream MiniCPM-V 4.5 also documents vLLM since v0.10.2.

What's Quantized, What's Not

Component	Precision	Reason
LLM Linear layers	W8A16 INT8	AutoRound iters=200, `actorder=False`
Vision encoder (SigLIP2 / vpm)	BF16	Tower keep
3D-resampler	BF16	Tower keep
Embeddings, LM head, norms	BF16	Standard practice

No audio / Whisper / CosyVoice2 — those exist on MiniCPM-o, not MiniCPM-V.

Benchmarks

Metric	Status
Throughput (tok/s)	In progress — T+7d milestone
MMLU delta vs BF16	In progress — T+7d milestone
RULER@128k	In progress — T+30d milestone

No fabricated numbers. Results will be published to this card when measured.

Limitations

Vision-language only — not MiniCPM-o; no native audio input or CosyVoice2 TTS.
LLM Linear only — vpm + 3D-resampler stay BF16.
FLAC not measured — do not treat this card as full_flac complete.
Pending-gold — gold AutoRound path; smoke=pass required before Hub catalog counts as product_done.

Citation

@misc{yu2025minicpmv45,
  title  = {MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe},
  author = {Tianyu Yu and others},
  year   = {2025},
  eprint = {2509.18154},
  archivePrefix = {arXiv},
  url    = {https://huggingface.co/openbmb/MiniCPM-V-4_5}
}

About

88plug AI Lab ships FLAC-target compressed-tensors quantizations for native vLLM v0.21.0+ deployment.

This release: Pending-gold — gold-path quantization in progress. Do not use datafree/RTN substitutes.

Browse all releases → huggingface.co/88plug

Downloads last month: -

Safetensors

Model size

9B params

Tensor type

I32

BF16

Model tree for 88plug/MiniCPM-V-4.5-W8A16

Base model

openbmb/MiniCPM-V-4_5

Quantized

(9)

this model

Paper for 88plug/MiniCPM-V-4.5-W8A16

MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe

Paper • 2509.18154 • Published Sep 16, 2025 • 63