Instructions to use IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8")
model = AutoModelForMultimodalLM.from_pretrained("IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8

SGLang

How to use IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8 with Docker Model Runner:
```
docker model run hf.co/IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8
```

MiniCPM-V-4.7-35B-A3B-NVFP4-FP8

A mixed-precision quantization of openbmb/MiniCPM-V-4.7-35B-A3B for NVIDIA Blackwell GPUs (DGX Spark / GB10, RTX PRO 6000, B200, RTX 50xx). The routed MoE experts, which hold most of the weights, are NVFP4. The attention, linear-attention and shared-expert layers are FP8. Everything else stays BF16, including the vision tower.

21.2 GiB. In our tests it matches the BF16 model on agentic tool calling, screenshot understanding and long-context retrieval, and decodes at 61 tok/s on a single DGX Spark.

Component	Precision
Routed experts (`mlp.experts.*.{gate,up,down}_proj`)	NVFP4 W4A4 (group 16, FP8 scales), calibrated
Full attention (`self_attn.{q,k,v,o}_proj`)	FP8 (per-channel weights, dynamic per-token activations)
Linear attention (`linear_attn.{in_proj_qkv,in_proj_z,out_proj}`)	FP8 dynamic
Shared experts (`mlp.shared_expert.*`)	FP8 dynamic
Router, shared-expert gate, `linear_attn.in_proj_{a,b}`, embeddings, `lm_head`	BF16
Vision tower and merger	BF16 (bit-identical to the base model)

Results

All numbers below are our own measurements on one DGX Spark (GB10), using vLLM 0.32.1rc1.dev435 (the vllm/vllm-openai:nightly image). Every model was run with the same flags. The quantized models used --gpu-memory-utilization 0.33 (about 40 GB), fp8 KV cache and 128k context. They are compared with the BF16 base model and with sambit9238/MiniCPM-V-4.7-35B-A3B-NVFP4, whose two-step pipeline this checkpoint builds on.

	BF16	sambit9238 NVFP4	This repo
Size	65.7 GB	22.0 GiB	21.2 GiB
Decode tok/s, 1 user, 1k / 16k / 64k prompt	–	42 / 41 / 37	61 / 58 / 50
Top-1 next-token agreement with BF16 (mean of 9 texts)	100%	93.3%	94.5%
Top-1 agreement with BF16 on 6 multi-turn agent transcripts	100%	80.8–96.6%	84.0–97.2% (higher on all 6)
Long-context retrieval, 16k–100k tokens, single and 3-needle	24/24	24/24	24/24
Agentic browser-testing harness, tasks passed (10 tasks × runs ²)	30/30 (100%)	50/50 (100%) ¹	50/50 (100%)
Open-ended bug hunt, seeded bugs found (4 per run × runs ²)	12/12 (100%)	18/20 (90%) ¹	19/20 (95%)
Screenshot element grounding (box centre inside element)	24/32	22/32	24/32
Grounding mean IoU	0.49	0.42	0.52
Spatial-relation / on-screen reading questions	8/8, 8/8	8/8, 8/8	8/8, 8/8
Seeded visual defects found / clean pages passed	5/12, 5/6	5/12, 5/6	5/12, 5/6

¹ sambit9238's checkpoint ships its own chat template, which enables thinking by default. BF16 and this repo use openbmb's template, which runs in instruct mode. So this row is not strictly like-for-like. The drift, long-context and vision rows set the thinking mode explicitly and are directly comparable.

² BF16: 3 runs per task; quantized models: 5 runs per task. BF16 decodes slowly, so it got fewer runs. With this few runs, a difference of one bug is within noise.

What each row measures:

Next-token agreement runs teacher-forced prompt_logprobs over fixed texts: six agent transcripts with tool calls and accessibility snapshots, Python source, and English prose. It counts how often the quantized model's most likely next token is the same one BF16 picks.
Long-context retrieval plants facts in synthetic SOC log files. It is too easy to separate the models: all three were perfect up to 100k tokens.
The agentic harness is a simulated security-operations web app driven through Playwright-MCP tool schemas. Its tasks include stale element references, a modal that intercepts clicks, a transient 503 error, pagination and open-ended bug hunting.
The screenshot benchmark uses 18 rendered app screens, 12 of them with one seeded rendering defect (overlap, clipping, misalignment, wrong colour and so on). Ground-truth boxes come from the DOM.

These are small, task-specific evaluations, not a substitute for standard benchmarks. We did not run OCRBench or MMMU.

Usage (vLLM)

You need a vLLM build that includes the MiniCPMV4_7ForConditionalGeneration architecture (nightly 0.32.1rc1.dev435 or newer; v0.30.0 does not have it) and a GPU with native NVFP4 support.

vllm serve <this-repo> \
  --max-model-len 131072 --kv-cache-dtype fp8 \
  --enable-prefix-caching --enable-chunked-prefill \
  --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

On GB10, vLLM selected the FLASHINFER_CUTLASS NVFP4 MoE backend automatically. For DGX Spark we also set CUTE_DSL_ARCH=sm_121a.
qwen3_coder also works as the tool-call parser. Tool calls use the Qwen3-coder XML format.
The chat template is openbmb's original (instruct mode by default). Pass chat_template_kwargs={"enable_thinking": true} to turn thinking on. With --reasoning-parser qwen3 the model sometimes leaves its final answer inside the reasoning block. If content is empty, fall back to reasoning_content.
There is no MTP head, because the base checkpoint ships no MTP weights. openbmb's DSpark draft model is the speculative-decoding option.

How it was made

Quantization. The Qwen3.5-MoE text backbone was quantized with llm-compressor (git main), with all MoE experts linearized and calibrated (moe_calibrate_all_experts=True, sequential pipeline). Run time was 2 h 47 min on the DGX Spark.
Calibration data. 512 samples, 1.83M tokens in total (median 1.7k, max 8k):
- 55% ultrachat_200k, formatted with the model's chat template
- 25% multi-turn Playwright-MCP tool-calling trajectories, formatted with this model's chat template and tool schemas
- 10% long sequences: concatenated trajectories of up to 8k tokens
- 10% Python source code
The trajectories come from our own simulated-app test harness and are not included in this repo.
Assembly. The quantized backbone was combined with the BF16 vision tower and merger, copied byte-for-byte from the base model into model-vision-bf16.safetensors. Vision modules are added to the quantization ignore list.

The scripts are in scripts/: Dockerfile (llm-compressor on nvcr.io/nvidia/pytorch:26.08-py3), quantize.py (recipe and calibration set), check_recipe.py (checks which layers get which precision on a tiny model), and assemble.py. sambit_quantize_nvfp4.py and assemble.py are adapted from sambit9238's scripts (Apache-2.0).

Differences from sambit9238's NVFP4:

Attention, linear attention and shared experts are FP8 instead of NVFP4 or BF16. This gives lower drift and about 45% faster decode, because BF16 linear-attention projections dominate the bytes read per token.
The calibration set includes agentic tool-calling and longer sequences.

Credits

Base model: OpenBMB MiniCPM-V 4.7 (Apache-2.0)
Two-step VLM quantization pipeline: sambit9238
Quantization tooling: llm-compressor and compressed-tensors

For the model's capabilities, intended use and limitations, see the original model card.

Downloads last month: 42

Safetensors

Model size

35B params

Tensor type

BF16

F8_E4M3

Model tree for IstroSec/MiniCPM-V-4.7-35B-A3B-NVFP4-FP8

Base model

openbmb/MiniCPM-V-4.7-35B-A3B

Quantized

(74)

this model