Instructions to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX
# Run inference directly in the terminal:
llama cli -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX
# Run inference directly in the terminal:
llama cli -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX
# Run inference directly in the terminal:
./llama-cli -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Use Docker

docker model run hf.co/Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

LM Studio
Jan

vLLM

How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Ollama

How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with Ollama:

ollama run hf.co/Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Unsloth Desktop

How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with Docker Model Runner:
```
docker model run hf.co/Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX
```

Lemonade

How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Run and chat with the model

lemonade run user.MiniCPM-V-4.7-35B-A3B-ROCmFPX-Q2_0_ROCMFPX

List all available models

lemonade list

Hermes Agent

How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX:Q2_0_ROCMFPX" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

MiniCPM-V-4.7-35B-A3B — ROCmFPX GGUF

openbmb/MiniCPM-V-4.7-35B-A3B quantized to the ROCmFPX formats (ROCmFP2/4/6/8, ROCmI4), a llama.cpp fork with AMD-oriented low-bit weight types for Vulkan and HIP/ROCm.

Start with Q4_0_ROCMFP4. It is the closest 4-bit file to the original. The other 4-bit files are recipes that ROCmFPX tuned and validated on AMD Strix Halo (Ryzen AI Max, gfx1151).

These files only load in the ROCmFPX fork. Stock llama.cpp, LM Studio and Ollama do not know these types.

Text only. MiniCPM-V 4.7 is a vision-language model, but the ROCmFPX fork does not yet have its vision projector, so these files are for text. For image input use upstream llama.cpp with standard quants.

Files and measured quality

Measured against the BF16 model (65GB, perplexity 6.1663) on WikiText-2 test, 40 chunks of 512 tokens. KLD is the mean KL divergence of the token distribution from BF16 (lower is better). Same top is how often the quant picks the same next token as BF16.

File	Size	PPL	KLD	Same top	Notes
`MiniCPM-V-4.7-35B-A3B-Q8_0_ROCMFPX_AGENT.gguf`	34GB	6.1528	0.0087	96.16%	8-bit with agent/tool-call routing, closest to BF16
`MiniCPM-V-4.7-35B-A3B-Q6_0_ROCMFPX_AGENT_LEAN.gguf`	27GB	6.1383	0.0128	95.34%	6-bit agent routing without Q8-heavy boosts
`MiniCPM-V-4.7-35B-A3B-Q4_0_ROCMFP4.gguf`	22GB	6.2530	0.0344	92.54%	Recommended. Balanced 4-bit ROCmFP4, best quality of the 4-bit files
`MiniCPM-V-4.7-35B-A3B-Q4_0_ROCMFP4_COHERENT.gguf`	19GB	6.3375	0.0597	89.81%	ROCmFP4 that protects tensors important for tool calls / JSON / code
`MiniCPM-V-4.7-35B-A3B-Q4_0_ROCMFP4_FAST.gguf`	18GB	6.3100	0.0691	89.33%	Speed-first 4-bit (4.25 bpw, one scale per block)
`MiniCPM-V-4.7-35B-A3B-Q4_0_ROCMFP4_STRIX.gguf`	18GB	6.3571	0.0689	89.03%	Strix Halo (gfx1151) quality/speed recipe
`MiniCPM-V-4.7-35B-A3B-Q4_0_ROCMI4.gguf`	18GB	6.1835	0.0654	89.78%	Signed 4-bit integers; Strix Halo experimental W4A4 path (opt-in build flag). HIP/ROCm only, no Vulkan shaders yet
`MiniCPM-V-4.7-35B-A3B-Q2_0_ROCMFPX.gguf`	11GB	8.3989	0.5395	72.42%	ROCmFP2 blocks (2.50 bpw), smallest file; clear quality loss

Removed or not published

These were measured but are not kept: a file above is the same size or smaller and closer to BF16, or they are near-copies of a file above.

Recipe	Size	PPL	KLD	Same top	Use instead
`Q8_0_ROCMFPX`	34GB	6.1937	0.0118	95.40%	`Q8_0_ROCMFPX_AGENT`
`Q6_0_ROCMFPX_AGENT`	31GB	6.1770	0.0131	95.25%	`Q6_0_ROCMFPX_AGENT_LEAN`
`Q6_0_ROCMFPX`	27GB	6.1709	0.0135	95.10%	`Q6_0_ROCMFPX_AGENT_LEAN`
`Q6_0_ROCMFPX_LEAN`	27GB	6.1572	0.0136	95.11%	`Q6_0_ROCMFPX_AGENT_LEAN`
`Q3_0_ROCMFPX_AGENT`	24GB	6.2214	0.0616	90.09%	`Q4_0_ROCMFP4`
`Q3_0_ROCMFPX`	18GB	6.2561	0.0982	87.69%	`Q4_0_ROCMFP4_FAST` or `Q4_0_ROCMI4`
`Q4_0_ROCMFP4_EVEN`	19GB	6.3424	0.0603	90.01%	`Q4_0_ROCMFP4_COHERENT`
`Q4_0_ROCMFP4_LEAN`	19GB	6.3467	0.0602	89.69%	`Q4_0_ROCMFP4_COHERENT`
`Q4_0_ROCMFP4_FAST_COHERENT`	18GB	6.3145	0.0694	89.36%	`Q4_0_ROCMFP4_FAST`
`Q4_0_ROCMFP4_STRIX_LEAN`	18GB	6.3554	0.0686	89.45%	`Q4_0_ROCMFP4_STRIX`

Q4_0_ROCMFP4_FAST_EVEN is byte-identical to Q4_0_ROCMFP4_FAST for this model, so it was never published.

How these were made

convert_hf_to_gguf.py --no-mtp --outtype bf16 from upstream llama.cpp. The config lists one MTP layer but the checkpoint ships no MTP weights, so the MTP head is left out.
Importance matrix from the BF16 model: 300 chunks × 512 tokens of WikiText-103 train text.
llama-quantize --imatrix from ROCmFPX (fb08d7c), one run per type.
Perplexity and KL divergence on the CPU backend (-dev none --no-op-offload). The fork's CUDA kernels for these types are not correct on NVIDIA, so no GPU numbers are used. These numbers describe the stored weights. AMD GPU kernels compute the same math in a different order and some paths also quantize activations (most of all the opt-in ROCmI4 W4A4 path), so results on an AMD GPU can be slightly worse. Reports of llama-perplexity results on AMD hardware are welcome.

Quantizing is CPU-only and deterministic: the same command gives byte-identical files on x86 and ARM.

Run

git clone https://github.com/charlie12345/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-server -j
build/bin/llama-server -m MiniCPM-V-4.7-35B-A3B-Q4_0_ROCMFP4.gguf -ngl 999 -fa on --jinja

License

Apache-2.0, same as the base model. All credit for the model goes to OpenBMB.

Downloads last month: 457

GGUF

Model size

35B params

Architecture

qwen35moe

Hardware compatibility

2-bit

4-bit

8-bit

View +1 variant

Model tree for Harsha001/MiniCPM-V-4.7-35B-A3B-ROCmFPX

Base model

openbmb/MiniCPM-V-4.7-35B-A3B

Quantized

(64)

this model