MiniCPM5-2B INT4 GQ OpenVINO

This repository contains an INT4 group-wise quantized OpenVINO conversion of openbmb/MiniCPM5-2B.

The model is intended for efficient local inference on OpenVINO-supported hardware, including Intel CPUs, GPUs, and NPUs (tested on U7-155H) with reduced memory footprint compared with the original floating-point model.

This is a community conversion. The model architecture, training, tokenizer, chat template, capabilities, and license originate from the original MiniCPM5-2B model by OpenBMB.

Model Details

Property	Value
Base model	`openbmb/MiniCPM5-2B`
Framework	OpenVINO
Task	Text generation with KV cache
Weight precision	INT4
Quantization scheme	Symmetric group-wise weight quantization
Group size	128
INT4 ratio	1.0
Export task	`text-generation-with-past`
Primary languages	Chinese / English
License	Apache-2.0

The model uses weight-only INT4 compression. Activations are not explicitly quantized by the export command used for this repository.

A group size of 128 provides a practical trade-off between compression efficiency, inference performance, and model quality for small language models.

Quantization Configuration

The main quantization parameters are:

weight format : INT4
symmetric     : true
ratio         : 1.0
group size    : 128

ratio=1.0 requests 4-bit compression for all eligible weight layers. Operations that cannot use the requested compression configuration may be handled according to the exporter's fallback behavior.

This repository uses data-free weight compression and does not apply AWQ, GPTQ, scale estimation, or calibration-dataset-based optimization.

Intended Use

This model is suitable for:

local conversational assistants;
Chinese and English text generation;
lightweight reasoning;
tool-calling and agent experiments;
code-related assistance;
local OpenVINO applications;
OpenVINO Model Server deployments;
edge and AI-PC inference experiments.

For full information about the model's original training, capabilities, evaluation results, and limitations, see the upstream model:

openbmb/MiniCPM5-2B

OpenVINO Inference

OpenVINO GenAI

import openvino_genai as ov_genai

model_path = "Jeff98/MiniCPM5-2B-int4-gq-ov"

pipe = ov_genai.LLMPipeline(
    model_path,
    "GPU",
)

result = pipe.generate(
    "Explain what OpenVINO is in simple terms.",
    max_new_tokens=256,
)

print(result)

If the repository has first been downloaded locally:

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline(
    r"/path/to/MiniCPM5-2B-int4-gq-ov",
    "GPU",
)

print(
    pipe.generate(
        "你好，请简单介绍一下你自己。",
        max_new_tokens=256,
    )
)

Depending on your OpenVINO installation and hardware, the target device may be changed to:

CPU
GPU
NPU

Device support and generation quality should be validated on the target hardware and OpenVINO version.

OpenVINO Model Server

The model can also be served through OpenVINO Model Server (OVMS).

Example:

ovms \
  --model_name MiniCPM5-2B \
  --model_path /path/to/MiniCPM5-2B-int4-gq-ov \
  --rest_port 8000 \
  --target_device GPU

On Windows PowerShell:

ovms `
  --model_name MiniCPM5-2B `
  --model_path "C:\models\MiniCPM5-2B-int4-gq-ov" `
  --rest_port 8000 `
  --target_device GPU

Once started, OVMS exposes an OpenAI-compatible API.

Example:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v3",
    api_key="unused",
)

response = client.chat.completions.create(
    model="MiniCPM5-2B",
    messages=[
        {
            "role": "user",
            "content": "Hello! Briefly introduce yourself."
        }
    ],
    max_tokens=256,
)

print(response.choices[0].message.content)

Tool Calling

MiniCPM5 supports tool-calling capabilities inherited from the original model.

When serving through a backend that applies the original MiniCPM5 chat template, ensure that the backend supports the Jinja features used by the template and provides the corresponding MiniCPM5 tool parser.

Tool-calling behavior may depend on:

serving backend;
chat-template implementation;
tool parser;
model generation configuration;
number and complexity of tool schemas.

Hardware Notes

CPU

The model can be used with OpenVINO CPU inference:

--target_device CPU

INT4 weight compression substantially reduces the model's memory footprint compared with floating-point weights.

Intel GPU

Intel GPU is a recommended deployment target for this conversion:

--target_device GPU

Actual latency and throughput depend on the GPU architecture, driver version, OpenVINO version, prompt length, and generation settings.

Intel NPU

OpenVINO support and model compatibility on Intel NPUs evolve over time.

Although an OpenVINO IR model may successfully compile on an NPU, this does not by itself guarantee identical numerical behavior to CPU or GPU inference. In particular, long-context or long-generation workloads should be validated on the specific NPU generation, driver, and OpenVINO version being used.

For production or long-reasoning workloads, compare the output against CPU/GPU inference before deployment.

Quantization Considerations

INT4 compression reduces model size and memory bandwidth requirements but may introduce a small accuracy loss relative to the original floating-point checkpoint.

This repository uses:

symmetric INT4
group size = 128
ratio = 1.0

Smaller group sizes or data-aware quantization methods may improve accuracy at the cost of conversion time, model size, or inference performance.

For applications that are particularly sensitive to reasoning or generation quality, consider comparing this model against:

the original floating-point model;
an INT8 OpenVINO conversion;
an INT4 model using data-aware methods such as AWQ or scale estimation.

Limitations

This conversion inherits all limitations of the original MiniCPM5-2B model.

Possible limitations include:

hallucinated or factually incorrect responses;
incorrect generated code;
unreliable tool selection or arguments;
degradation on very long conversations;
quality loss caused by 4-bit weight compression;
hardware-dependent differences between CPU, GPU, and NPU inference;
differences between serving backends and chat-template implementations.

The model should not be assumed to produce identical outputs across Transformers, OpenVINO CPU, OpenVINO GPU, and OpenVINO NPU backends.

Repository Purpose

This repository only provides an optimized OpenVINO representation of the original MiniCPM5-2B model.

No additional fine-tuning, preference optimization, or alignment training has been performed.

openbmb/MiniCPM5-2B
        │
        │ Optimum Intel / OpenVINO export
        │
        ├── INT4 weight compression
        ├── symmetric quantization
        ├── group size 128
        └── text-generation-with-past
        │
        ▼
Jeff98/MiniCPM5-2B-int4-gq-ov

Acknowledgements

The original MiniCPM5 model was developed by the OpenBMB team.

Original model: https://huggingface.co/openbmb/MiniCPM5-2B
OpenBMB: https://github.com/OpenBMB
OpenVINO: https://github.com/openvinotoolkit/openvino
Optimum Intel: https://github.com/huggingface/optimum-intel

Please refer to the original model repository for model architecture, training methodology, benchmark results, ethical considerations, and upstream licensing information.

License

This repository follows the Apache License 2.0 used by the original openbmb/MiniCPM5-2B checkpoint.

Users should review the upstream model card and license before redistribution or deployment.

Downloads last month: 34

Model tree for Jeff98/MiniCPM5-2B-int4-gq-ov

Base model

openbmb/MiniCPM5-2B

Finetuned

(40)

this model