AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637

MiniCPM-V-4.6-GPTQ on AXERA NPU

Ready-to-run deployment package for openbmb/MiniCPM-V-4.6-GPTQ on AX637 / NPU1.

• This release packages the AX637 axllm runtime together with the compiled text and vision .axmodel files.
• The packaged text runtime uses the GPTQ W4A16 build.
• The packaged vision runtime uses a fixed-shape 448x448 MiniCPM-V-4.6 vision encoder.
• The package supports text-only chat, single-image understanding, and video understanding through the OpenAI-compatible axllm serve API.
• The package is intended for board-side deployment through the bundled axllm runtime.

Supported Platform

• AX637 / NPU1

Validated Devices

This package has been validated on the following AX637-based device:

• AX637 / NPU1 development board

Performance

All measurements below were taken on AX637 / NPU1. TTFT stands for time to first token. In this table, TTFT is measured end-to-end from request arrival at axllm serve to the first generated token, so the multimodal rows include media preprocessing and vision encoding time.

The text-only smoke prompt, What is 1+1? Reply with the number only., was kept within one 128-token prefill chunk. To avoid one-time startup effects, the text row below excludes the first request after service startup. Its Decode figure was measured with longer text-only generations (max_tokens=256) to better reflect sustained decode throughput; very short smoke replies under-report decode speed because EOS and response-tail overhead become relatively larger. The image row was measured with the packaged fixed-shape 448x448 vision encoder, assets/sample.png, and What animal is the main subject in this image? Reply in Chinese only.. The video row used the packaged sample video with video:assets/red-panda-openai.mp4:2 and Describe this video briefly..

Scenario	Input tokens	Prefill chunks	TTFT	Decode
Text-only smoke prompt	25	1 x 128	611.11 ms avg (609.72-612.52 ms)	10.06 token/s avg
Image prompt	97	1 x 128	1561.41 ms avg (1473.59-1736.26 ms)	10.26 token/s avg
Video prompt	480	4 x 128	11333.09 ms avg (9834.19-14312.73 ms)	9.05 token/s avg

The packaged runtime uses the following context layout:

• prefill_len=128
• kv_cache_len=511
• prefill_max_token_num=512

Input tokens in the table above refers to the full request length after chat templating, not just the visual soft tokens. For the shipped 448x448 vision encoder, each selected image block contributes 64 visual soft tokens. Under the current packaged runtime settings, the sample video request in this README uses 480 total input tokens and spans 4 prefill chunks.

Startup Runtime Footprint

Item	Value
Flash total (text + post + vision axmodels)	1.29 GiB (1316.25 MiB)
Package flash total (excluding vision_cache/)	1.81 GiB (1856.38 MiB)
Runtime CMM increment during board-side startup	1.36 GiB (1394.16 MiB)

The runtime CMM value above was measured during board-side startup on the validated AX637 board configuration and should be treated as a practical reference value.

Vision Encoder Latency

Measured on AX637 / NPU1 with /opt/bin/ax_run_model -m minicpmv4_6_vision_448.axmodel -g 0 -w 1 -r 5.

Model	Resolution	Soft Tokens	Time (ms)
minicpmv4_6_vision_448.axmodel	448x448	64	543.970 ms avg

For this packaged AX637 runtime, the visual token count is fixed by the shipped vision encoder configuration:

• vision_width = 448
• vision_height = 448
• vision_patch_size = 14
• patch grid = (448 / 14) x (448 / 14) = 32 x 32
• raw patch tokens = 32 x 32 = 1024
• current packaged build uses the 16x visual compression path
• Soft Tokens = 1024 / 16 = 64

So, for the fixed-shape runtime shipped in this repository, the relation is:

Soft Tokens = (vision_width / patch_size) x (vision_height / patch_size) / 16

Input tokens in the performance table can be larger than the visual Soft Tokens because axllm counts the full templated request, including user text and chat-template tokens in addition to the visual tokens. For the packaged assets/sample.png request in this README, the runtime reports input_num_token=97, which still fits within a single 128-token prefill chunk.

Soft Tokens is not a runtime-configurable value in this package. This repository ships only minicpmv4_6_vision_448.axmodel, so the board-side AX637 runtime always uses 448x448 -> 64 soft tokens for image encoding.

Package Layout

.
├── README.md
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── lib/
│   └── runtime shared libraries
├── assets/
│   ├── openai_api_demo.png
│   ├── red-panda-openai.mp4
│   └── sample.png
├── minicpmv4_6_vision_448.axmodel
├── qwen3_5_text_p128_l0_together.axmodel
├── ...
├── qwen3_5_text_p128_l23_together.axmodel
├── qwen3_5_text_post.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── config.json
├── post_config.json
└── minicpm_v46_tokenizer.txt

The packaged axllm runtime, runtime shared libraries, and compiled .axmodel files live at the repository root.

Sample Image

The axllm flow uses the sample image: assets/sample.png

Sample Video

The package also includes a packaged sample video for board-side video understanding validation:

• assets/red-panda-openai.mp4

Direct Inference with axllm

The axllm workflow is still being refined. The instructions below reflect the current validated flow and may be adjusted as the packaging continues to evolve.

Download the Model Package

Download the release package from Hugging Face:

mkdir -p AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637
cd AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637
hf download AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637 --local-dir .

Install axllm

Option 1: use the validated binary included in this repository:

chmod +x ./bin/axllm

Option 2: install axllm from the public repository:

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

Option 3: install with a one-line command:

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

Option 4: download the prebuilt binary from GitHub Actions CI:

If you do not have a local build environment, download the latest CI-generated axllm binary from GitHub Actions: https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm Then run:

chmod +x axllm
sudo mv axllm /usr/bin/axllm

Run on the Board

The package root is already arranged for axllm, so no extra runtime path arguments are required.

For multimodal testing, you can use the packaged sample image shown above: ./assets/sample.png, or the packaged sample video: ./assets/red-panda-openai.mp4.

./bin/axllm run .

In interactive mode:

• press Enter directly for text-only chat
• input an image path for single-image chat
• input video:/path/to/frames_dir or video:/path/to/video.mp4 for video chat

Serve with axllm

From the package root on the board:

./bin/axllm serve . --port 8000

Expected model id:

AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511

Health check:

curl http://127.0.0.1:8000/health

A typical startup log looks like this:

INF Init | LLM init start
INF Init | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
INF Init | attention config: layers=24 sliding=0 full=6 linear=18 sliding_window=0 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
...
INF Init | max_token_len : 511
INF Init | kv_cache_size : 512, kv_cache_num: 511
INF init_groups_from_model | prefill_token_num : 128
INF init_groups_from_model | prefill_max_token_num : 512
INF Init | MiniCPM-V-4.6 token ids: image_pad=248056 video_pad=248057
INF Init | VisionModule init ok: type=MiniCPMV46VL, tokens_per_block=64, embed_size=1024, out_dtype=fp32
INF Init | LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511

You can then send requests to the server using the API endpoints shown in the log. For example, to check the health status and list the available models:

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models

Example output:

{
  "concurrency": 0,
  "max_concurrency": 1,
  "status": "healthy"
}
{
  "data": [
    {
      "created": 1780908633,
      "id": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511",
      "object": "model",
      "owned_by": "openai-api"
    }
  ],
  "object": "list"
}

Text Request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "中国和美国的首都分别是哪里"}
        ]
      }
    ],
    "max_tokens": 32
  }'

Example output:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "中国的首都是北京，美国的首都是华盛顿。"
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511",
  "object": "chat.completion"
}

Image Request

python3 - <<'PY'
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

img = Path("assets/sample.png").read_bytes()
payload = {
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What animal is the main subject in this image? Reply in Chinese only."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + base64.b64encode(img).decode()
                    },
                },
            ],
        }
    ],
    "max_tokens": 32,
}
req = Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(req, timeout=60) as resp:
    print(resp.read().decode())
PY

Example output:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "龙虾"
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511",
  "object": "chat.completion"
}

Video Request

axllm serve accepts either a frames directory or a raw video file:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "video:/path/to/frames_dir"}},
          {"type": "text", "text": "Describe this video briefly."}
        ]
      }
    ],
    "max_tokens": 128
  }'

For a raw video file, use video:/path/to/video.mp4. If you need to request a specific sampling FPS, use the form video:/path/to/video.mp4:2.

To test the packaged sample video from the package root, you can set:

VIDEO_PATH="$(pwd)/assets/red-panda-openai.mp4"

and then use video:${VIDEO_PATH}:2 in the request payload.

Example output:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The video shows a red panda and another animal."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX637-C128-P512-CTX511",
  "object": "chat.completion"
}

Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

• Original Hugging Face model: openbmb/MiniCPM-V-4.6-GPTQ
• AXERA conversion and deployment workflow: AXERA-TECH/MiniCPM-V-4.6.axera

Discussion

• GitHub Issues
• QQ group: 139953715