HanzoHuang/Qwen3-VL-4B-Instruct-RKLLM

Qwen3-VL-4B-Instruct · RKLLM

Text and image chat on Rockchip RK3576 and RK3588, converted from Qwen/Qwen3-VL-4B-Instruct.

• Max context length: 8192

• Conversion toolkit: RKLLM Toolkit 1.3.1 and RKNN Toolkit2 2.3.2.
• Inputs / output: text and one image → text. Video and audio input are unavailable in this runtime.
• License: Apache 2.0; see the original model and license.

Choose your files

Use files for your board's exact SoC. RK3576 offers compact W4A16-g128 and W8A8; RK3588 uses W8A8. Download size is not runtime RAM.

Board	Quantization	Size	Language model
RK3576	W4A16-g128	3.16 GiB	Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm
RK3576	w8a8	4.62 GiB	Qwen3-VL-4B-Instruct_RK3576_w8a8.rkllm
RK3588	w8a8	4.52 GiB	Qwen3-VL-4B-Instruct_RK3588_w8a8.rkllm

Every language model requires its board's matching vision encoder below. Both RK3576 language variants share the RK3576 encoder.

Board	Size	Vision encoder
RK3576	0.83 GiB	Qwen3-VL-4B-Instruct_vision_RK3576.rknn
RK3588	0.81 GiB	Qwen3-VL-4B-Instruct_vision_RK3588.rknn

Run on your board

Download the RK3576 W4A16-g128 language model and matching encoder:

hf download HanzoHuang/Qwen3-VL-4B-Instruct-RKLLM \
  v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm \
  v1.3.1/RK3576/Qwen3-VL-4B-Instruct_vision_RK3576.rknn \
  --local-dir ./models

sudo docker run --rm -it --pull=always --privileged \
  -p 8001:8001 -v /dev:/dev \
  -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
  -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e RKLLM_EMBED_FLASH=1 \
  -e MODEL_FILE=Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm \
  -e VISION_MODEL_FILE=Qwen3-VL-4B-Instruct_vision_RK3576.rknn \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest \
  serve --max_context_len 4096

For RK3588, download both RK3588 files, mount that directory, set TARGET_PLATFORM=rk3588, and select its W8A8 language filename. Use the runtime documentation for health checks and OpenAI-compatible image requests.

Release notes

• Source acquired from ModelScope Qwen/Qwen3-VL-4B-Instruct, pinned revision 0196dc7bb23f3c742616147c3254d0e4f1207787. The HF link above identifies the canonical upstream model; no equivalence to an HF Git commit is assumed.
• Export context: 8192 tokens. Functional API tests used 4096; the speed/memory benchmark used 2048. Longer runtime context has not been validated on these boards.
• Fixed vision input: RGB, 448 × 448, batch 1, 196 image tokens, mean/std 0.5. The runtime pads to a square using RGB 127, then resizes bilinearly. Vision output order is pooler followed by three deepstack features for each token.
• Language calibration: the SDK's 20 image-and-text examples, captured using the source processor and language-model input hook. W4A16-g128 uses GRQ; W8A8 uses normal quantization. Float32 CUDA loading; two NPU cores on RK3576 and three on RK3588.
• Export compatibility fixes select the four language-width vision outputs under Transformers 5 and import only the requested calibration model family. Generated upstream examples and source checkpoint weights are unchanged.
• All three variants passed seven API smoke checks on their target board: arithmetic, independent requests, SSE, exact conversation recall, image description, image color over SSE, and text after an image. Tests used the baked RKLLM 1.3.1 runtime application with embed_flash=1.
• RK3576 vision encoder cosine against original float32 embeddings: 0.99534508.
• RK3588 vision encoder cosine against original float32 embeddings: 0.99543323.

These are functional and numerical smoke checks, not a broad quality benchmark. Quantization and fixed image resolution may reduce accuracy.

Measured on the target boards:

Board	Quantization	Decode tok/s	Startup RAM (GB)	Stable RAM (GB)	Date
RK3576	w4a16-g128	5.51	4.01	4.01	2026-10-23
RK3576	w8a8	4.11	5.83	5.83	2026-10-23
RK3588	w8a8	6.58	5.58	5.62	2026-10-22

Speed is the median native decode rate over three 128-token runs after a 32-token warmup, greedy decoding, EOS ignored, context 2048. RAM is additional board usage above a five-second idle baseline: sampled loading peak and ten-second ready-state median, in decimal GB. The matched encoder is loaded; embed_flash=1. First-token timing stays in the private report. Measurements depend on the board, clocks, cooling, and background load.

RK3576 measurements temporarily disabled zram swap and restored it after each run; both had zero swap growth. RK3588 retained its normal swap configuration.

SHA-256 checksums

0a014ecf5f8513878e4c9fefd6c4ceec8f105005d2920588dbd5ded188a6c421  v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm
b4a132c43fa10a5ce86b81f100a1cfa6f0aa1ffbc477b20e32a38d4f64b50a40  v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w8a8.rkllm
c8074176816c68ab635c70a7a0db3044a15f0f7b272fc45c0e6cc68875db159e  v1.3.1/RK3576/Qwen3-VL-4B-Instruct_vision_RK3576.rknn
cbf7b3bb2eec550f268fbc64fadcf53791283acdfba48fe0ff621aa46aba884b  v1.3.1/RK3588/Qwen3-VL-4B-Instruct_RK3588_w8a8.rkllm
478e5922e5a28d130146bad9b22dfc66b4eeb53880477744bc3130eab1fb321d  v1.3.1/RK3588/Qwen3-VL-4B-Instruct_vision_RK3588.rknn

Credits

Qwen provides the original model. Conversion uses Rockchip RKLLM, RKNN Toolkit2, and the conversion wrapper.