HanzoHuang/Qwen3-VL-2B-Instruct-RKLLM

Qwen3-VL-2B-Instruct · RKLLM

• Max context length: 8192

Text and image chat on Rockchip RK3576 and RK3588, converted from Qwen/Qwen3-VL-2B-Instruct.

• Conversion toolkit: RKLLM Toolkit 1.3.1 and RKNN Toolkit2 2.5.0.
• Inputs / output: text and one image → text. Video and audio input are unavailable in this runtime.
• License: Apache 2.0; see the original model and license.

Choose your files

Use files for your board's exact SoC. RK3576 offers compact W4A16-g128 and W8A8; RK3588 uses W8A8. Download size is not runtime RAM.

Board	Quantization	Size	Language model
RK3576	W4A16-g128	1.74 GiB	Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm
RK3576	w8a8	2.28 GiB	Qwen3-VL-2B-Instruct_RK3576_w8a8.rkllm
RK3588	w8a8	2.22 GiB	Qwen3-VL-2B-Instruct_RK3588_w8a8.rkllm

Every language model requires its board's matching vision encoder below. Both RK3576 language variants share the RK3576 encoder.

Board	Size	Vision encoder
RK3576	0.81 GiB	Qwen3-VL-2B-Instruct_vision_RK3576.rknn
RK3588	0.80 GiB	Qwen3-VL-2B-Instruct_vision_RK3588.rknn

Run on your board

Download the RK3576 W4A16-g128 language model and matching encoder:

hf download HanzoHuang/Qwen3-VL-2B-Instruct-RKLLM \
  v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm \
  v1.3.1/RK3576/Qwen3-VL-2B-Instruct_vision_RK3576.rknn \
  --local-dir ./models

sudo docker run --rm -it --pull=always --privileged \
  -p 8001:8001 -v /dev:/dev \
  -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
  -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e RKLLM_EMBED_FLASH=1 \
  -e MODEL_FILE=Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm \
  -e VISION_MODEL_FILE=Qwen3-VL-2B-Instruct_vision_RK3576.rknn \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest \
  serve --max_context_len 4096

For RK3588, download both RK3588 files, mount that directory, set TARGET_PLATFORM=rk3588, and select its W8A8 language filename. Use the runtime documentation for health checks and OpenAI-compatible image requests.

Release notes

• Source acquired from ModelScope Qwen/Qwen3-VL-2B-Instruct, pinned revision ae9985b208c074c10cfbe3a61b5cb7268cdc9c53. The HF link above identifies the canonical upstream model; no equivalence to an HF Git commit is assumed.
• Export context: 8192 tokens. Functional API tests used 4096; the speed/memory benchmark used 2048. Longer runtime context has not been validated on these boards.
• Fixed vision input: RGB, 448 × 448, batch 1, 196 image tokens, mean/std 0.5. The runtime pads to a square using RGB 127, then resizes bilinearly. Vision output order is pooler followed by three deepstack features for each token.
• Language calibration: the SDK's 20 image-and-text examples, captured using the source processor and language-model input hook. W4A16-g128 uses GRQ; W8A8 uses normal quantization. Float32 CUDA loading; two NPU cores on RK3576 and three on RK3588.
• Export compatibility fixes select the four language-width vision outputs under Transformers 5 and import only the requested calibration model family. Generated upstream examples and source checkpoint weights are unchanged.
• All three variants passed seven API smoke checks on their target board: arithmetic, independent requests, SSE, exact conversation recall, image description, image color over SSE, and text after an image. Tests used the baked RKLLM 1.3.1 runtime application with embed_flash=1.
• RK3576 vision encoder cosine against original float32 embeddings: 0.99767705.
• RK3588 vision encoder cosine against original float32 embeddings: 0.99805251.

These are functional and numerical smoke checks, not a broad quality benchmark. Quantization and fixed image resolution may reduce accuracy.

Measured on 2026-10-22:

Board	Quantization	Decode tok/s	Startup RAM (GB)	Stable RAM (GB)
RK3576	w4a16-g128	11.11	2.60	2.57
RK3576	w8a8	9.02	3.18	3.18
RK3588	w8a8	14.05	3.03	3.07

Speed is the median native decode rate over three 128-token runs after a 32-token warmup, greedy decoding, EOS ignored, context 2048. RAM is additional board usage above a five-second idle baseline: sampled loading peak and ten-second ready-state median, in decimal GB. The matched encoder is loaded; embed_flash=1. First-token timing stays in the private report. Measurements depend on the board, clocks, cooling, and background load.

RK3576 W8A8 was measured with zram swap temporarily disabled and restored afterward. Earlier measurements with background swap growth were excluded.

SHA-256 checksums

f687dccc90da6ef74d5308af3a05dba5cf3fea68c359b151381b7dbdc8a06c8f  v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm
77ae22a5d1dd8c15833c57e390de4afc3aa4f1867c5d24b0a16e32e8d1e20adf  v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w8a8.rkllm
5ec488c125b6604cd623913ce5126db3f5075f5e1412f8d8f912ca14c2513dbc  v1.3.1/RK3576/Qwen3-VL-2B-Instruct_vision_RK3576.rknn
e8fff0de10e2f48ba13af759faca3a44cc07f661e95aeca412850a7eba08a92e  v1.3.1/RK3588/Qwen3-VL-2B-Instruct_RK3588_w8a8.rkllm
e90e160835c1a4d3d77095954aa518da513179ee0692b1bcc7a73f971f74d5a2  v1.3.1/RK3588/Qwen3-VL-2B-Instruct_vision_RK3588.rknn

Credits

Qwen provides the original model. Conversion uses Rockchip RKLLM, RKNN Toolkit2, and the conversion wrapper.