Qwen2.5-3B-Instruct-RKLLM

RKLLM-converted versions of Qwen2.5-3B-Instruct for Rockchip RK3576 and RK3588.

.rkllm files require the Rockchip RKLLM runtime and cannot be loaded directly with Transformers, llama.cpp, or Ollama.

Base Model

• Model: Qwen2.5-3B-Instruct
• Author: Alibaba Cloud Qwen Team
• Original Model: Qwen/Qwen2.5-3B-Instruct
• Original License: Qwen Research License

Refer to the upstream model card for the original model’s capabilities and limitations.

Supported Platforms

SoC	Quantization	Status
RK3576	W4A16	✅ Tested
RK3576	W8A8	✅ Tested
RK3588	W8A8	✅ Supported (not tested)

RK3576 W4A16 and W8A8 have been tested. RK3588 W8A8 is supported as a target platform but has not been board-tested because no RK3588 board is available.

Conversion

• RKLLM Toolkit: 1.2.3
• Target Platforms: RK3576, RK3588

Available Variants

File	Quantization
Qwen2.5-3B-Instruct_w4a16_RK3576.rkllm	W4A16
Qwen2.5-3B-Instruct_w8a8_RK3576.rkllm	W8A8
Qwen2.5-3B-Instruct_w8a8_RK3588.rkllm	W8A8

Repository Structure

.
├── Qwen2.5-3B-Instruct_data_quant.json
├── RK3576
│   ├── Qwen2.5-3B-Instruct_w4a16_RK3576.rkllm
│   └── Qwen2.5-3B-Instruct_w8a8_RK3576.rkllm
└── RK3588
    └── Qwen2.5-3B-Instruct_w8a8_RK3588.rkllm

Usage

Download the desired RKLLM model and deploy it with the RKLLM runtime on the matching Rockchip platform.

For faster deployment without setting up the environment manually, see Hanzo-Huang/rkllm-docker.

Docker Deployment

Use the model-specific image for the fastest RK3576 W4A16 deployment. The model is included in the image:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  ghcr.io/hanzo-huang/rkllm-docker/qwen2.5-3b-instruct:w4a16-rk3576

Alternatively, use the reusable environment image and mount a downloaded model:

sudo docker run --rm -it \
  --privileged \
  -p 8001:8001 \
  -v /dev:/dev \
  -v ./:/app/models:ro \
  -e MODEL_PATH=/app/models/RK3576/Qwen2.5-3B-Instruct_w4a16_RK3576.rkllm \
  -e TARGET_PLATFORM=rk3576 \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest

The second command requires downloading the selected .rkllm file first. Use the matching model variant and target platform.

License

The original Qwen2.5-3B-Instruct model is released under the Qwen Research License. Converted artifacts are derived from the original model and remain subject to the applicable upstream license terms.

Acknowledgements

Thanks to the Alibaba Cloud Qwen Team for releasing the original model and to Rockchip and RKLLM contributors for the deployment toolchain.