Qwen3-4B-Instruct-2507 — RKLLM v1.2.3 (w8a8, RK3588)

RKLLM conversion of Qwen/Qwen3-4B-Instruct-2507 for Rockchip RK3588 NPU inference.

Converted with RKLLM Toolkit v1.2.3. The 2507 suffix denotes the July 2025 refresh of Qwen3-4B with improved instruction following.

Note: This model does not produce <think> reasoning blocks in this RKLLM build (thinking mode is disabled). See RKLLM thinking mode limitations.

Key Details

Property	Value
Base Model	Qwen/Qwen3-4B-Instruct-2507
Toolkit Version	RKLLM Toolkit v1.2.3
Runtime Version	RKLLM Runtime ≥ v1.2.1 (v1.2.3 recommended)
Quantization	w8a8, group size 128
Target Platform	RK3588
NPU Cores	3
Max Context Length	16384 tokens
Optimization Level	1
Hybrid Ratio	0.0
Thinking Mode	❌ Disabled
Languages	English, Chinese (multilingual)

Why This Model?

Qwen3-4B-Instruct-2507 is the July 2025 update to Alibaba's Qwen3-4B, with improved reasoning and instruction following. At 4B parameters it is the largest text-only model in this RK3588 lineup and handles complex prompts well despite the quantization.

The 16k context window (vs 8k on smaller models) enables long document summarisation and multi-turn conversations.

Hardware Tested

Orange Pi 5 Plus — RK3588, 16GB RAM, Armbian Linux
RKNPU driver 0.9.8
RKLLM Runtime v1.2.3

Usage

With the RKLLM API Server

mkdir -p ~/models/Qwen3-4B-Instruct-2507
cd ~/models/Qwen3-4B-Instruct-2507
git lfs install && git clone https://huggingface.co/GatekeeperZA/Qwen3-4B-Instruct-2507-RKLLM-v1.2.3 .

Use with GatekeeperZA/RKLLM-API-Server.

With the official RKLLM API demo

git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm/examples/rkllm_api_demo
./build/rkllm_api_demo /path/to/Qwen3-4B-Instruct-2507-rk3588-w8a8_g128-opt-1-hybrid-ratio-0.0-16k.rkllm 8192 16384

File Listing

File	Description
`Qwen3-4B-Instruct-2507-rk3588-w8a8_g128-opt-1-hybrid-ratio-0.0-16k.rkllm`	Quantized model for RK3588 NPU

Compatibility Notes

Minimum runtime: RKLLM Runtime v1.2.1. v1.2.3 recommended.
RKNPU driver: ≥ 0.9.6
SoCs: RK3588 / RK3588S. Not compatible with RK3576 without reconversion.
RAM: ~5.5GB loaded. Requires 8GB+ board (16GB recommended).

Acknowledgements

Alibaba Qwen Team for Qwen3
Rockchip / airockchip for the RKLLM toolkit and runtime
Converted by GatekeeperZA

Downloads last month: -

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for GatekeeperZA/Qwen3-4B-Instruct-2507-RKLLM-v1.2.3

Base model

Qwen/Qwen3-4B-Instruct-2507

Finetuned

(1929)

this model