Qwen3.5-0.8B for RK3588 NPU
This repository provides a hardware-accelerated port of Qwen3.5-0.8B optimized for Rockchip RK3588 NPU.
User:<image>Describe the image.
Answer: This is a surreal, cinematic-style photograph taken from an astronaut’s perspective on the Moon. The scene captures a moment of quiet contemplation or perhaps a lighthearted celebration — possibly a beer toast — set against the vastness of space and Earth.
Foreground:
A person in a white, quilted astronaut suit with a fur-lined hood is seated on the lunar surface. They are holding a green glass bottle (likely beer) in one hand and resting their other arm on a small, dark green cooler or box labeled “Beer” — suggesting they’re enjoying a drink while relaxing. The texture of the suit shows stitching details, emphasizing realism despite the cosmic backdrop.
Midground:
The lunar regolith is gray and dusty, dotted with rocks and debris. To the right, a metal ladder leans against the ground, hinting at human presence or exploration history. In the distance, faint outlines of buildings suggest Earth looms behind them — its blue oceans and green land visible through the atmosphere’s haze.
Background:
The sky is filled with stars and swirling nebulae, creating a dreamlike, starry expanse. The curvature of Earth dominates the horizon, appearing as a large, glowing orb against the dark cosmos. Its surface shows continents and clouds, adding depth and scale to the image.
Lighting & Atmosphere:
The lighting is dramatic — strong highlights on the astronaut’s suit and bottle contrast with deep shadows in the cratered terrain. The overall mood is serene yet slightly melancholic, blending human presence with alien isolation. There’s a sense of solitude and wonder, as if humanity has finally found peace beyond Earth.
Style:
The image has a high-contrast, saturated look — reminiscent of digital art or post-processing — giving it a polished, almost staged feel. It evokes themes of exploration, friendship, and the beauty of space travel.
In short: This is not just an astronaut on the Moon — it’s a moment frozen in time, filled with emotion, isolation, and the quiet magic of being alone in the universe.
Model Files
| Component | File | Precision |
|---|---|---|
| LLM | qwen3.5-0.8b_w8a8_rk3588.rkllm |
W8A8 |
| Vision Encoder | qwen3.5-0.8b-vision_rk3588.rknn |
FP16 |
Hardware Requirements
- Rockchip RK3588 / RK3588S
- RKNPU2 driver
- Tested on:
- Rock 5C
- Ubuntu 22.04 / 24.04 (Joshua Riek)
Runtime Requirements
- RKLLM runtime (version 1.3.0)
- RKNN runtime (rknpu2)
- OpenCV (for image preprocessing)
Model performance benchmark (FPS)
All models, with C++ examples, can be found on the Q-engineering GitHub.
All LLM models are quantized to w8a8, while the VLM vision encoders use fp16.
| model | RAM (GB)1 | llm cold sec2 | llm warm sec3 | vlm cold sec2 | vlm warm sec3 | Resolution | Tokens/s |
|---|---|---|---|---|---|---|---|
| Qwen3.5-9B | 9.2 | 97.1 | 97.1 | 11.5 | 11.5 | 448 x 448 | 3.2 |
| Qwen3.5-4B | 5.4 | 52.8 | 6.2 | 8.5 | 0.9 | 448 x 448 | 5.2 |
| Qwen3.5-2B | 2.9 | 23.9 | 3.2 | 8.5 | 0.8 | 448 x 448 | 11.0 |
| Qwen3.5-0.8B | 1.3 | 10.6 | 1.9 | 2.7 | 0.2 | 448 x 448 | 21.6 |
| Qwen3-2B | 3.1 | 21.9 | 2.6 | 10.0 | 0.9 | 448 x 448 | 11.5 |
| Qwen3-4B | 8.7 | 49.6 | 5.6 | 10.6 | 1.1 | 448 x 448 | 5.7 |
| InternVL3.5-1B | 1.9 | 8.3 | 8.0 | 1.5 | 0.8 | 448 x 448 | 24 |
| InternVL3.5-2B | 3.0 | 22 | 8.0 | 2.7 | 0.8 | 448 x 448 | 11.2 |
| InternVL3.5-4B | 5.4 | 50 | 8.0 | 5.9 | 0.8 | 448 x 448 | 5 |
| InternVL3.5-8B | 8.8 | 92 | 8.0 | 50.5 | 5.8 | 448 x 448 | 3.5 |
| Qwen2.5-3B | 4.8 | 48.3 | 4.0 | 17.9 | 1.8 | 392 x 392 | 7.0 |
| Qwen2-7B | 8.7 | 86.6 | 34.5 | 37.1 | 20.7 | 392 x 392 | 3.7 |
| Qwen2-2.2B | 3.3 | 29.1 | 2.5 | 17.1 | 1.7 | 392 x 392 | 12.5 |
| InternVL3-1B | 1.3 | 6.8 | 1.1 | 7.8 | 0.75 | 448 x 448 | 30 |
| SmolVLM2-2.2B | 3.4 | 21.2 | 2.6 | 10.5 | 0.9 | 384 x 384 | 11 |
| SmolVLM2-500M | 0.8 | 4.8 | 0.7 | 2.5 | 0.25 | 384 x 384 | 31 |
| SmolVLM2-256M | 0.5 | 1.1 | 0.4 | 2.5 | 0.25 | 384 x 384 | 54 |
1 The total used memory; LLM plus the VLM.
2 When an llm/vlm model is loaded for the first time from your disk to RAM or NPU, it is called a cold start.
The duration depends on your OS, I/O transfer rate, and memory mapping.
3 Subsequent loading (warm start) takes advantage of the already mapped data in RAM. Mostly, only a few pointers need to be restored.
Example Usage
Notes
- KV Cache up to 16384
- This is not a Transformers-compatible model
- This repository provides precompiled NPU binaries
- CPU fallback is not supported
- Downloads last month
- 39