UkoTerpuh/Qwen3.5-9B-RK3588-RKLLM-1.3.1-W8A8-G512

Qwen3.5-9B — experimental RK3588 RKLLM 1.3.1 conversion

Experimental text-only conversion. Not recommended as a quality-equivalent replacement for the upstream model or a validated GGUF deployment. This is a quantized conversion of official Qwen weights, not a fine-tune or merge. Repository hosting does not imply hosted inference support.

Artifact

• File: qwen3.5-9b-stock-w8a8-g512-rk3588-ctx8192.rkllm
• Size: 10,202,149,748 bytes.
• SHA-256: eb60f9d229fed13bf0b140b06bbe0e2b1f0ccd02090a39934144d12697c8eccd
• Target: Rockchip RK3588, three NPU cores; RKLLM 1.3.1 runtime.
• Compiled maximum context: 8192 tokens. This is not a guarantee that every input of that length fits after chat framing or on every device.
• Only text inference was converted and evaluated. No vision encoder is included.

Conversion provenance

• Source: Qwen/Qwen3.5-9B, revision c202236235762e1c871ad0ccb60c8ee5ba337b9a.
• Rockchip RKLLM toolkit: 1.3.1, SDK repository commit f7390530443bf84f0394255a449d7cbe81e69d1c.
• CPU conversion; float32 source loading; PyTorch 2.6.0; Transformers 5.8.0.
• Quantization: w8a8_g512, algorithm normal, optimization level 1, hybrid_rate=0, target rk3588.
• Calibration: 128 public text-only training examples, 64 English and 64 Ukrainian, 64–512 tokens each, sampled deterministically with seed 20261022.
• Calibration source: CohereLabs/aya_collection_language_split, revision a3af2fde4b4cb5b2775830b11244a1a20b5f004f. Calibration text is not redistributed here.
• Official source tokenizer/chat template; evaluated with enable_thinking=False.

Validation and limitations

Evaluated on RK3588 using native RKLLM 1.3.1 through RKLLama, with a service RAM limit of 14.5 GiB and swap limit of 0.5 GiB. These are deployment observations, not universal hardware requirements.

A private application-specific quality/API suite scored 56/80, compared with 76/80 for a retained stock Q8_0 GGUF Rockchip llama.cpp/NPU deployment. This is not a public standard benchmark or a controlled engine-only comparison. The tests are not distributed in this repository.

Important distinctions:

• Repeated multiplication test: 137 * 29 produced 4013, not 3973, in all 14 quality/load requests using that question.
• Midnight rollover produced 24:15 rather than 00:15; the GGUF reference also failed that case.
• The largest original context test returned empty HTTP-200 responses.
• Some strict-schema and tool-selection requests were rejected by the API serving layer. These are serving-stack limitations, not demonstrated reasoning failures of the weights.
• Simple multilingual, instruction-boundary and ordinary JSON checks passed.
• A separate API-adapted 12-request resource replay completed without OOM or request timeout, with a kernel-recorded service RAM peak of approximately 12.20 GiB. Some exact-output assertions still failed. Completion is not correctness.
• The adapted resource replay removed an unsupported stream_options request field; it did not establish full OpenAI API compatibility.
• The compared deployments did not have fully matched repetition/presence penalties and sampling defaults. Quantization, runtime implementation, source revision identity and serving defaults are potential contributors; the exact cause of the quality gap has not been isolated.
• The GGUF reference already used Rockchip NPU acceleration. This is not a CPU-versus-NPU comparison.
• Native long-prompt cold latency was similar to the reference, and the tested deployment did not show comparable immediate-repeat prompt-cache acceleration.

Use the exact source tokenizer and a compatible RKLLM runtime/wrapper. A successful load or a basic greeting is not sufficient validation for deployment. No hardware-specific service configurations, credentials, private fixtures, or operational logs are included.

Download and runtime guidance

Download the artifact with the Hugging Face CLI:

hf download UkoTerpuh/Qwen3.5-9B-RK3588-RKLLM-1.3.1-W8A8-G512 qwen3.5-9b-stock-w8a8-g512-rk3588-ctx8192.rkllm SHA256SUMS --local-dir ./qwen35-rk3588
cd qwen35-rk3588
sha256sum -c SHA256SUMS

If this repository is private, authenticate with an authorized Hugging Face account first. Public downloads do not require write access. Allow enough disk space for the model and downloader temporary files.

This is an RKLLM artifact, not a GGUF file: it cannot be loaded directly by ordinary llama.cpp. Use an RK3588 device and a compatible RKLLM 1.3.1 runtime/application. Obtain vendor software and API examples from the pinned Rockchip SDK. The validation used an RKLLama wrapper with RKLLM 1.3.1-compatible bindings; this repository does not include that deployment or promise compatibility with every wrapper version.

Use tokenizer assets from the exact upstream source revision, including its chat template and special token definitions. Evaluation disabled thinking with enable_thinking=False; use the equivalent supported setting in your application. Keep context and generation limits within both the artifact limit and your device's available memory. Test actual answers before adoption: a successful initialization is not evidence of correct inference. No hosted inference endpoint is supplied.

Licensing and attribution

The upstream Qwen3.5-9B weights are distributed under Apache-2.0, including Copyright 2026 Alibaba Cloud. The exact upstream license is included as LICENSE; NOTICE identifies this artifact and its conversion modifications. The model-card license metadata applies to the converted model, not to third-party software.

The Rockchip license at the pinned toolkit commit permits source/binary redistribution subject to its notice, non-endorsement and third-party-license conditions. The inspected license and CPython 3.10 toolkit 1.3.1 wheel metadata contained no additional restriction on distributing converted model outputs. A verbatim copy of that license, including its ggml/llama.cpp MIT notices, is supplied conservatively in THIRD_PARTY_NOTICES.txt. This is a scoped review of those materials, not a legal warranty or a claim about terms outside them. No vendor toolkit, runtime libraries or executables are redistributed here; obtain and use them under their own applicable terms.

Calibration used the pinned public dataset identified above, whose dataset card declares Apache-2.0. Calibration examples are not redistributed here. This is an unofficial experimental conversion and is not endorsed by Qwen, Alibaba Cloud, Rockchip or the calibration dataset authors.