Robin1333k/Qwen3.5-2B-Base-MLX-6bit

Qwen3.5-2B-Base — MLX, 6-bit

A 6-bit MLX quantization of Qwen/Qwen3.5-2B-Base, converted with mlx_lm.convert (--q-bits 6 --q-group-size 64, affine). ~1.5 GB, for on-device inference on Apple Silicon.

Only the weights are quantized — this is not a fine-tune. The behaviour is the base model's.

Why 6-bit and not 4-bit

The obvious choice for a ~2B model on-device is 4-bit, and for this model it is a mistake. Measured on a word-acceptance bake-off (higher is better — mean words accepted per suggestion, EN/DE, from a flush and a trailing-space prefix):

quantization	size	EN flush	EN space	DE flush	DE space
4-bit (affine, g64)	1.0 GB	1.25	0.58	1.42	0.42
mixed 4/6-bit	1.0 GB	1.25	0.58	1.42	0.42
6-bit (affine, g64)	1.5 GB	2.08	0.54	1.50	0.58
bf16 (reference)	4.3 GB	2.08	0.75	1.50	0.58

Naive 4-bit loses ~40 % on English against bf16 (1.25 vs 2.08) — enough to put it below a smaller 1.7B model, i.e. the quantization gave back more than the extra parameters bought. mixed_4_6 measured identical to plain 4-bit here. 6-bit recovers full bf16 quality on every flush column.

Its one regression is EN trailing-space (0.54 vs 0.75), which does not matter for the intended use: the consumer strips trailing whitespace before inference, so that column is never exercised.

The general lesson, if you are quantizing a small base model yourself: re-check against bf16. The 4-bit build looked fine in casual use and was 40 % worse where it counted.

Intended use

Built for AutoComplete, a system-wide inline text-prediction tool for macOS — the model predicts the continuation of what you are typing, a few words at a time. That shapes how it is used, and the conventions are worth repeating if you want comparable behaviour:

• Raw prefix, no chat template, no BOS. It is a base model. Wrapping the prompt in a chat template makes it answer questions instead of continuing text.
• A flush prefix — trailing whitespace stripped before encoding, the leading space reconciled on render, so foo and foo produce the same prompt.
• Greedy decoding (temperature 0) with a mild repetition penalty, stopped at the first clause boundary after content.

chat_template.jinja ships as it came from the original model, for anyone who wants it.

Languages

Verified in English and German. German is the weaker axis — the bake-off measures roughly half the mid-clause acceptance rate of English, and in longer generations it produces occasional hard grammar errors (verb-final order in subordinate clauses) and English-shaped collocations. That is a property of the base model at this size, not of the quantization: 6-bit matches bf16 on German exactly.

Provenance

Converted from the official Qwen/Qwen3.5-2B-Base weights. Apache 2.0, inherited from the base model — see the license link above. All credit for the model belongs to the Qwen team; this repository adds a quantization and a measurement, nothing more.