Micklavin/Qwen3.5-9B-4bit

Qwen3.5-9B — 4-bit MLX

A 4-bit quantization of Qwen/Qwen3.5-9B for MLX on Apple Silicon.

Produced for Triad, a local three-model token-fusion experiment. These are the exact weights that project was built and validated against, published so its results can be reproduced.

Size on disk: 4.7 GB (down from ~18.4 GB at bf16)

Requirements

• Apple Silicon
• mlx-lm >= 0.31.3

pip install "mlx-lm>=0.31.3"

The PyPI release is sufficient — this model's model_type: qwen3_5 is supported natively (mlx_lm/models/qwen3_5.py ships in the wheel). No extra packages needed.

Usage

from mlx_lm import generate, load

model, tokenizer = load("Micklavin/Qwen3.5-9B-4bit")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is the chemical symbol for gold?"}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Thinking mode

Qwen3.5's chat template supports enable_thinking. Triad disables it so all ensemble members answer in the same phase:

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)

Quantization provenance

Converted with mlx_lm.convert:

convert(
    hf_path="Qwen/Qwen3.5-9B",
    mlx_path="models/qwen-4bit",
    quantize=True,
    q_bits=4,
    q_group_size=64,
    dtype="bfloat16",
)

Resulting config: {"group_size": 64, "bits": 4, "mode": "affine"}, model_type: qwen3_5.

Component	Version
mlx	0.32.0
mlx-lm	0.31.3
transformers	5.12.1

The reproduction script is scripts/quantize_models.py.

Evaluation

None beyond a smoke test. No benchmark comparison against the bf16 original was run, so the quantization's quality cost is unmeasured. Treat it as an untested 4-bit conversion rather than a validated one.

License

Apache 2.0, inherited from the base model. See the base model's LICENSE. Quantization does not alter the license or your obligations under it.