codelion/Ornith-1.0-9B-OptiQ-6bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

A 6-bit mixed-precision MLX quant of deepreinforce-ai/Ornith-1.0-9B, built on the Qwen3.5-9B architecture. Sensitive layers are kept at 8-bit and robust ones at 4-bit.

17.6 GB of bf16 weights become 8.0 GB, which fits a 16 GB Mac.

Image input works. The vision tower is kept at bf16 in a sidecar, so this quant takes images as well as text.

Quantization details

Property	Value
Predominant precision	6-bit
Layers at 8-bit (sensitive)	145
Layers at 4-bit (robust)	104
Total quantized layers	249
Group size	64
Vision tower	bf16, 333 tensors, in optiq/optiq_vision.safetensors
Size on disk	8.0 GB, from a 17.6 GB bf16 base

We follow the same naming convention llama.cpp uses for Q6_K and similar mixed-precision quants: the "6-bit" label is the predominant precision, not the weighted average.

The base model ships no MTP head, so this quant has no speculative-decoding sidecar.

How the bit-widths were chosen

Every layer was measured directly on this model. Each (layer, bit-width) pair was scored by KL divergence against the bf16 reference on a six-domain calibration mix, and a knapsack solver spent the bit budget where the measured error was largest.

The full sweep ships with the model as optiq/sensitivity.json: 249 layers scored at both candidate widths. That is the measurement, not just the outcome, so this architecture can be re-quantized at another target without repeating it.

Only the language tower is quantized. The vision tower stays at bf16, which is how every OptiQ VLM ships.

Usage

Text

Everything OptiQ-specific lives in an optiq/ subfolder, so a stock *.safetensors glob ignores it and mlx-lm sees a clean language model.

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("codelion/Ornith-1.0-9B-OptiQ-6bit")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain the difference between TCP and UDP."}],
    add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

This is a reasoning model: it thinks inside <think>...</think> before answering, so give it enough max_tokens to finish.

Images

Image input needs mlx-optiq, which loads the bf16 vision sidecar and feeds the merged embeddings to the quantized language tower:

pip install mlx-optiq

from PIL import Image
from optiq.runtime.engine import OptiqEngine

engine = OptiqEngine("codelion/Ornith-1.0-9B-OptiQ-6bit")
answer = engine.generate("What is in this image?",
                         images=[Image.open("photo.jpg")], max_tokens=512)
print(answer.text)

Or serve it over an OpenAI-compatible endpoint that accepts image content parts:

optiq serve --model codelion/Ornith-1.0-9B-OptiQ-6bit

Verification

Text, arithmetic reasoning, and image understanding were all exercised on the finished artifact before release.

No task benchmarks were run on this quant; for measured quality numbers on the base architecture, see the Qwen3.5-9B OptiQ card.

Quantization does not change the behaviour or alignment of the base model. Use it under the same terms as the original.