Qwen3.8-9B-Distill-MLX-6bit — Q6-class / ~6-bit MLX mixed-precision

MLX-native quantized version of Qwen3.8-9B-Distill, built with oMLX Universal Dynamic Quantization (oQe). Standard mlx-lm safetensors — format-compatible with mlx-lm, oMLX, LM Studio, and mlx-swift.

Repository naming: Originally published as RolanDorisTech/Qwen3.8-9B-Distill-MLX-oQ6e. Renamed to RolanDorisTech/Qwen3.8-9B-Distill-MLX-6bit for clearer discovery. Weights and oQ6e quantization unchanged — old URLs redirect automatically.

Search terms: Qwen3.8 9B MLX Q6, Qwen3.8 9B MLX 6-bit, Qwen3.8 9B MLX 6bit, RolanDorisTech/Qwen3.8-9B-Distill-MLX-oQ6e

Technical quantization: oQ6e — Q6-class mixed-precision with activation-importance weighting (imatrix). Q6, 6-bit, 6bit are common search terms for this class — not the same algorithm as standard Q6_K_M / uniform 6-bit.

Averages ~6 bits per weight (mixed-precision, non-uniform — not the same algorithm as standard Q6_K_M/uniform 6-bit). If you're looking for a 6-bit MLX build of this model, this is it.

YouTube: Apple Silicon Local AI tutorials and benchmarks
https://www.youtube.com/@RolanDorisTech

Contents

• Quick Facts
• What are oQ and oQe?
• How to Use
• Family
• Measured Performance
• Architecture & Compatibility
• Base Model Provenance
• Limitations
• Credits
• References

Quick Facts

Size	7.0GB
Quant	oQ6e (mixed + activation-importance)
Base	empero-ai/Qwen3.8-9B-Distill (Qwen/Qwen3.5-9B architecture)
Context	262,144 tokens native
License	Apache-2.0
Format	MLX safetensors (mlx-lm, oMLX, LM Studio, mlx-swift)
YouTube	RolanDorisTech — Apple Silicon Local AI

What are oQ and oQe?

oQ measures each layer's real quantization sensitivity and allocates bits accordingly, instead of a flat uniform budget. oQe adds activation-importance weighting (imatrix) on top.

Full methodology, benchmarks, and research citations

Not all weights matter equally. Layers like lm_head, token embeddings, and the first and last transformer blocks are disproportionately sensitive: a small quantization error there can distort the output distribution and degrade reasoning. Other layers tolerate much more aggressive compression with little effect.

Conventional uniform quantization uses a largely fixed precision policy, whereas oQ measures layer sensitivity and allocates a mixed-precision budget accordingly.

oQ replaces that flat budget with a measured one. It runs calibration data through the model, measures each layer's real sensitivity to quantization error, and allocates more bits to layers where error hurts most and fewer bits where it doesn't.

oQe goes further: it adds an activation-importance calibration pass and uses per-channel activation statistics (an imatrix) to weight quantization error, giving greater importance to errors on channels that matter more during calibration.

Why it matters (published oMLX benchmark on Qwen3.5-35B-A3B, MMLU, 300 samples):

Bits	Uniform Q (mlx-lm)	oQ
2-bit	14.0%	64.0%
3-bit	76.3%	85.0%
4-bit	79.7%	83.3%

The 2-bit uniform result was substantially lower in this test.

Note: This benchmark evaluates oQ on Qwen3.5-35B-A3B; it documents the oQ methodology and is not a quality benchmark for these oQe Qwen3.8 Distill models.

For full methodology see: https://github.com/jundot/omlx/blob/main/docs/oQ_Quantization.md

Related quantization research also shows that sensitivity- and activation-aware weighting can improve weight-only quantization results:

• SqueezeLLM (Kim et al., arXiv:2306.07629) reports LLaMA-7B C4 perplexity of 28.26 for uniform RTN, 18.08 for sensitivity-agnostic non-uniform quantization, and 7.75 for sensitivity-based non-uniform quantization.
• vLLM's LLM Compressor, using its imatrix_mse observer, reduces WikiText-2 perplexity on Llama-3.1-8B (W4A16) from 6.96 to 6.85, and to 6.83 with GPTQ. Docs: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/imatrix/

The takeaway: in this specific MMLU test, oQ achieved higher accuracy than uniform quantization at the same nominal bit width. This benchmark is on Qwen3.5-35B-A3B and documents the oQ methodology; it is not a quality benchmark for these Qwen3.8 Distill oQe models.

Output is standard MLX safetensors — format-compatible with mlx-lm, oMLX, LM Studio, and mlx-swift. Runtime compatibility should be verified against the specific application/version.

These Qwen3.8 Distill models in this release use oQe.

How to Use

Install:

pip install mlx-lm

Or if you use uv:

uv pip install mlx-lm

Generate:

mlx_lm.generate --model RolanDorisTech/Qwen3.8-9B-Distill-MLX-6bit --prompt "Explain oQ vs oQe" --max-tokens 250 --temp 0.6 --top-p 0.95 --top-k 20

LM Studio: Search and download:

RolanDorisTech/Qwen3.8-9B-Distill-MLX-6bit

Prompt format uses Qwen3 chat template with <think> tags. chat_template.jinja is included.

Family

All three sizes are community distills of Qwen/Qwen3.8-2.4T-A95B into Qwen3.5 architecture, NOT official Alibaba releases.

Variant	Size	Base	Traces	Base Distill
2B-oQ4e	1.1GB	Qwen/Qwen3.5-2B	~30,000	empero-ai/Qwen3.8-2B-Distill
2B-oQ8e	1.9GB	Qwen/Qwen3.5-2B	~30,000	empero-ai/Qwen3.8-2B-Distill
4B-oQ4e	2.3GB	Qwen/Qwen3.5-4B	~45,000	empero-ai/Qwen3.8-4B-Distill
4B-oQ8e	4.2GB	Qwen/Qwen3.5-4B	~45,000	empero-ai/Qwen3.8-4B-Distill
9B-oQ4e	4.9GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill
9B-oQ5e	6.0GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill
9B-oQ6e (this model)	7.0GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill
9B-oQ8e	8.9GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill

Note: This is a community distillation, NOT an official Alibaba/Qwen release.

Measured Performance

Local measurement by RolanDorisTech on M1 Max 64GB; not an upstream Empero or Qwen benchmark.

(Accuracy benchmarks to come — community evaluations like MMLU, GSM8K, HumanEval etc. will be published in a future update when time allows; current numbers are local speed only. Planning to queue overnight in oMLX with extended thinking ON.)

Test prompt: "Q: If 2x + 3 = 11, what is x? Think step by step inside tags." — Result correct x=4, preserved.

For this Qwen3.8-9B-Distill-MLX-6bit model: 37 tokens prompt at 84.3 tok/s, 33.3 tok/s gen, peak 7.664 GB

Variant	Size	Prompt tok/s	Gen tok/s	Peak Mem	Quant Time
2B-oQ4e	1.1GB	267.7	118.6	1.241	1m43s
2B-oQ8e	1.9GB	122.2	97.5	2.131	43s
4B-oQ4e	2.3GB	133.7	65.4	2.684	4m01s
4B-oQ8e	4.2GB	130.9	48.4	4.651	1m46s
9B-oQ4e	4.9GB	101.5	43.0	5.471	7m23s
9B-oQ5e	6.0GB	91.3	37.1	6.555	3m20s
9B-oQ6e (this)	7.0GB	84.3	33.3	7.664	3m21s
9B-oQ8e	8.9GB	91.8	28.7	9.679	3m24s

Architecture & Compatibility

• Architecture: Qwen3.5 hybrid — repeating groups of 3 Gated DeltaNet blocks followed by 1 Gated Attention block. 248,320-token vocabulary (padded).
• Native context: 262,144 tokens inherited from Qwen3.5 base.
• Requirements: recent mlx-lm / transformers with Qwen3.5 + Gated DeltaNet support. For PyTorch/CUDA, needs flash-linear-attention and causal_conv1d kernels — without them, linear-attention layers fall back to slow PyTorch ops.
• Sampler: temperature=0.6, top_p=0.95, top_k=20 (Qwen3.5 recommended). Greedy decoding can cause repetition loops.
• Output: The model is trained to open responses with a block; allow up to 16,384 new tokens and strip the ... span when hiding reasoning from end users.

Base Model Provenance

Note: This model is quantized from empero-ai/Qwen3.8-9B-Distill, a community model developed by Empero, not from an official Alibaba/Qwen 9B checkpoint.

• Teacher: Qwen/Qwen3.8-2.4T-A95B teacher
• Student base: Qwen/Qwen3.5-9B
• Training: SFT (off-policy distillation) on ~70,000 teacher traces
• Lineage: Qwen/Qwen3.8-2.4T-A95B → empero-ai/Qwen3.8-9B-Distill → MLX BF16 → oQe quant

For full lineage: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill

The Qwen3.8 open-model releases documented by Qwen include Qwen/Qwen3.8-2.4T-A95B and Qwen/Qwen3.8-27B; this release should not be described as an Alibaba/Qwen "Qwen3.8-9B" checkpoint.

Limitations

Accuracy benchmarks (MMLU, GSM8K, etc.) for these specific released models have not been run yet — only inference speed above is measured. Benchmarking is planned but takes significant compute time per model; results will be added when available.

Distilled reasoning model. May hallucinate. Lossy relative to BF16. Quantization quality depends on model, quantization level, and calibration; no direct same-size accuracy comparison with uniform g32/g64 is claimed here.

Text-trained/fine-tuned; vision behavior is inherited from the Qwen3.5 base and was not evaluated for this release.

Native context: 262,144 tokens inherited from Qwen3.5; practical usable context depends on the inference runtime, KV-cache implementation, and available memory.

Credits

• Base: empero-ai/Qwen3.8-9B-Distill (safetensors) — itself a full-parameter distillation of Qwen/Qwen3.8-2.4T-A95B into Qwen/Qwen3.5-9B, Apache 2.0. GGUF repo empero-ai/Qwen3.8-9B-Distill-GGUF is a separate quantized artifact for llama.cpp, not used as source for this MLX conversion. Released by Empero under Apache-2.0; upstream card states license inherited from base.
• Quantization: oMLX Universal Dynamic Quantization (oQe). Built by RolanDorisTech, September 23, 2026.
• Machine: M1-Max Mac Studio 64GB, 32 Core GPU, macOS 27.0
• Tool: oMLX Quantization panel
• Base conversion: Qwen3.8-9B-Distill BF16 MLX master 17.2GB (converted from empero-ai/Qwen3.8-9B-Distill safetensors, not GGUF)
• Method: oQe ON, Reuse ON, Automatic cache, Strict OFF, Preserve MTP OFF, bfloat16 for norms and scales
• Protection: lm_head 8-bit, embedding and early/late layers boosted per the oQe quantization configuration

References

Methodology:

• oQ docs: https://github.com/jundot/omlx/blob/main/docs/oQ_Quantization.md
• SqueezeLLM paper: https://arxiv.org/abs/2306.07629
• vLLM imatrix docs: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/imatrix/

Base model lineage:

• Empero 9B Distill: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill
• Empero 2B Distill: https://huggingface.co/empero-ai/Qwen3.8-2B-Distill
• Empero 4B Distill: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill
• GGUF variant (not used as source): https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF

Official Qwen releases:

• Official Qwen3.8 repo: https://github.com/QwenLM/Qwen3.8
• Qwen3.8-2.4T-A95B: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
• Qwen3.8-27B: https://huggingface.co/Qwen/Qwen3.8-27B
• Qwen3.5-9B base: https://huggingface.co/Qwen/Qwen3.5-9B

Video guides: YouTube: Apple Silicon Local AI tutorials and benchmarks
https://www.youtube.com/@RolanDorisTech