Instructions to use RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-4B-Distill-MLX-4bit — Q4-class / ~4-bit MLX mixed-precision

MLX-native quantized version of Qwen3.8-4B-Distill, built with oMLX Universal Dynamic Quantization (oQe). Standard mlx-lm safetensors — format-compatible with mlx-lm, oMLX, LM Studio, and mlx-swift.

Repository naming: Originally published as RolanDorisTech/Qwen3.8-4B-Distill-MLX-oQ4e. Renamed to RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit for clearer discovery. Weights and oQ4e quantization unchanged — old URLs redirect automatically.

Search terms: Qwen3.8 4B MLX Q4, Qwen3.8 4B MLX 4-bit, Qwen3.8 4B MLX 4bit, RolanDorisTech/Qwen3.8-4B-Distill-MLX-oQ4e

Technical quantization: oQ4e — Q4-class mixed-precision with activation-importance weighting (imatrix). Q4, 4-bit, 4bit are common search terms for this class — not the same algorithm as standard Q4_K_M / uniform 4-bit.

Averages ~4 bits per weight (mixed-precision, non-uniform — not the same algorithm as standard Q4_K_M/uniform 4-bit). If you're looking for a 4-bit MLX build of this model, this is it.

YouTube: Apple Silicon Local AI tutorials and benchmarks
https://www.youtube.com/@RolanDorisTech

Quick Facts
What are oQ and oQe?
How to Use
Family
Measured Performance
Architecture & Compatibility
Base Model Provenance
Limitations
Credits
References

Quick Facts


Size	2.3GB
Quant	oQ4e (mixed + activation-importance)
Base	empero-ai/Qwen3.8-4B-Distill (Qwen/Qwen3.5-4B architecture)
Context	262,144 tokens native
License	Apache-2.0
Format	MLX safetensors (mlx-lm, oMLX, LM Studio, mlx-swift)
YouTube	RolanDorisTech — Apple Silicon Local AI

What are oQ and oQe?

oQ measures each layer's real quantization sensitivity and allocates bits accordingly, instead of a flat uniform budget. oQe adds activation-importance weighting (imatrix) on top.

Full methodology, benchmarks, and research citations

Not all weights matter equally. Layers like lm_head, token embeddings, and the first and last transformer blocks are disproportionately sensitive: a small quantization error there can distort the output distribution and degrade reasoning. Other layers tolerate much more aggressive compression with little effect.

Conventional uniform quantization uses a largely fixed precision policy, whereas oQ measures layer sensitivity and allocates a mixed-precision budget accordingly.

oQ replaces that flat budget with a measured one. It runs calibration data through the model, measures each layer's real sensitivity to quantization error, and allocates more bits to layers where error hurts most and fewer bits where it doesn't.

oQe goes further: it adds an activation-importance calibration pass and uses per-channel activation statistics (an imatrix) to weight quantization error, giving greater importance to errors on channels that matter more during calibration.

Why it matters (published oMLX benchmark on Qwen3.5-35B-A3B, MMLU, 300 samples):

Bits	Uniform Q (mlx-lm)	oQ
2-bit	14.0%	64.0%
3-bit	76.3%	85.0%
4-bit	79.7%	83.3%

The 2-bit uniform result was substantially lower in this test.

Note: This benchmark evaluates oQ on Qwen3.5-35B-A3B; it documents the oQ methodology and is not a quality benchmark for these oQe Qwen3.8 Distill models.

For full methodology see: https://github.com/jundot/omlx/blob/main/docs/oQ_Quantization.md

Related quantization research also shows that sensitivity- and activation-aware weighting can improve weight-only quantization results:

SqueezeLLM (Kim et al., arXiv:2306.07629) reports LLaMA-7B C4 perplexity of 28.26 for uniform RTN, 18.08 for sensitivity-agnostic non-uniform quantization, and 7.75 for sensitivity-based non-uniform quantization.
vLLM's LLM Compressor, using its imatrix_mse observer, reduces WikiText-2 perplexity on Llama-3.1-8B (W4A16) from 6.96 to 6.85, and to 6.83 with GPTQ. Docs: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/imatrix/

The takeaway: in this specific MMLU test, oQ achieved higher accuracy than uniform quantization at the same nominal bit width. This benchmark is on Qwen3.5-35B-A3B and documents the oQ methodology; it is not a quality benchmark for these Qwen3.8 Distill oQe models.

Output is standard MLX safetensors — format-compatible with mlx-lm, oMLX, LM Studio, and mlx-swift. Runtime compatibility should be verified against the specific application/version.

These Qwen3.8 Distill models in this release use oQe.

How to Use

Install:

pip install mlx-lm

Or if you use uv:

uv pip install mlx-lm

Generate:

mlx_lm.generate --model RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit --prompt "Explain oQ vs oQe" --max-tokens 250 --temp 0.6 --top-p 0.95 --top-k 20

LM Studio: Search and download:

RolanDorisTech/Qwen3.8-4B-Distill-MLX-4bit

Prompt format uses Qwen3 chat template with <think> tags. chat_template.jinja is included.

Family

All three sizes are community distills of Qwen/Qwen3.8-2.4T-A95B into Qwen3.5 architecture, NOT official Alibaba releases.

Variant	Size	Base	Traces	Base Distill
2B-oQ4e	1.1GB	Qwen/Qwen3.5-2B	~30,000	empero-ai/Qwen3.8-2B-Distill
2B-oQ8e	1.9GB	Qwen/Qwen3.5-2B	~30,000	empero-ai/Qwen3.8-2B-Distill
4B-oQ4e (this model)	2.3GB	Qwen/Qwen3.5-4B	~45,000	empero-ai/Qwen3.8-4B-Distill
4B-oQ8e	4.2GB	Qwen/Qwen3.5-4B	~45,000	empero-ai/Qwen3.8-4B-Distill
9B-oQ4e	4.9GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill
9B-oQ5e	6.0GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill
9B-oQ6e	7.0GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill
9B-oQ8e	8.9GB	Qwen/Qwen3.5-9B	~70,000	empero-ai/Qwen3.8-9B-Distill

Note: This is a community distillation, NOT an official Alibaba/Qwen release.

Measured Performance

Local measurement by RolanDorisTech on M1 Max 64GB; not an upstream Empero or Qwen benchmark.

(Accuracy benchmarks to come — community evaluations like MMLU, GSM8K, HumanEval etc. will be published in a future update when time allows; current numbers are local speed only. Planning to queue overnight in oMLX with extended thinking ON.)

Test prompt: "Q: If 2x + 3 = 11, what is x? Think step by step inside tags." — Result correct x=4, preserved.

For this Qwen3.8-4B-Distill-MLX-4bit model: 37 tokens prompt at 133.7 tok/s, 65.4 tok/s gen, peak 2.684 GB

Variant	Size	Prompt tok/s	Gen tok/s	Peak Mem	Quant Time
2B-oQ4e	1.1GB	267.7	118.6	1.241	1m43s
2B-oQ8e	1.9GB	122.2	97.5	2.131	43s
4B-oQ4e (this)	2.3GB	133.7	65.4	2.684	4m01s
4B-oQ8e	4.2GB	130.9	48.4	4.651	1m46s
9B-oQ4e	4.9GB	101.5	43.0	5.471	7m23s
9B-oQ5e	6.0GB	91.3	37.1	6.555	3m20s
9B-oQ6e	7.0GB	84.3	33.3	7.664	3m21s
9B-oQ8e	8.9GB	91.8	28.7	9.679	3m24s

Architecture & Compatibility

Architecture: Qwen3.5 hybrid — repeating groups of 3 Gated DeltaNet blocks followed by 1 Gated Attention block. 248,320-token vocabulary (padded).
Native context: 262,144 tokens inherited from Qwen3.5 base.
Requirements: recent mlx-lm / transformers with Qwen3.5 + Gated DeltaNet support. For PyTorch/CUDA, needs flash-linear-attention and causal_conv1d kernels — without them, linear-attention layers fall back to slow PyTorch ops.
Sampler: temperature=0.6, top_p=0.95, top_k=20 (Qwen3.5 recommended). Greedy decoding can cause repetition loops.
Output: The model is trained to open responses with a block; allow up to 16,384 new tokens and strip the ... span when hiding reasoning from end users.

Base Model Provenance

Note: This model is quantized from empero-ai/Qwen3.8-4B-Distill, a community model developed by Empero, not from an official Alibaba/Qwen 4B checkpoint.

Teacher: Qwen/Qwen3.8-2.4T-A95B teacher
Student base: Qwen/Qwen3.5-4B
Training: SFT (off-policy distillation) on ~45,000 teacher traces
Lineage: Qwen/Qwen3.8-2.4T-A95B → empero-ai/Qwen3.8-4B-Distill → MLX BF16 → oQe quant

For full lineage: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill

The Qwen3.8 open-model releases documented by Qwen include Qwen/Qwen3.8-2.4T-A95B and Qwen/Qwen3.8-27B; this release should not be described as an Alibaba/Qwen "Qwen3.8-4B" checkpoint.

Limitations

Accuracy benchmarks (MMLU, GSM8K, etc.) for these specific released models have not been run yet — only inference speed above is measured. Benchmarking is planned but takes significant compute time per model; results will be added when available.

Distilled reasoning model. May hallucinate. Lossy relative to BF16. Quantization quality depends on model, quantization level, and calibration; no direct same-size accuracy comparison with uniform g32/g64 is claimed here.

Text-trained/fine-tuned; vision behavior is inherited from the Qwen3.5 base and was not evaluated for this release.

Native context: 262,144 tokens inherited from Qwen3.5; practical usable context depends on the inference runtime, KV-cache implementation, and available memory.

Credits

Base: empero-ai/Qwen3.8-4B-Distill (safetensors) — itself a full-parameter distillation of Qwen/Qwen3.8-2.4T-A95B into Qwen/Qwen3.5-4B, Apache 2.0. GGUF repo empero-ai/Qwen3.8-4B-Distill-GGUF is a separate quantized artifact for llama.cpp, not used as source for this MLX conversion. Released by Empero under Apache-2.0; upstream card states license inherited from base.
Quantization: oMLX Universal Dynamic Quantization (oQe). Built by RolanDorisTech, September 23, 2026.
Machine: M1-Max Mac Studio 64GB, 32 Core GPU, macOS 27.0
Tool: oMLX Quantization panel
Base conversion: Qwen3.8-4B-Distill BF16 MLX master 7.1GB (converted from empero-ai/Qwen3.8-4B-Distill safetensors, not GGUF)
Method: oQe ON, Reuse ON, Automatic cache, Strict OFF, Preserve MTP OFF, bfloat16 for norms and scales
Protection: lm_head 8-bit, embedding and early/late layers boosted per the oQe quantization configuration