Luni/Qwen3.8-Flash-Coder-MLX-4bit

Qwen3.8-Flash-Coder - MLX 4-bit

MLX 4-bit affine quantization of Jab1718/qwen3.8-flash-coder-85gb-bf16.

Model Details

• Base Model: Jab1718/qwen3.8-flash-coder-85gb-bf16
• Architecture: qwen3_next (48 hybrid layers with GDN linear attention + QSA sparse attention).
• MoE Layout: 160 routed experts per layer (pruned from 512 via moe-slice), 10 active experts per token.
• Quantization: Affine 4-bit (group size 64) with FP16 scales and biases.
• Router Handling: Router gates (mlp.gate, mlp.shared_expert_gate) were kept at 8-bit to avoid routing instability.
• Disk Footprint: ~22.5 GB across 6 shards.

Memory Requirements

Weights take ~22.5 GB in memory. Actual RAM usage depends on prompt length and KV cache:

• 32 GB or 36 GB Macs: feasible for moderate context, but monitor unified memory usage and keep background applications low.
• 48 GB+ Macs: leaves sufficient headroom for standard context windows.

Usage with MLX-LM

Requires a current mlx-lm installation with support for qwen3_next:

pip install --upgrade mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("Luni/Qwen3.8-Flash-Coder-MLX-4bit")

response = generate(
    model,
    tokenizer,
    prompt="Write a concurrent job queue in Go using channels and worker pools.",
    max_tokens=512,
    verbose=True,
)
print(response)