Luni/Qwen3.8-Flash-Coder-MLX-8bit

Qwen3.8-Flash-Coder - MLX 8-bit

MLX 8-bit affine quantization of Jab1718/qwen3.8-flash-coder-85gb-bf16.

Model Details

• Base Model: Jab1718/qwen3.8-flash-coder-85gb-bf16
• Architecture: qwen3_next (48 hybrid layers with GDN linear attention + QSA sparse attention).
• MoE Layout: 160 routed experts per layer (pruned from 512 via moe-slice), 10 active experts per token.
• Quantization: Affine 8-bit (group size 64) with FP16 scales and biases.
• Router Handling: Router gates (mlp.gate, mlp.shared_expert_gate) kept at 8-bit.
• Disk Footprint: ~43.0 GB across 10 shards.

Memory Requirements

Weights take ~43.0 GB in memory. Intended for Apple Silicon configurations with 64 GB, 96 GB, or 128 GB unified memory. It will swap on 32 GB or 36 GB devices.

Usage with MLX-LM

Requires a current mlx-lm installation with support for qwen3_next:

pip install --upgrade mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("Luni/Qwen3.8-Flash-Coder-MLX-8bit")

response = generate(
    model,
    tokenizer,
    prompt="Implement a custom memory allocator in C using mmap and a free list.",
    max_tokens=512,
    verbose=True,
)
print(response)