ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit

Qwen3.8-Flash-Next Uncensored — MLX-Serve 4-bit: How to Run Locally

The uncensored Qwen3.8-Flash-Next (125B total / 6B active MoE) packed for mlx-serve, so it actually fits and flies on a 128 GB Apple Silicon Mac:

• ~68 GB resident — the 51B-parameter n-gram table lives in ngram_table.bin and is memory-mapped from disk, not held in RAM
• ~56 tok/s decode serial, ~60 tok/s with MTP measured on an M5 Max 128 GB (custom Metal kernels for the MoE gather, Gated-DeltaNet decode, hyper-connections, and sparse attention)
• Vision tower and MTP speculative head included
• Full 262,144-token context
• Base weights are orcarouter's abliteration of Qwen/Qwen3.8-Flash-Next — refusals removed, everything else intact

The other ways to run this model uncensored on a Mac either don't fit 128 GB (the original MLX repo is 163 GiB) or run slower (GGUF IQ4_XS ≈ 26 tok/s on the same hardware; generic Python MLX paths ≈ 27–36 tok/s).

About the base model

This is a 4-bit MLX-Serve pack of orcarouter/Qwen3.8-Flash-Next-Uncensored — the full-precision BF16 abliterated (refusal-removed) build of Qwen's Qwen3.8-Flash-Next. See orcarouter's card for the full abliteration methodology and evaluation; this section summarizes it.

Base model	Qwen/Qwen3.8-Flash-Next
Architecture	qwen4_exp (Qwen4 preview) — 48 layers, hidden 2560, hybrid attention (36 Gated-DeltaNet linear + 12 full-attention), 512 fused experts, top-10 + shared expert, 51B-param PLE n-gram embedding, Hyper-Connections residual, native vision + video tower, MTP speculative head
Total / active	125B / ~6B per token
Uncensoring	Abliteration (refusal-direction removal, Arditi et al. 2024) on the BF16 weights by orcarouter — a surgical weight edit that collapses refusal behaviour while leaving general capability ~unchanged. Refusal rates drop from ~94–100% to ~0–2% on AdvBench / JailbreakBench / StrongREJECT (orcarouter's measurement).
This repo	Quantization + MLX-Serve format conversion of those weights — no further modification

⚙️ Requirements

Hardware	Apple Silicon Mac, 96 GB unified memory minimum (128 GB recommended for long agent contexts)
Disk	~107 GB for the pack, on the internal SSD (the n-gram table is gathered from disk on the hot path — keep it off slow external drives)
Engine	mlx-serve v26.8.11-pre-release.1 or newer (first build with qwen4_exp support)
No sysctl needed	At 68 GB resident this loads under the default Metal wired limit — no iogpu.wired_limit_mb fiddling

📦 What's in the pack

Built with mlx-serve's own converter (tests/convert_qwen38_flash_next.py) from the bf16 uncensored weights:

Component	Precision
Routed experts (the 512-expert bulk)	4-bit, group 64
n-gram PLE table (51B params, 160-wide rows)	4-bit, group 32 → ngram_table.bin, mmapped
All other 2-D projections (attention, shared expert, GDN)	8-bit, group 64
Routers, gates, GDN in_proj_a/b, all 1-D tensors	bf16
MTP speculative head	included (--mtp to use)
Vision tower	bf16 pass-through

The architecture's x·(1+w) RMSNorm offset is folded at convert time.

🚀 Run it

1. Get mlx-serve (binary release, or brew install ddalcu/mlx-serve/mlx-serve):

curl -LO https://github.com/ddalcu/mlx-serve/releases/download/v26.8.11-pre-release.1/mlx-serve-bin-macos-arm64.tar.gz
tar xzf mlx-serve-bin-macos-arm64.tar.gz

2. Download this pack (installing hf_transfer first makes this run at full line rate):

pip install -U huggingface_hub hf_transfer
HF_HUB_ENABLE_HF_TRANSFER=1 hf download ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit \
  --local-dir ~/.mlx-serve/models/ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit

3. Chat right in the terminal:

./mlx-serve --model ~/.mlx-serve/models/ARC4NUM/Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit \
  --prompt "Hello!" --max-tokens 512 --temp 0.7

4. Or serve it — OpenAI, Anthropic, and Ollama APIs on one port:

./mlx-serve serve --host 127.0.0.1 --port 11234
# OpenAI:    http://127.0.0.1:11234/v1/chat/completions
# Anthropic: http://127.0.0.1:11234/v1/messages
# Ollama:    http://127.0.0.1:11234/api/chat

Works out of the box with Claude Code, pi, OpenCode, Codex, aider, Open WebUI, Raycast — anything that speaks one of those wires. mlx-serve launch <agent> preconfigures and starts your coding agent against it.

MTP speculative decoding (the model's own draft head) is opt-in: pass --mtp at launch or "enable_mtp": true per request. Fastest on code and echo-heavy agent loops; a touch slower on freeform prose.

🎛️ Recommended Settings

Qwen's official sampling settings for this model:

Parameter	Thinking mode	Instruct (non-thinking)
temperature	1.0	0.7
top_p	0.95	0.80
top_k	20	20
min_p	0.0	0.0
presence_penalty	0.0	1.5

Thinking effort defaults to xhigh and this model will happily ruminate; for agent work, --reasoning-budget 2048 (or the reasoning_effort request field: low / medium / xhigh) keeps it moving. The model supports preserve thinking — keeping prior turns' traces in context — which costs tokens but can help multi-turn accuracy.

📊 Measured performance

M5 Max, 128 GB, this pack, --temp 0 300-token generations:

Path	Decode	Resident memory
mlx-serve serial	55.7 tok/s	~68 GB
mlx-serve --mtp	60.3 tok/s	~68 GB
mlx-vlm (Python, same weights re-packed)	30.5 tok/s	~104 GB
llama.cpp GGUF IQ4_XS (community, same hardware class)	~26 tok/s	~94 GB

The Python paths are kernel-bound, not bandwidth-bound — the n-gram gathers, hyper-connections, and GDN decode run as generic graphs there. mlx-serve's fused kernels are the whole difference. The author reports ~60 serial / 78 MTP on an M4 Max for the base-model pack; MTP wiring is still marked partial in this pre-release, so expect free speed from future engine updates.

Context memory: only 12 of 48 layers are full attention (2 KV heads × 256 head-dim), so KV cache costs ~24 KB/token — ~2.4 GB at 100k context. Long agent sessions fit comfortably inside 128 GB.

🧪 Reproduce this pack

python3 tests/convert_qwen38_flash_next.py \
  --src /path/to/Qwen3.8-Flash-Next-Uncensored-bf16 \
  --dst ./Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit
python3 tests/convert_qwen38_flash_next.py --add-vision \
  --src /path/to/Qwen3.8-Flash-Next-Uncensored-bf16 \
  --dst ./Qwen3.8-Flash-Next-Uncensored-MLX-Serve-4bit

(~10 minutes total; streams shard-by-shard, never needs the full 335 GB in RAM.)

⚠️ Disclaimer — read before use

Reproduced from the source model card and applying equally to this quantized redistribution, which performs format conversion only.

This model has had its safety alignment substantially removed via abliteration (orthogonalizing the refusal direction out of the residual stream). As a direct consequence:

• It will comply with harmful, unethical, offensive, or illegal requests that the original Qwen3.8-Flash-Next would refuse. It has no meaningful built-in guardrails.
• It is released strictly for legitimate research — interpretability, AI-safety and refusal-mechanism study, red-teaming, robustness evaluation, and controlled experiments.
• You assume full responsibility and liability for how you use it and for everything it generates. Do not deploy it to end users or in production without adding your own safety, moderation, and abuse-prevention layers.
• Use must comply with the Apache 2.0 License inherited from the base model, and all laws and regulations that apply to you.
• The authors and uploaders accept no liability for any misuse or harm. Its outputs do not reflect the views of the uploaders or of Qwen / Alibaba.

By downloading or using this model you acknowledge and accept the above.

Attribution & lineage

This repository changes precision and file format only — no weights were trained, merged, or otherwise modified beyond quantization. The chain:

Stage	Who	What
Base model	Qwen	Qwen3.8-Flash-Next (125B-A6B), all capabilities
Abliteration	orcarouter	Refusal-direction removal ("uncensored")
Engine, kernels & converter	ddalcu / mlx-serve	The inference engine and the pack conversion tooling used here
This repo	ARC4NUM	4-bit MLX-Serve pack of the abliterated weights + this guide

License: Qwen Community 1.0, inherited unchanged from the base model and applying to this derivative. Review it before commercial use.