True2456/Qwen3.8-Flash-Next-MLX-4bit-ablation

Qwen3.8-Flash-Next MLX 4-bit (with refusal directions)

This repository is identical to True2456/Qwen3.8-Flash-Next-MLX-4bit, with the same weights byte for byte, plus one extra file: refusal_directions.safetensors. It holds one direction per decoder layer, which Rindi-Agent's Refusal ablation setting projects out of the residual stream at inference time.

• The weights are not modified. At 0, the default, the model behaves exactly like the stock one, and the setting can be changed or turned off at any time.
• Where to set it: Rindi-Agent › Settings › General › Refusal ablation. 0 is the stock model, around 1–2 loosens over-eager refusals, and higher values remove refusals broadly at a cost in coherence. It is meant for security and red-team lab work.
• Other runtimes ignore the file.

If you don't need the setting, use the original repository.

Affine MLX quantization of Qwen/Qwen3.8-Flash-Next (qwen4_exp). This is the 4-bit bank the Rindi Flash-Next serve path uses for resident MoE experts, the lm_head, and the MTP drafter on Apple Silicon.

It is not a drop-in for stock mlx-lm until that package has qwen4_exp. It is not the ANE MIL graphs (those are M5/h17 compiled programs and will ship separately).

Files in this repo

path	role	size
model.safetensors	language weights (experts 4-bit gs64; GDN/attn/embed mostly 8-bit)	~71 GB
mtp/mtp-4bit.safetensors	MTP drafter	~1.4 GB
model-indexer.safetensors	QSA indexer	~41 MB
vision/	vision tower (unused by the text ANE server)	~856 MB
tokenizer + chat_template.jinja	Qwen chat template (low / medium / xhigh)	—
ngram/	PLE n-gram embedding table (unmodified official bf16 shards 00006–00036 plus ngram-extra)	~95 GB
ngram_index.json	PLE hash constants	—
refusal_directions.safetensors	per-layer refusal directions for Rindi-Agent's Refusal ablation setting ([layers, hidden])	~0.5 MB

The n-gram table is the official Qwen/Qwen3.8-Flash-Next PLE embeddings, not quantized. Without those files the runtime can still load with a zeros PLE fallback.

Recipe

Default 4-bit group size 64. Per-family overrides in config.json:

• experts gate_up / down: 4-bit
• GDN in/out proj, attention, embed, MTP: 8-bit
• router and gated-residual (hyper): 16-bit

Converted on an M5 Max from the official BF16 checkpoint. Do not mix this file with conversions that folded Qwen4ExpRMSNorm's 1+w into the saved gains (those generate noise).

Requirements

• Apple Silicon. The ANE hybrid path that consumes this bank is currently M5 (h17), ~128 GB unified memory.
• A qwen4_exp MLX runtime (local mlx-lm / mlx-vlm checkout, not an old PyPI wheel).
• Qwen Community License 1.0 (see LICENSE). Redistribution of this derivative is allowed; commercial MaaS / coding-assistant products have extra terms in that license.

Acknowledgements

Weights, architecture, and tokenizer: Qwen. Quantization and Apple Silicon packaging: this repository.