WaveCut/Qwen3.8-Flash-Next-REAM-288-MLX-4bit

Qwen3.8-Flash-Next-REAM-288 (MLX-4bit)

Drop-in alternative to sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit: same 288 slots, same file layout, same runtime — but absorbed experts are merged in, not dropped. Experts 4-bit gs64 affine (backbone 4-bit, n-gram table 4-bit gs32). Loads on stock mlx-vlm (git main with qwen4_exp); every non-expert tensor is byte-identical to the sh0wie build (his RMSNorm/shards.N fixes included), only the 48x3 switch_mlp expert tensors differ. No mtp.* tensors — use the author's standalone MTP drafter for speculative decoding (compatible with any expert count).

pip install git+https://github.com/Blaizzy/mlx-vlm.git
python -m mlx_vlm.generate --model WaveCut/Qwen3.8-Flash-Next-REAM-288-MLX-4bit \
  --prompt "Refactor this function to add input validation." --max-tokens 512

Method

REAM (arXiv 2604.04356, Samsung SAIL Montréal: Merging Improves Pruning of Experts in LLMs), offline adaptation: instead of dropping the 224 non-kept experts per layer (REAP), each is merged into its most-similar kept expert with saliency-proportional coefficients.

• Kept set / slot order: exactly the per-layer REAP-288 manifest published by sh0wie (saliency calibrated on ~686K tokens of agentic-coding traffic) — full credit to that work; this build is a direct A/B against it.
• Similarity (offline, no calibration run): cosine of router gate rows.
• Merge coefficients: measured saliency from sh0wie's saliency_full_505.npz (its argsort reproduces his pinned manifest bit-for-bit), tempered and shrunk: w_i = (S_i/S_c)^2 * router_cos * count_confidence, with the centroid keeping at least 90% of itself. Raw proportional weights are catastrophic here (KL 1.397) because the real kept/dropped saliency gap is only ~1.6x — the merge dissolves the expert REAP chose to keep. Hungarian permutation alignment of intermediate channels before averaging (fp32 accumulate, bf16 out).
• Trajectory flat_tsg_tsD: flat one-shot groups (canonical REAM)
• Router: absorbed rows deleted; kept rows byte-identical to stock.
• MTP block: same machinery, k-center centroids on its own gate rows.
• A six-point ladder (see campaign/reports/truesal_ladder.md) locates the optimum: KL is minimised when kept experts give away ~9% of their mass — both heavier blending and near-pruning are worse.

Evaluation (A/B vs REAP prune, same harness, same hardware)

model	KL vs stock (mean)	rare-name probe (10 names)	HumanEval subset (25 tasks)
stock 512e	0	0.97	90.0% (full 50)
REAP-288 recon (prune)	1.3066	0.91	92.0% (stock on same 25: 92.0%)
REAM-288 flat_tsg_tsD (this)	1.2564	0.94	not re-run for tsD; flat_sim (previous main, same pipeline) scored 88.0% vs stock 88.0% on its slice

KL: teacher-forced top-256 KL vs the stock model over a fixed mixed corpus (en/ru/code/rare-entities/agentic, ~10K positions). Probe: rare entities x 10 seeded sampled generations, pass = name reproduced intact; aggregates recomputed on the name subset common to all rows (a monitoring bug killed the winner's third probe batch mid-run). HumanEval: 25-task slices of a frozen 50-task subset, no-thinking, greedy; the two 288e models landed on different slices (2-task overlap), so each is paired with the stock score on its own slice. Single-run evals; ±1-2 point differences are noise. RU slice of the probe (3 Cyrillic names, 30 gens): stock 27, prune 21, merge 24.

4-bit MLX head-to-head (identical harness)

These are this repo and the author's, measured against each other.

Both published 4-bit builds were run through the same script (eval_mlx.py / probe_mlx.py on mlx-vlm), on the same GPU, over the same problems with the same prompts, chat template, decoding and checkers — only the checkpoint differs. HumanEval is the full 164; GSM8K is 200 problems drawn evenly over the whole test split.

4-bit MLX build	HumanEval-164	GSM8K-200	rare-name probe
REAP-288 (author's prune)	89.0%	56.0%	60.0%
REAM-288 (this merge)	89.6%	58.0%	62.7%

How to read this:

• Neither difference is significant. On code, paired McNemar gives p = 1.000 (4 problems solved only by the merge, 3 only by the prune, 14 defeat both). On maths, p = 0.585 (17 vs 13). At these sample sizes the merge and the prune are the same model on both benchmarks.
• The earlier 25-problem numbers were noise, and are retracted here. Two non-overlapping 25-problem GSM8K draws put one build at 72% and 40%; the pooled 29/50-vs-24/50 edge (p = 0.18) that an earlier version of this card reported did not survive 200 problems.
• What does survive is KL: 1.2564 for the merge against 1.3066 for the prune, measured against the stock 512-expert model. That is the one metric where the merge is consistently ahead.
• GSM8K numbers are capped by the token budget. At max_new=320 many completions are cut off before the answer line; of six problems both builds failed at that cap, five pass at 768. The cap applies equally to every row, so it biases the absolutes, not the comparison.
• 4-bit hurts non-Latin names badly, and the merge is not to blame. All three Cyrillic names score 0/5 on both builds, the author's untouched one included. The same merge in bf16 scores 93.3% against 62.7% in 4-bit over the same 15 names — a 30-point gap that belongs entirely to the quantizer. (The bf16 selection table above reports 94.0% for this build over the 10-name subset the selection gate used; both numbers are real, they are just different name sets.) For factual recall on rare non-Latin entities, use bf16.

Known limitations of the MLX path

• Factual recall degrades in mlx-vlm for this architecture regardless of the checkpoint: on identical hardware and prompts the author's own REAP-288 MLX build also misdates events and mangles rare names, so the n-gram PLE path is the suspect, not the merge. Use the bf16 transformers checkpoint when factual accuracy matters.

Build note: shard byte layout must be preserved

ple-store.json (layout: "safetensors_ranges") addresses the 320M-row n-gram table by absolute byte offsets into named shard files, bypassing the safetensors header. One file — model-00005-of-00131.safetensors — holds both the first n-gram chunk and the layer-1 experts, so rewriting it with any writer (ours or mx.save_safetensors) shifts the blob, the ple-store offsets land mid-tensor, and mlx-vlm emits NaN logits — even when every tensor is bit-identical to the author's.

Shards are therefore patched in place: the author's file is copied verbatim and only the expert payloads are overwritten at their original offsets. If you rebuild from these weights, keep the layout, or regenerate ple-store.json to match your new offsets.

Credits

sh0wie / HamsterResearch (REAP-288 manifests, conversion, fixes), Samsung SAIL Montréal (REAM), Qwen team (base model, license inherited).