NovaeonStudio/Qwen3.8-Flash-Next-GSQ-RCO-Q2-mlx

Qwen3.8-Flash-Next · GSQ-RCO Q2 · MLX

ISTA-DASLab's GSQ-RCO 2-bit experts, carried losslessly from GGUF into MLX. Runs on a single Apple-silicon Mac with ~41 GB of RAM: the 51B n-gram table stays on the SSD.

---

TL;DR

Base model	Qwen/Qwen3.8-Flash-Next: 125B MoE (6B active, 512 routed experts, 10 per token), plus a 51B n-gram embedding table (PLE) and a 4B MTP head
Routed experts (95 % of the matmul weights)	GSQ-RCO Q2_0 from ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, repacked bit-for-bit into MLX affine 2-bit, group size 64
Hyper-connection projections	re-quantized from the original BF16 to MLX 8-bit, group size 64 (enables oMLX's fused hyper-connection kernels)
MTP head	the original mtp.* head (Lightning MTP speculative decoding): experts 4-bit/64, other linears 8-bit/64
Everything else	from mlx-community/Qwen3.8-Flash-Next-4bit: attention, linear attention, shared experts, router, vision tower, 4-bit/32 n-gram table
Download	75.6 GB (experts 37.8, n-gram table 32.0, rest 4.3, MTP 1.5)
Resident memory	~39–41 GB with the n-gram table on SSD (mmap), ~71 GB fully resident
Tested runtime	oMLX 0.7.0 (bundled mlx 0.32.2, mlx-vlm 0.7.1) on a MacBook Pro M5 Max with 128 GB

Why this exists

ISTA-DASLab's GSQ-RCO GGUFs are among the best low-bit builds of Flash-Next. GSQ learns the per-group scales and grid assignments with a Gumbel-Softmax relaxation, and RCO picks a quantization type per tensor under a size budget. According to their card, the Q2_0 build keeps 96.67 on AIME25 and 89.39 on GPQA-Diamond at 2.40 bpw, with 3.4× the prompt throughput of the lookup-table IQ builds.

MLX cannot run GGUF lookup-table formats (IQ2_XS, IQ3_XXS, IQ3_S): converting those means dequantizing and re-quantizing, which throws away exactly what GSQ optimized. Q2_0 is the exception. In this build all 144 routed-expert tensors are Q2_0, and Q2_0 has the same structure as MLX's native affine 2-bit format:

llama.cpp Q2_0 (64 weights/block):  w = (q - 1) * d          q ∈ {0..3}, d = fp16 scale, 4 values/byte LSB-first
MLX affine 2-bit, group 64:         w = scale * q + bias     16 values/uint32 LSB-first (little-endian)
=> scale = d, bias = -d, packed bits copied verbatim

The quantized expert codes are therefore identical to ISTA's GGUF. The only change is that the fp16 block scale is stored as bf16 (MLX activation dtype), a relative scale difference of at most ~0.2 %. We verified this per tensor against llama.cpp's reference dequantize_row_q2_0.

What's inside

experts-00..47.safetensors   GSQ-RCO Q2_0 routed experts (gate/up/down), lossless repack, 2-bit g64
base-*.safetensors           attention, GDN linear attention, shared experts, router, vision tower,
                             hyper-connections (8-bit g64 from original BF16)
ple-*.safetensors            51.2B n-gram (PLE) table, 4-bit g32 (from mlx-community, quantized once from BF16)
mtp-00.safetensors           original MTP head for Lightning MTP (experts 4-bit g64, linears 8-bit g64)
conversion/                  the scripts used to build this checkpoint (reproducible)

Per-module bits are listed in config.json → quantization (2/64 experts, 8/64 hyper-connections, router and gates, 4/32 elsewhere).

Usage (oMLX, recommended and tested)

1. Download into your oMLX model directory, e.g. ~/.omlx/models/NovaeonStudio/Qwen3.8-Flash-Next-GSQ-RCO-Q2-mlx, and reload models in the admin UI.
2. Per-model settings (admin UI or PUT /admin/api/models/<id>/settings):

{
  "qwen4_ple_ssd_offload": true,
  "mtp_enabled": true,
  "turboquant_kv_enabled": true,
  "turboquant_kv_bits": 8.0
}

• qwen4_ple_ssd_offload: true keeps the 32 GB n-gram table memory-mapped on the SSD (needs a fast internal SSD). Set it to false if you have the RAM; that gives roughly +20 % decode speed.
• mtp_enabled: true with the default adaptive depth was the fastest setting we measured. Fixed depth 2 and max depth 4 were slower or equal.
• oMLX's fused hyper-connection kernels activate automatically. Look for Enabled Qwen4-Exp hyper-connection optimizations: 96 exact hybrid projection pairs, 97 compiled decode paths in the log (with MTP enabled, the hybrid pairs are skipped by design).

Then use any OpenAI-compatible client:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="<your omlx key>")
r = client.chat.completions.create(
    model="Qwen3.8-Flash-Next-GSQ-RCO-Q2-mlx",
    messages=[{"role": "user", "content": "Explain sparse attention in one paragraph."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(r.choices[0].message.content)

Other runtimes: plain mlx-vlm ≥ 0.7.1 has a qwen4_exp implementation. We only validated an earlier stage of this build with it (experts plus external PLE, without MTP or the 8-bit hyper-connections), so this exact checkpoint is tested with oMLX only.

Performance

MacBook Pro M5 Max, 128 GB, oMLX 0.7.0. Thinking off, temperature 0.7. Decode is single stream on a ~50-token prompt with 600 output tokens.

configuration	resident RAM	decode tok/s	prefill (~7k tok) tok/s	4 parallel streams, aggregate tok/s
first working build (no MTP, 4-bit/32 hyper-connections)	39 GB	39	1,775	94
+ fused hyper-connection kernels (8-bit/64)	39 GB	44–48	1,390–1,650	88
+ Lightning MTP (adaptive)	39 GB	63–66	1,810–1,860	~100
n-gram table on SSD, nothing else loaded	41 GB	57–67	1,770–2,310	89–93
n-gram table resident, nothing else loaded	71 GB	69–80	~2,000	117–120

Long context (first build, n-gram table on SSD, no MTP): a 212k-token prompt was prefilled in 134 s (~1,590 tok/s); prefill throughput stays nearly flat from 9k to 212k tokens. MTP acceptance in typical prose is 64–88 % at ~2.1–2.7 tokens per verify cycle.

Quality

Upstream (ISTA-DASLab, GGUF Q2_0, measured with llama.cpp)

	AIME25	GPQA-Diamond	LiveCodeBench v6	5-task zero-shot avg
BF16	100.00	91.92	87.43	76.94
GSQ-RCO Q2_0	96.67	89.39	81.14	78.00

The routed experts in this checkpoint are bit-identical to that build. The non-expert tensors are different (MLX 4-bit/8-bit instead of ISTA's per-tensor GGUF mix), so these numbers describe the expert quantization, not a re-measurement of this checkpoint.

Our own agent evals (this checkpoint, oMLX)

These are internal regression suites we use to choose a local "seat" model for an agent fleet: tool calling, JSON, instruction following, German, long context, and multi-step tasks in a sandbox. Reference: our production seat, a Qwen3.8-35B-A3B distill at 8-bit.

suite	this model	35B-A3B seat
golden (43 cases, thinking off)	36/43	39/43
mitigations (17 cases: enforced JSON, thinking-on arithmetic, tool-assisted math, swarm contracts)	14/17 (thinking-on reasoning 5/5)	15/17
agentic (13 multi-step sandbox tasks), thinking off / on	12/13 · 13/13	13/13 · 12/13

Honest reading:

• With thinking on, it is a strong agent: 13/13 agentic tasks and flawless thinking-on arithmetic.
• With thinking off, it answers too fast: single-shot arithmetic comes back as a bare number without working it out (1/6). That is expected for a model that, per Qwen, is designed to think by default.
• It is eager with tools: it sometimes makes one more call than needed (a news search after already answering a price question correctly; continuing to explore instead of returning a required structured result). Give it a clear "finish" instruction.
• We did not run a BF16 or plain-4-bit A/B on our suites, so we cannot attribute these gaps to the 2-bit experts versus the model's own behaviour.

How it was built (reproducible)

All scripts are in conversion/:

1. ggufq2.py: a minimal GGUF v3 reader plus the lossless Q2_0 → MLX 2-bit g64 repack, and a reference port of llama.cpp's dequantizer for verification.
2. convert.py: takes the mlx-community 4-bit checkpoint as skeleton, drops its routed experts, and writes 48 expert files repacked from the GSQ-RCO GGUF.
3. fetch_orig.py: pulls only the needed BF16 tensors (398 hyper-connection tensors, 31 MTP tensors; ~6.5 GB) from the original repo with HTTP byte ranges instead of the full 354 GB.
4. optimize.py: re-quantizes the hyper-connection projections to 8-bit/64 and adds the quantized MTP head.

Checks we ran:

• The repack reproduces llama.cpp's dequantized Q2_0 weights up to the fp16→bf16 scale rounding.
• Cosine similarity between each GSQ expert and the corresponding mlx-community 4-bit expert is 0.85–0.91 (as expected for 2-bit vs 4-bit). A deliberately mismatched pair (gate vs up) gives ≈ 0.0, which confirms tensor naming and orientation.
• All 32 MTP parameters map 1:1 onto the oMLX runtime's module tree (none missing, none extra).
• Generation is coherent in English and German, and on math traps (e.g. 9.9 vs 9.11).

Limitations

• Tested with oMLX 0.7.0 only. Other MLX runtimes need qwen4_exp support, per-module quantization configs and, for MTP, a Lightning-MTP implementation.
• The vision tower (from mlx-community) is included but was not evaluated.
• The ple_* n-gram table is quantized 4-bit/32 from BF16 (mlx-community), not GSQ's IQ4_NL.
• SSD offload needs a fast internal NVMe. External or network storage will hurt decode.
• Speed numbers are from one M5 Max machine running other services; treat them as indicative.

License & attribution

• Model weights derive from Qwen3.8-Flash-Next, © 2026 Qwen, under the Qwen Community License 1.0 (see LICENSE). Read the conditions carefully: products above 100M MAU or US$20M monthly revenue must display the model name, and "Model as a Service" and "AI Work Assistant" businesses need a separate license from Qwen for commercial use.
• Routed-expert quantization: GSQ (arXiv:2604.18556, code) and RCO (arXiv:2605.00649, code) by the Deep Algorithms and Systems Lab, IST Austria, via ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (Apache-2.0 release).
• MLX skeleton: mlx-community/Qwen3.8-Flash-Next-4bit (converted with mlx-vlm).
• Runtime: oMLX and mlx-vlm.
• Conversion, MTP and hyper-connection work, and evaluation: Novaeon Studio.

If you use the quantization, please cite GSQ and RCO.