Schiltmans/Qwen3.8-Flash-Next-Coder-cut240

Qwen3.8-Flash-Next Coder cut240 for MLX on 48 GB Macs

ISTA-DASLab's IQ1_M Coder GGUF of Qwen3.8-Flash-Next (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF at 5348543e) with 768 of its 12,288 routed experts removed, served through MLX as an OpenAI-compatible endpoint for coding agents. Each layer keeps 185 to 255 of 256 routed experts, 240 on average; kept tensors are byte for byte as published. This repository is the model folder and its runner in one; the runtime is a TensorFold fork that reads the GGUF tensors in place through mlx-kquant.

What the runner does

• Drafting is on by default: ggml-org's MTP head for Qwen3.8-Flash-Next (routed experts cut from 512 to 256 at load) proposes up to 3 tokens a round, a second proposer copies spans from the prompt, and the model checks each drafted token against its own sample. Decode throughput was 1.11 to 1.50 times that of drafting off, depending on the kind of request.
• When a client disconnects, the server stops generating for it.
• Agent session-title requests run in the background, uninterrupted once they have sent a token, so a title reply is never joined from two runs.

DRAFTING=off ./serve.sh turns drafting off and keeps the disconnect and title behaviour. Some evidence/ files come from an earlier build of the runner, drafting off and without that behaviour; each says which.

Use

Requires an Apple Silicon Mac with 48 GB and macOS 26 or later (measured on an M4 Pro), about 62 GB of free disk (57 GB model files, 2.2 GB head, about 1 GB environment) and uv or python3.11. Close memory-heavy apps first: serve.sh will not start if macOS reports memory pressure or has less than an estimated 36 GB free or held by cached files. This is a startup check; it does not guarantee that memory will suffice later.

uvx --from huggingface_hub hf download Schiltmans/Qwen3.8-Flash-Next-Coder-cut240 --local-dir Flash-Next-Coder-cut240   # ~57 GB
cd Flash-Next-Coder-cut240
./install.sh     # .venv with the pinned packages and TensorFold
./setup.sh       # downloads the MTP head at a pinned revision, checks every large file
./serve.sh       # http://127.0.0.1:8088/v1, model "local"

setup.sh stops on any checksum mismatch; opencode.example.jsonc holds the opencode settings used for the agentic results. Sampling defaults to temperature 1.0, top-p 0.95, top-k 20 and xhigh reasoning; request values take precedence.

KV_PAGES=on CONTEXT=65536 ./serve.sh serves a 65536-token window, with the prompt cache's keys and values in pages that entries sharing a prefix share; close other apps first (Results). serve.sh then asks for an estimated 38 GB of free or cached memory at startup. It refuses a window above 32768 without KV_PAGES=on, and any window above 65536.

Results

M4 Pro, 48 GB, browsers closed; details and every run are in evidence/.

Same output with and without drafts, on one server

Each of 15 cases (3 short prompts; 12 of 2K-26K tokens, two seeds, a second turn) went twice to one drafting server, drafted and one token at a time ("draft": false): the token ids matched in all 15. On the drafting server the 12 long cases decoded at 27.4-32.4 tok/s drafted and 21.9-23.5 tok/s one token at a time. A DRAFTING=off server differs: its one-token steps use other matrix kernels, which round differently. Of 24 seeded requests, none gave the same complete reply on both servers, though each server reproduced all 24 of its own in a second session. Both sample the same model. evidence/identity.md lists every case and checks a from-scratch install of this repository against the measured build.

Decode throughput by kind of request

Drafting on and off, in tokens per second (evidence/drafting-speed.md). The two configurations generate different sequences, so the table compares whole configurations, not the same tokens. Accepted is the share of drafted tokens kept.

kind	on	off	ratio	accepted
chat	34.0	28.6	1.19	0.55
prose	33.6	28.6	1.18	0.57
code	33.5	28.6	1.17	0.54
Dutch / German	32.0	28.7	1.11	0.47
math	43.5	28.9	1.50	0.73
random-looking output	37.6	28.8	1.31	0.61

Agentic coding

24 Exercism Python exercises a pass, opencode, 600 s each (evidence/agentic-drafting.md); swap did not grow in any pass.

drafting	passes	solved	decode tok/s
on	4	90 of 96 (21-24 a pass)	33.4-34.8
off	2	43 of 48	25.8-26.1

Solved per hour was higher with drafting on in these passes (30.2 against 22.9), but that compares 4 passes with 2, passes of one setting spread widely (25.9-36.9 an hour with drafting on), and they were not designed to measure it. An earlier balanced block of 6 + 6 passes in another configuration (the head with all 512 experts, before the other two changes) gave 1.08 times the solved per hour, 95% interval 0.98-1.21 (exercises resampled), not significant. No agentic speed-up is claimed.

Memory with the window full

Highest whole-machine wired memory sample (every 0.5 s) while a ~30.8K-token prompt fills the 32768-token window and four facts are recovered (evidence/window.md).

configuration	runs	highest sample per run, GB	median
drafting on	5	36.50, 36.53, 37.05, 37.27, 37.37	37.05
drafting off	5	35.05, 35.52, 35.62, 35.83, 36.29	35.62

Every run recovered all four facts with no swap-outs; in every run the highest sample came from a spike (Known issues). The 37.05 GB run, a from-scratch install of this repository, had one more desktop app open (about 2 GB, not wired). In the agentic passes the median was 32.5 GB on and 31.2 GB off. These figures were measured with large apps closed; they do not measure how much memory other apps can use.

64K window with KV pages

With KV_PAGES=on the prompt cache's checkpoints and entries share the pages that hold a common prefix; the request being served decodes from contiguous buffers, as with KV pages off (how it works: docs/kv-pages.md in the TensorFold fork). Three runs per setup, each a ~61K-token prompt in the 65536-token window with four facts to recover (evidence/window-64k.md).

apps open	runs	facts recovered	highest sample per run, GB	median	decode tok/s
one editor	3	4/4 each	37.20, 37.24, 37.21	37.21	24.2-26.8
an editor and a chat app	3	4/4 each	37.63, 37.49, 37.47	37.49	22.9-26.4

Whole-machine wired memory, sampled every 0.5 s, macOS's own ~2.4 GB included; the runs' memory guard stops at 38.65 GB (1.4 GB above the median with one editor open, 1.2 GB with the chat app); swap-outs 0 in every run. This fits without margin: close other apps before a 64K session. The first ~61K-token prompt, read without a cache, takes about 195 s. In this mode MLX keeps 0.5 GiB of freed buffers for reuse (2 GiB otherwise). One conversation in the prompt cache.

check	runs	same replies
16K-token prompts, KV pages on against off, greedy	1 pair	16 of 16
the same with a 64-page pool that fills	1 pair	13 of 16; the 3 that differ resumed from another cached entry after evictions

With KV_PAGES unset, the server gives the same replies as the published build (identity, drafting and 32K-window checks in evidence/window-64k.md).

What the cut changes

Measured on the earlier build of the runner, drafting off: cut240 against the unpruned GGUF on the same runtime (evidence/).

	cut240	unpruned
highest wired sample with the window full (one run each)	36.38 GB	38.03 GB
highest wired sample during agentic passes (4 each)	35.4 GB	36.8 GB
decode at 30K tokens	25.9 tok/s	25.8 tok/s
exercises solved over 4 passes	90 of 96	89 of 96

llama.cpp, serving the unpruned GGUF with its n-gram drafting, solved 94 of 96 of the same exercises (evidence/agentic.md).

Known issues

• In the 32K-window runs, wired memory spikes 3.4 to 4.7 GB above its median level within a few seconds of a request ending, around the time the next one starts. The cause is not known. These spikes set the window table's highest samples.
• A turn arriving while the server writes a session title waits for it (36 titles in the six agentic passes: median wait 8.8 s, once 50.5 s behind a 1,635-token title, 2.1% of pass time in all). A title request without max_tokens gets the server's limit of 16,384 tokens (evidence/background.md).
• About 5 of 24 opencode sessions a pass ended without a title; the cause is not known.
• A client resetting a kept-alive connection leaves a ConnectionResetError traceback in the server log; serving continues.
• Above the 32768-token window, only the 65536-token window with KV pages was measured; it fits without margin (Results).
• A KV page pool smaller than one prompt (TF_KV_POOL_PAGES) empties the prompt cache, so every request starts from an empty cache. The default pool holds one window of pages plus 32.

Files

file	what
experts-cut240.gguf	the routed experts of ISTA-DASLab's shard 1 without 768 removed ones; kept bytes unchanged
trunk.gguf	every other tensor of ISTA-DASLab's shard 1, unchanged
ngram.gguf	ISTA-DASLab's shard 2, the per-layer n-gram embedding tables, unchanged
cut_experts.json and the other small files	the kept expert ids per layer, Qwen's tokenizer and chat template, and the model config that names the files above
mtp/keep256.json	the 256 MTP-head experts serve.sh keeps (how they were chosen: NOTICE)
install.sh, setup.sh, serve.sh	install the runtime, fetch the head and check checksums, start the server
SHA256SUMS	checksums of the large files and the head
evidence/	the measurements behind this card

Tensor elements: 56.6B in experts-cut240.gguf, 4.9B in trunk.gguf and 51.2B in ngram.gguf (n-gram embedding tables). The Hub's "Model size" counts only experts-cut240.gguf.

Licences

• Weights: Qwen Community License 1.0 (LICENSE), as a derivative of Qwen3.8-Flash-Next.
• ISTA-DASLab's source GGUF: Apache-2.0 (LICENSE-APACHE-2.0); NOTICE lists the changes made here.
• Scripts: MIT (LICENSE-MIT). setup.sh fetches the MTP head from ggml-org (NOTICE).