OpensourceWTF/Qwen3.8-Flash-Next-exl3

Qwen3.8-Flash-Next, EXL3 packs for mlx-serve

Each branch of this repo holds one of turboderp's EXL3 quantizations of Qwen3.8-Flash-Next, repacked into the file layout that mlx-serve loads on Apple silicon. turboderp made them with exllamav3 and publishes them in turboderp/Qwen3.8-Flash-Next-exl3. The quantizations are his and nothing is quantized again, so please credit turboderp when you use them. mlx-serve serves them with the turboexl3_qwen38 model plugin, or with full-EXL3 support for Qwen3.8 and the turboexl3 quant plugin, the route the packs were first measured on. The experts load into memory, while the n-gram table, 26 to 39 GB depending on the branch, stays on SSD and is read per token, so it costs disk and file cache rather than memory. Each bitrate is a branch, and main holds only this card and the license.

Branches

Branch	bpw	Branch size	n-gram table, not in the branch	KLD / PPL (turboderp's)	Source and credit	Status
5.05bpw	5.05	84.38 GB	39.04 GB	0.0040 / 1.3848	turboderp's 5.05bpw_h6_ng6	live
4.05bpw	4.05	68.59 GB	39.04 GB	0.0066 / 1.3897	turboderp's 4.05bpw_h6_ng6	live
3.05bpw	3.05	52.72 GB	32.64 GB	0.0174 / 1.4074	turboderp's 3.05bpw_h5_ng5	live
2.05bpw	2.05	36.85 GB	26.24 GB	0.0681 / 1.4871	turboderp's 2.05bpw_h4_ng4	live

Sizes are in GB, where 1 GB is 10^9 bytes. The branches do not include the n-gram table, and The n-gram table says where each one comes from. KLD and PPL are turboderp's, measured with exllamav3 on his files, as Quality explains. Each branch's card says what is in its pack, where each file comes from and how to run it with mlx-serve.

Related repositories

Repo	What it holds
turboderp/Qwen3.8-Flash-Next-exl3	turboderp's EXL3 quantizations, a branch per bitrate. It is the source of every branch here, and of the n-gram tables.
OpenSourceWTF/qwen38-exl3-mlx-serve-converter	The conversion scripts that repack his files into mlx-serve's layout, including the n-gram table step for 2.05 and 3.05.
OpensourceWTF/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve	The earlier repo of 5.05bpw, now a stub that points here.
OpensourceWTF/Qwen3.8-Flash-Next-EXL3-4.05-mlx-serve	The earlier repo of 4.05bpw, now a stub that points here.
OpensourceWTF/Qwen3.8-Flash-Next-EXL3-3.05-mlx-serve	The earlier repo of 3.05bpw, now a stub that points here.
OpensourceWTF/Qwen3.8-Flash-Next-EXL3-2.05-mlx-serve	The earlier repo of 2.05bpw, now a stub that points here.

Each branch holds its earlier repo's files, without ngram_embedding.safetensors and with a new card.

How to download and run

Requirements

The packs can be served in two ways. The first is the turboexl3_qwen38 model plugin. The second is the route the packs were first measured on, and every speed table on this card comes from it.

We tested these packs on one Apple M5 Max with 128 GB of memory and did not test other Macs. We tested text only, not image or video input. The packs do not load in exllamav3, transformers, vLLM, SGLang or mlx-lm. For exllamav3, use turboderp's repository.

The turboexl3_qwen38 model plugin

The main branch of CommunityServe/plugins has the turboexl3_qwen38 model plugin, which serves these four packs. Its source is in plugins/turboexl3_qwen38. It claims a qwen4_exp pack only when the pack's config has a trunk_quant block, as these packs' config.json does.

The plugin needs mlx-serve 26.11.1 or later and an M5-class GPU (NAX), and it refuses any other GPU by name at load. MTP needs mlx-serve with ddalcu/mlx-serve#1017, which is merged. Without it, serve with --no-mtp.

The plugin is built with scripts/build.sh turboexl3_qwen38 from the root of CommunityServe/plugins, and the plugin's README gives the build steps. The repo README and docs/using-plugins.md cover installing a plugin and turning it on.

qwen38-exl3-full and the turboexl3 quant plugin

The packs were first measured on this route, which takes two parts.

The first part is mlx-serve with full-EXL3 support for Qwen3.8, which means the trunk_quant block, the EXL3 n-gram table and the EXL3 MTP head. That support is the branch qwen38-exl3-full of davidtai/mlx-serve, the head of ddalcu/mlx-serve#997 before that pull request became the host support of the qwen38_exl3 model plugin.

The second part is the turboexl3 quant plugin. It is required, because it claims the pack's expert_quant and trunk_quant blocks. Build it from the plugin/ directory of davidtai/turboexl3.

Download a branch

Pass the branch name to --revision. This command downloads 5.05bpw, and the steps below continue with it:

hf download OpensourceWTF/Qwen3.8-Flash-Next-exl3 --revision 5.05bpw --local-dir ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve

Each branch's card gives the same steps with its own folder name.

The n-gram table

The branches do not include ngram_embedding.safetensors, the n-gram table, and each pack needs it in its folder. mlx-serve maps the file and reads the rows each token needs, so the table costs disk space and OS file cache, not resident memory.

For 4.05bpw and 5.05bpw, the table is turboderp's file, unchanged. Download ngram_embedding.safetensors from turboderp/Qwen3.8-Flash-Next-exl3 at branch 4.05bpw_h6_ng6 or 5.05bpw_h6_ng6 and put it in the pack folder as it is. Both branches ship the same file, with sha256 c63613666649014e668cdbd32a4e3f1e8beda697eb4f3a1874300d93a1bfbc75 and a size of 39,040,193,720 B. For the 5.05bpw folder above, these commands fetch the file and check it:

hf download turboderp/Qwen3.8-Flash-Next-exl3 ngram_embedding.safetensors --revision 5.05bpw_h6_ng6 \
  --local-dir ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve
shasum -a 256 ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve/ngram_embedding.safetensors
#   expected: c63613666649014e668cdbd32a4e3f1e8beda697eb4f3a1874300d93a1bfbc75 (39,040,193,720 B)

For 2.05bpw and 3.05bpw, turboderp stores the table as 128 row shards in his ngram_embedding.safetensors, on branches 2.05bpw_h4_ng4 and 3.05bpw_h5_ng5. mlx-serve reads one tensor, so the converter makes the pack's table from his file. The converter's scripts/restack_exl3_q2_full.py, in its --phase finalize with --ngram <his file>, clones the file with APFS clonefile, so no data is copied, and rewrites only its header to name the same bytes as one tensor. The converter README describes this in "What the conversion does", in the n-gram table bullet.

Our results are a 3.05 table of 32,640,183,408 B (sha256 f7fa0ddfe932…) and a 2.05 table of 26,240,152,672 B (sha256 1f5d5d099b46…). Each branch's card has the exact steps.

Build the turboexl3 quant plugin and start the server

For the second way, build mlx-serve with the support listed in Requirements and its plugin kit, then build the turboexl3 plugin from plugin/:

# in the mlx-serve checkout
zig build -Doptimize=ReleaseFast
zig build kit      # stages zig-out/kit/mlx-serve-plugin-kit-<version>/

# in a checkout of github.com/davidtai/turboexl3
cd plugin
ln -s <mlx-serve checkout>/zig-out/kit/mlx-serve-plugin-kit-<version> kit
zig build          # writes zig-out/lib/libturboexl3.dylib
mlx-serve plugin check zig-out/lib/libturboexl3.dylib --model ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve
#   expected: claims expert_quant (by_marker), trunk_quant (by_marker)
mlx-serve plugin install zig-out/lib/libturboexl3.dylib
# then set "enabled": true in ~/.mlx-serve/plugins.json

Then start the server:

MLX_SERVE_NOCACHE_LOAD=1 mlx-serve --model ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve --serve

MLX_SERVE_NOCACHE_LOAD=1 reads the shards past the OS file cache, so the load does not also fill memory with cached copies of the shards. Our measurements used it.

The default numerics use turboexl3's kernels, which reorder some sums. MLX_SERVE_BIT_EXACT=1 selects exact numerics: the routed experts run on Sushi, the EXL3 kernels built into mlx-serve, and no sum is reordered. The two modes give different text.

turboexl3 runs the routed experts only on an Apple M5 or later with macOS 26.2 or later. On other Macs, the plugin leaves the routed experts to Sushi, but we did not test this.

The plugin links no MLX of its own, so the MLX version of the kit must be the same as the server's.

Quality

Branch	KLD, turboderp's	PPL, turboderp's	HumanEval pass@1 (of 164)	HumanEval+ (of 164)	MBPP pass@1 (of 378)	MBPP+ (of 378)
5.05bpw	0.0040	1.3848	0.988 (162)	0.945 (155)	0.955 (361)	0.812 (307)
4.05bpw	0.0066	1.3897	0.988 (162)	0.963 (158)	0.958 (362)	0.796 (301)
3.05bpw	0.0174	1.4074	0.982 (161)	0.939 (154)	0.950 (359)	0.820 (310)
2.05bpw	0.0681	1.4871	0.970 (159)	0.927 (152)	0.958 (362)	0.807 (305)

The KLD and PPL columns are turboderp's. They are the mean KL divergence D_KL(p_BF16 ‖ p_quant) and the perplexity over a self-generated in-domain trace of 14,496 input and 45,588 output tokens, measured by turboderp with exllamav3 on his files. On that trace BF16 has a perplexity of 1.38474, and the KLD noise floor is 0.00262. The packs hold his quantized bytes, but these numbers were not measured through mlx-serve.

The EvalPlus columns are our end-to-end check through mlx-serve: EvalPlus on each pack, greedy pass@1, with the number of problems passed in parentheses. The setup is in the coding evals of turboexl3's results.

The charts below are turboderp's, linked from his model card. In them, the EXL3 series is his plain quants, and our packs are its 2.05, 3.05, 4.05 and 5.05 points. EXL3-SC is his self-calibrated quants, and the other series are other formats he measured.

Speed

The speed tables were measured on one Apple M5 Max with 128 GB under macOS 27.0.1. The server was our mlx-serve tree with turboexl3, before its port to the current mlx-serve, on two builds. 5.05bpw ran on build S21, which is build PR17B plus the later decode optimizations that the branch qwen38-exl3-full ports, at MTP depth 4. 4.05bpw, 3.05bpw and 2.05bpw ran on build PR17B at MTP depth 3, and they were not re-measured on build S21. Compare branches within one build only. These tables come from the second way that Requirements describes, not from the turboexl3_qwen38 plugin.

How the speed was measured

Each cell is one run, with no repeats. For each prompt length and mode, a warm-up server first ran the prompt for one token, which puts the prompt's n-gram rows into the OS file cache. Then the other pack files were evicted from the cache, a new server started, the SoC cooled below 40 °C, and one request ran.

The request was the prompt as one user message in Qwen's chat template with thinking on, greedy (temperature 0), with seed 1234 and at most 1,024 new tokens. The prompts are fixed texts, Python source code with one coding task, and they are the same bytes for every pack and mode.

The server flags were --ctx-size 262144 --max-concurrent 1 --mtp-max-depth 3 --mtp-typical 0.2 --prefix-cache-mem 1GB, with --mtp-max-depth 4 on build S21. The environment variables were MLX_SERVE_GPU_CEILING_MB=114688, MLX_SERVE_CACHE_LIMIT=2147483648, MLX_SERVE_NOCACHE_LOAD=1 and MLX_SERVE_NGRAM_WARM=0.

The prefill and decode rates come from the server's log line for the request. TTFT is the client's time from sending the request to the first streamed token. Peak memory is the server's peak phys_footprint, which includes Metal memory but not the file cache of the n-gram table.

turboexl3 reorders sums, so the two modes give different answers. With MTP, their decode rates also depend on how many drafts each answer accepts.

In the tables, "turboexl3" is the default numerics and "exact" is MLX_SERVE_BIT_EXACT=1. Prefill and decode are in tokens per second, and * marks a run whose answer reached the 1,024-token cap.

Build PR17B, MTP depth 3: 4.05bpw, 3.05bpw and 2.05bpw

None of the 18 runs of 4.05bpw reached the cap, against one of 18 for 3.05bpw and one of 18 for 2.05bpw.

Prefill, tok/s

prompt	tokens	4.05bpw, turboexl3	4.05bpw, exact	3.05bpw, turboexl3	3.05bpw, exact	2.05bpw, turboexl3	2.05bpw, exact
1K	1,031	1373.0	1025.5	1572.3	1580.5	1603.2	1648.9
2K	2,013	1423.7	1916.3	2080.1	2057.0	2114.1	2147.0
4K	4,025	2116.1	1938.8	2497.5	2464.7	2554.9	2518.1
8K	8,100	2711.0	2488.6	2769.8	2498.3	2809.4	2412.1
16K	16,275	2830.0	2538.2	2834.5	2599.3	2771.8	2714.5
32K	32,424	2759.3	2492.8	2766.3	2578.8	2767.0	2664.4
64K	64,987	2639.1	2485.6	2659.6	2484.4	2680.0	2543.3
128K	129,949	2512.4	2398.9	2522.2	2399.0	2544.4	2451.4
256K	258,992	2344.0	2270.0	2348.0	2273.7	2362.3	2318.8

Decode, tok/s

prompt	tokens	4.05bpw, turboexl3	4.05bpw, exact	3.05bpw, turboexl3	3.05bpw, exact	2.05bpw, turboexl3	2.05bpw, exact
1K	1,031	141.0	119.7	140.0	131.8	131.5	126.2
2K	2,013	134.8	114.2	131.3*	126.6	123.6*	119.6
4K	4,025	132.3	116.5	135.2	124.5	125.3	113.3
8K	8,100	135.7	116.1	134.4	128.2	127.6	118.2
16K	16,275	137.6	112.5	132.3	126.6	120.6	110.7
32K	32,424	126.4	108.7	128.5	117.8	117.8	107.5
64K	64,987	119.5	102.5	116.6	112.0	110.9	108.5
128K	129,949	114.3	96.8	110.4	106.3	101.3	98.9
256K	258,992	105.5	89.6	99.4	93.6	96.0	91.8

TTFT, s

prompt	tokens	4.05bpw, turboexl3	4.05bpw, exact	3.05bpw, turboexl3	3.05bpw, exact	2.05bpw, turboexl3	2.05bpw, exact
1K	1,031	0.76	1.01	0.66	0.66	0.65	0.63
2K	2,013	1.42	1.06	0.98	0.99	0.96	0.95
4K	4,025	1.91	2.09	1.62	1.64	1.59	1.61
8K	8,100	3.00	3.27	2.94	3.26	2.90	3.37
16K	16,275	5.77	6.43	5.76	6.28	5.89	6.01
32K	32,424	11.77	13.03	11.74	12.60	11.74	12.19
64K	64,987	24.66	26.18	24.47	26.19	24.28	25.59
128K	129,949	51.78	54.23	51.58	54.23	51.13	53.07
256K	258,992	110.60	114.20	110.41	114.02	109.74	111.80

Peak memory, GiB, turboexl3

prompt	tokens	4.05bpw	3.05bpw	2.05bpw
1K	1,031	71.5	56.7	41.9
2K	2,013	71.1	56.4	41.6
4K	4,025	71.3	56.7	42.9
8K	8,100	73.5	59.1	42.4
16K	16,275	75.1	60.2	45.4
32K	32,424	75.7	60.8	46.1
64K	64,987	77.2	62.3	47.8
128K	129,949	79.9	65.4	50.8
256K	258,992	84.8	70.4	55.6

Build S21, MTP depth 4: 5.05bpw

Three of the 18 runs reached the cap. At 256K the server narrowed its prefill chunk to 4,096 tokens, against 8,192 at the other lengths.

prompt	tokens	prefill, turboexl3	prefill, exact	decode, turboexl3	decode, exact	TTFT s, turboexl3	TTFT s, exact	peak GiB, turboexl3
1K	1,031	1093.7	955.7	134.9	126.8	0.95	1.09	86.2
2K	2,013	1625.5	1161.2	126.5*	122.0*	1.25	1.74	85.9
4K	4,025	1987.2	1884.1	138.2	123.6	2.04	2.15	86.5
8K	8,100	2555.2	2018.0	126.6	123.1	3.18	4.03	87.9
16K	16,275	2711.8	2264.2	124.2*	122.3	6.02	7.20	90.1
32K	32,424	2674.0	2334.2	122.8	114.3	12.15	13.91	90.2
64K	64,987	2550.7	2242.2	113.6	107.8	25.51	29.02	92.2
128K	129,949	2500.4	2179.0	108.9	103.6	52.03	59.70	94.9
256K	258,992	2329.1	2073.1	95.2	92.3	111.31	125.03	100.1

Serial decode

The serial-decode runs used --no-mtp in place of the two MTP flags and the same method otherwise. Four of the 18 runs reached the cap.

prompt	tokens	prefill, turboexl3	prefill, exact	decode, turboexl3	decode, exact	TTFT s, turboexl3	TTFT s, exact	peak GiB, turboexl3
1K	1,031	1157.9	1014.1	55.9	56.8	0.90	1.03	84.0
2K	2,013	1578.1	1439.9	55.0*	56.0*	1.29	1.41	83.3
4K	4,025	2173.5	1756.9	55.1*	56.2*	1.86	2.30	83.3
8K	8,100	2609.6	2211.2	54.9	55.9	3.12	3.68	85.5
16K	16,275	2751.4	2277.3	54.9	55.8	5.93	7.16	87.2
32K	32,424	2698.0	2290.3	54.2	55.1	12.04	14.18	87.7
64K	64,987	2604.9	2269.8	53.2	54.2	24.98	28.67	88.9
128K	129,949	2503.1	2203.3	52.0	53.1	51.97	59.04	91.1
256K	258,992	2345.8	2094.2	51.2	51.9	110.51	123.78	95.8

How the packs were made

In every branch, the quantized weights are turboderp's bytes, rearranged into mlx-serve's file layout. The n-gram table is his too, but it is not in the branches, as The n-gram table explains. The unquantized f16 tensors were rounded and are stored as bf16.

Three files are ours: the carry shard (trunk-exl3-carry.safetensors), config.json and model.safetensors.index.json. The vision shard (trunk-model-vision.safetensors) and the small files come from our earlier pack.

The conversion scripts do the repack, and each branch's card lists every file with its source and the change made to it.

Credits

The model, the tokenizer, the chat template and the license are Qwen's. turboderp made the EXL3 quantizations that these packs are made from, with exllamav3. EXL3 is based on QTIP by Cornell RelaxML.

ddalcu wrote mlx-serve and its Qwen3.8 model code, and made Sushi, the EXL3 kernels built into mlx-serve. The pack ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit is the source of the packs' vision tower and base config.

turboexl3 provides the EXL3 Metal kernels of the default mode. The conversion scripts are OpenSourceWTF/qwen38-exl3-mlx-serve-converter.

License

The weights are derived from Qwen3.8-Flash-Next and stay under the Qwen Community License 1.0. That license is the LICENSE file, the same as in Qwen's repository, and turboderp's and ddalcu's repositories use the same license.

The LICENSE file is the license, and what follows is only a short summary of it. Keep the copyright notice and the permission notice with every copy. A commercial product or service that uses the model and has more than 100,000,000 monthly active users, or more than US$20,000,000 monthly revenue, must show the model name in its user interface.

A licensee that runs, or whose affiliate runs, a "Model as a Service" or "AI Work Assistant" business (as the license defines them) needs a separate license from Qwen before any commercial use. Internal use that gives third parties no access to the model, its outputs or its capabilities does not need one. There is no warranty.

The code (mlx-serve, Sushi, turboexl3, exllamav3 and the conversion scripts) has its own licenses.