it works great on M1 Max 64GB
Been testing for the past few hours, will continue to. Running AtomicChat/Qwen3.8-Flash-Next-GGUF:Q4_K_M on a Macbook M1 Max 64gb, it gets 11/17 tok/s depending on the size of the used context (I'm going with 65k), the performance is good enough to let it run in background while I'm working on other things, and it seems to be quite great!
I've been testing qwen3.8 27B 4bit as well, I'll try to give both the same requests, this one seems to be both faster and better, thank you!
Adding an M2 Max data point, and one thing that may explain your 11–17 spread.
M2 Max, 64 GB, AD-3.84bpw-IQ4_XS-M64 — llama-server -ngl 99 -fit off -np 1, build b10698:
context prefill decode
8k 309.0 22.9
32k 308.2 20.2
128k 306.6 20.2
256k fails, Metal OOM
Prefill on a 5 000-token prompt, three runs within 0.2 %; decode streamed over 400 tokens. Steady from the first tokens — no warm-up ramp.
Two things that mattered on 64 GB:
The wired limit. sudo sysctl iogpu.wired_limit_mb=53248 (52 GB). Default is lower and -ngl 99 then fails; the weights want ~46 GB in Metal. Measured with vm_stat "Pages wired down" — footprint reports ~1.8 GB for this process and misses Metal buffers entirely.
The variant. You are on Q4_K_M, which is ~55 GB of weights. That does not fit under a 52 GB limit, and a spread of 11–17 tok/s by context is what memory pressure looks like. IQ4_XS is ~46 GB and sits inside it — that may be worth more to you than the extra 0.4 bpw.
-fa auto already picks flash attention here, and -b/-ub 2048 exceeds the limit. --spec-type ngram-simple gave about +12 % decode on code-editing prompts at no memory cost.
Hi @grokked-it , I also observed a sharp generation speed drop strictly correlated with the used context on the M4 Max 64GB and it my case it's linear. See my post https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF/discussions/13.
RE: @Blackbeard82 - memory could be a factor due to the model architecture but it wasn't in my experiment. In one of my trials with a small -c and a very conservative iogpu.wired_limit_mb at 56GB (my OS and server stack takes 3GB) my memory usage was stable at 60GB and there was no swapping, yet I still got the linear generation speed drop.
context prefill decode
8k 309.0 22.9
32k 308.2 20.2
128k 306.6 20.2
256k fails, Metal OOM
are these prefill speed all for the 5000 tokens prefill at different context length limit settings? curious to see a test with your set up for input and output speed at different context length actually consumed :)
Good catch — you were right, and I have measured it properly now.
Those were context limit settings, not consumed context. Same ~5 000-token prompt in every row; only -c changed. And worse, the 8k row came from a different run with a short prompt, so it was not comparable to the other three at all.
Here is the actual curve on M2 Max 64 GB, same server, prompt length varied, 220 tokens generated each time:
prompt tokens prefill t/s decode t/s
1 188 291.0 21.7
5 808 302.6 19.7
17 464 261.8 16.5
34 894 220.5 13.2
So your observation holds: decode falls with consumed context, about 39 % from 1k to 35k, and prefill falls with it. Memory was steady throughout — no swap, no growth in wired pages — so I would agree with you that memory is not the mechanism.
One thing that may be worth trying on your M4 Max, since it costs nothing: --spec-type ngram-mod. On a copy-heavy task (return a file with one line changed) it took decode from 22.1 to 54.5 t/s here, with byte-identical output. It only helps when the output repeats the prompt, but that is most of what code editing is.