Linear speed degradation correlated with context length
Thanks for making this available.
I've used this model for a little while over the weekend on my Mac Studio M4 Max 64GB. One thing I discovered was the performance degrades with context linearly. On my machine the generation speed in tokens per second can be reliably predicted by 1000/(31.6 + n * 4.7 * 10^-4). It starts out at around 31 T/S in a fresh conversation and drops to 20 pretty quickly, after one or two turns of agent actions. It really compounds with how much this model thinks too. Memory was not the bottleneck.
Is this expected and / or explained by the model's architecture? any ideas to optimize the degradation slope?
adding more details - 4.27bpw variant,
--jinja --flash-attn on
-ngl 99
--fit off
--no-warmup
--kv-unified
-ctk q8_0 -ctv q8_0
--cache-ram 0
llama cpp b10679
same result on different context length limits:
"q38fn_slots_4_in_140k_unified": "--parallel 4 --ctx-size 140000"
"q38fn_slots_4_in_180k_unified": "--parallel 4 --ctx-size 180000"
"q38fn_slots_4_in_240k_unified": "--parallel 4 --ctx-size 240000"
"q38fn_slots_6_in_240k_unified": "--parallel 6 --ctx-size 240000"
"q38fn_slots_6_in_256k_unified": "--parallel 6 --ctx-size 262144"
I had to push the iogpu.wired_limit_mb to 60GB for the full context length, but with --ctx-size 140000and iogpu.wired_limit_mb at 56GB I saw the same linear degradation and memory was definitely not the bottleneck, total memory usage was steady at 60GB and no swapping.
Your formulation is the right one — it is time per token that grows linearly, not tok/s that falls. Fitting our M2 Max 64 GB data the same way:
build ms/token
b10698 45.3 + 8.73e-4 · n
b10731 45.3 + 5.90e-4 · n
The slope drops 32 %, the intercept does not move (−0.2 %).
That is #28023, merged today 04:24 UTC — eleven lines, qwen4exp: sum the indexer heads by slices. The head summation went through a transpose and a reduction over a dimension of 4, and the transpose copied a large surface twice for nothing. You are on b10679, so you do not have it; it is in b10730 and later.
Measured here, same prompts, same flags, only the binary swapped:
prompt tokens decode b10698 → b10731 prefill b10698 → b10731
1 188 21.7 → 21.9 291.0 → 290.9
5 808 19.7 → 20.4 302.6 → 306.9
17 464 16.5 → 18.0 261.8 → 275.2
34 894 13.2 → 15.2 220.5 → 241.5
So: part of the slope was a kernel inefficiency, not architecture. The rest is inherent — attention still grows with context.
Two other things, in case they help:
--spec-type ngram-mod does not change the slope, but it multiplies throughput where the output repeats the prompt. On a copy-heavy task here (return a 60-line file with one line changed) decode went 22.1 → 54.5 t/s, byte-identical output. For code editing that is most of the work. ngram-simple gave 50.4; --spec-draft-n-max made no difference at 8 or 16.
Your --parallel 4-6 may be working against you for single-turn latency: the KV cache is divided across slots, so each conversation gets a fraction of the window and slots compete. We run -np 1. And your q8_0 KV cache cost about 4.5 % prefill in our measurements (310 → 296 t/s), so it buys memory rather than speed.
For reference, your fit against ours at n = 35 000: yours 20.8 t/s, ours 15.2 t/s on b10731. The M4 Max is ahead on both terms, as expected.