Throughput scaling with threads and context length?
The 615-740 tok/s vs 116 for Q8_0 is a big gap. Was it all measured on the single 8-thread Xeon with context 128? I would like to know how it scales with thread count and longer contexts, since the GGUF side usually gains more from both. I plan to run the AVX2 kernel and llama-bench -ngl 0 on my own CPU and can share the numbers here.
Thanks for comment . Yes, all of the paper's numbers were one laptop: Xeon W-11955M, 8 threads, greedy generation at a 128-token context, 256-token runs. Our kernel's prefill is sequential while llama.cpp batches it.
I don’t have long contex for this particular model but i do have a related long-context test from our next model's kernel (same design, random weights, so speed only), on the same laptop against SmolVLM2-256M at Q8_0 in llama.cpp: llama.cpp went from 217 tok/s in short chat to 80 at 4k and 53 at 7.7k context, while our kernel went from 469 to 241 at 9.3k and 210 at 12k. So at least in that test, longer context hurt llama.cpp more, not less.
But please share ur numbers , problem is I can’t port to any existing runtime because the output layer is not a normal matrix. It's a 1-bit readout straight off the frozen table: 262,144 rows of 512 bits, scored with bit operations plus a per-token prior, and that is a large part of why it's fast and small. GGUF has no op for that.
Thanks for the details. The long-context result is the opposite of what I expected, especially with your prefill being sequential while llama.cpp batches it. Understood on the 1-bit readout, so I will run your AVX2 kernel directly and llama-bench -ngl 0 on the same CPU, and post both numbers here when I have them.
Ran both on the same machine: an 8-vCPU KVM VM on a recent Intel Xeon (family 6 model 207, AVX2/AVX-512), 8 threads, load average 0.7 at start.
- cgemma.c (repo commit a3d3da6),
./cgemma kernel/armC.bin bench 128 256, 3 runs: 855, 859, 774 tok/s generation (prefill 1236-1668 tok/s, peak RSS ~61 MB). - llama.cpp b11515 (CPU build),
llama-bench -m gguf/armA-Q8_0.gguf -ngl 0 -t 8 -p 128 -n 256: tg256 320.6 ± 15.0 tok/s, pp128 4108 ± 310 tok/s. With-d 128, tg256 is 295.5 ± 20.4 tok/s.
So I get about 2.4-2.9x on generation here, compared with 5.3-6.4x in the paper. llama.cpp seems to benefit more from this CPU than from the laptop.
One issue: with PIN=1 on Linux all 8 threads ended up on the same CPU (0.3 tok/s). pin_thread(0) runs on the main thread before the workers are created, so the workers inherit the single-CPU mask and sched_getaffinity only sees that one CPU. The numbers above are without PIN.