QuantPasture

/

Kimi-K2.5-GGUF

Like 30

Follow

Aes Sedai's Quant Pasture 8

While doing some "without GPUs" stability tests, and since some Reddit users keep asking for EPYC CPU-only numbers, I ran a couple of EPYC 9355 benchmarks without GPU on Q4_X. This is with 12x channel DDR5-6400:

./llama-sweep-bench \
    --model "$MODEL_PATH" \
    --no-mmap \
    -mla 3 -amb 512 \
    -b 8192 -ub 8192 \
    -ctk f16 -ctv f16 -c 65536 \
    --merge-qkv \
    --threads 16 \
    --threads-batch 28 \
    --warmup-batch \
    -n 512

./llama-sweep-bench \
    --model "$MODEL_PATH" \
    --no-mmap \
    -mla 3 \
    -ctk q8_0 -c 131072 \
    --merge-qkv \
    --threads 24 \
    --threads-batch 32 \
    --warmup-batch \
    -n 512

A better chart to see TG t/s falling as context grows on CPU-only:

thanks for the tests, I thought of moving to turin from milan, but looks like it make more sense to invest in rtx 6000 pro

@ArtemProc RTX PRO 6000 is great for long context. I own one, and I’m thinking about replacing the 5090 with another 6000, too. But unless you buy 8, 6, or at least 4 of them, it won’t let you run the largest models. Hybrid inference is much more affordable... ehm, was much more affordable before the RAM price hike - and as long as you’re okay with quantizing the KV cache, 32 or 48 GB VRAM is enough.

That said, the situation has changed dramatically over the last few months. I paid $7,500 for 1,152 GB DDR5-6400 ECC RAM last year. Right now it would cost over $46,000 if the prices listed on memory.net reflect reality. This is crazy talk. If RAM prices keep rising, I’ll sell my RAM to buy a new house.

About performance: did you notice the drop in TG t/s around ~25,000 KV in the CPU-only chart? That might be caused by thermal throttling - see the related discussion here. Also, this is running inside a VM, and I’m not pushing the CPU beyond the “Balanced” profile in BIOS. You could likely squeeze another ~20–25% out of it if trying hard.

You might also find this benchmark run of Kimi-K2-Thinking relevant if thinking about your future setup.

Yes, it is obvious for me about hybrid vs VRAM only ofcourse.
I am running iq4_k_nl, on 7713 with ddr4 2933 8 channel (used memory was at the electronic disposal point, i.e. free)
Tg starts at 7 and drops to 5 on context 35k+
Pps start at 25 and drops to 19.
It is clean Turin 12 channel bw is doing a great job here.
Btw smt is most probably useless here + less thread may allow better per core boost.
I was thinking of building on Turin in september 2025, but could not justify the need.
It is more like hobby and not business, I just love building systems.

P.s. Ddr5: Nice investment if you can turn it in money;)

@ArtemProc Interesting. So it seems at longer context, TG t/s is not significantly better on 9355 vs 7713 and DDR5-6400 vs DDR4-2933, only PP t/s is. In that case, it seems your investment was even better than mine 😀.

Pps is really important for the agentic coding usage. As you have it, why not. Tg it is ok for wait, but in long sessions pps is killer of the joy.

@sousekd @ArtemProc - i've been testing Kimi & GLM 5 on GPUs only recently, posted some number in ik_llama - https://github.com/ikawrakow/ik_llama.cpp/issues/1481. Let me know if you want me to run anything on a bunch of RTX 6000 Pros. Btw does the image input work for all of those quants here?

@curiouspp8 Image input does work on the quants here if you include the mmproj. I recently KLD / PPL tested these quants and discovered that the something is catastrophically bad about the IQ2_S quant, and the IQ3_S isn't great either but it's not nearly as bad. The IQ2_XXS seems to have the lowest KLD / PPL compared to the IQ2_S / IQ3_S.

I've made a couple of new quants at those sizes that shows a more reasonable and expected curve for the KLD / PPL and I'll try to get them uploaded today or tomorrow and update the model card.

I've tried ubergarm ones and smol-IQ2_KL seems like a nice one for 4 cards, which is very convenient, as even if there are more GPUs the overhead adds up. Quality was very workable <100k context. Now looking for a quant that supports the image, would you say IQ2_XXS 283 GB is the best shot I should I go higher?

@curiouspp8 I've just finished uploading the remade quants, I'd say go with whichever one you can comfortably fit with the context you need to run. All three of the non-Q4_X quants look much better now on the KLD / PPL charts compared to the previous quants.