UD-Q4_K_XL expected strong performance drop compared to AtomicChats's Q4_K_M?
Hi! I’m currently testing your SassyDiffusion/Qwen3.8-Flash-Next-GGUF build and wanted to ask about its performance.
For reference, my hardware is:
- CPU: Intel Core i9-14900HX
- GPU: RTX 4070 Laptop GPU, 8 GB VRAM
- RAM: 64 GB DDR5
I tested the following AtomicChat build:
On my system it runs at approximately 16 tokens/sec.
However, when I run your UD-Q4_K_XL build, I only get around 6 tokens/sec on the same system.
I would prefer to use your quantization because, as I understand it, it should provide better quality than the standard Q4_K_M build. But the performance difference is quite large.
So I wanted to ask:
Is this ~2.7× performance drop expected for your UD-Q4_K_XL quantization, or could there be something unusual/incorrect with the build itself?
If possible, could you also compare your quantization against the AtomicChat builds here:
https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF
They provide the following data:
| Build | In memory | On SSD | Total | Mean KLD | Same top-1 | PPL ratio |
|---|---|---|---|---|---|---|
| AD-3.84bpw-IQ4_XS-M64 | 45.8 GB | 39.1 GB | 84.9 GB | 0.2277 | 82.68% | 1.102 |
| AD-4.27bpw-Q4_K_M-M64 | 54.5 GB | 38.4 GB | 92.9 GB | 0.0842 | 89.49% | 1.026 |
| AD-5.00bpw-Q5_K_M-M64 | 56.1 GB | 54.4 GB | 110.5 GB | 0.0837 | 89.55% | 1.026 |
It would be really useful if you could provide the same statistics for your UD-Q4_K_XL build.
Thanks!
Hi, can you try if you can reproduce this with the official unsloth Q4_K_XL?
This quant only moves the per layer embedding table out from a gguf shard and replaces it with bf16, everything else are as is from unsloth.
You would need about 95GB of RAM+VRAM combined for Q4_K_XL to fit everything in memory (except for embeddings), not sure how you run this quant with only 72GB of memory, but you might just need more RAM?
I just checked AtomicChat's Q4_K_M, it looks like a very aggressive custom quantization, it does not seem like a standard Q4_K_M quant. The file type in the gguf metadata even shows IQ2_S instead of Q4_K_M and their quant size is about the same as unsloth's IQ4_XS despite using a bigger Q5_1 for the ngram embedding (compared to unsloth's IQ4_NL), which means the actual model size is even smaller without the embeddings (if using IQ4_NL for the embedding, the model size is about the same as unsloth's Q2_K_XL or IQ3_XS, that is very sus for a Q4_K_M), in their model card, their Q4_K_M has 0.0842 KLD, that also seem suspiciously high for a Q4_K_M quant, I'd check with them if something's wrong with their Q4_K_M quant before using it.
They named it Q4_K_M because the total BPW (including PLE) is 4.27 — indeed, Q5_K_M captures this logic even more clearly.