I prefer this quant to unsloth
AtomicChat is the new source for me to download GGUF recently. I used their Qwen3.8-27B-AD-Q5_K_M-Q4_K_M.gguf and it is better than Qwen3.8-27B-UD-Q4_K_XL.gguf from unsloth. The quant is a little bigger than Q4_K_XL but the quality is also better while keeping the decoding speed the same.
For QWEN3.8 flash next, first I downloaded IQ4_XS from unsloth and it did not work well in my setup because of its size 90GB. So I search for other GGUF and found IQ4_XS from AtomicChat which only 79GB (maybe it because of ngram that was quantized differently)
I hope this will help other people who have similar setup: 1 RTX3090 24GB and 64GB RAM DDR5
I got 18-20 t/s tg, 115k context size, kv cache q8_0, with vision
Use the latest llamacpp.
llama-server.exe -m F:\models\qwen38-flash\Qwen3.8-Flash-Next-AD-3.84bpw-IQ4_XS-M64-00001-of-00028.gguf --alias qwen38-flash -ngl 99 --n-cpu-moe 32 -fa on --fit off --ctx-size 114688 -ctk q8_0 -ctv q8_0 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --host 0.0.0.0 --port 2000 --load-mode mmap --tensor-read-lazy auto --verbosity 4 --reasoning on --reasoning-preserve -b 2048 -ub 1024 --reasoning-budget -1 --jinja -n -1 --mmproj F:\models\mmproj\mmproj-Qwen3.8-Flash-Next-BF16.gguf --no-mmproj-offload
NOTE: about the intelligence of this IQ4_XS - idk if it better than other quant or not but it DEFINITELY better than qwen3.8-27b. What I found is it use less thinking token and fix the bugs faster than qwen3.8-27b. Even if it is slower in decoding but use less token and solve problem faster. In my tests, both qwen3.8 can fix the bugs but the qwen flash next is the winner for token efficiency. E.g. the flash next use only 50% of 115k context size to fix a bug while 27b with the same context size need to compact 2-4 times to fix the same bug.
I am running this on GB10 (DGX Spark). It works well at first, but the issue is that avg token generation goes down from 20 tg/s to even 5 tg/s when a session gets long (e.g. 250k). Have you seen the same issue?
I am running this on GB10 (DGX Spark). It works well at first, but the issue is that avg token generation goes down from 20 tg/s to even 5 tg/s when a session gets long (e.g. 250k). Have you seen the same issue?
Same. I tested it in deepseek harness and it dropped to 4-5 t/s after multiple compactions but this also happened to unsloth quants so I think it is about the model design. From more tests, This quant stop decoding randomly sometimes too, it just stop while decoding which not found with unsloth quants. The advantage of the quant is it is faster and smaller comparing to the unsloth one with the same quality. The disadvantage is it just stop decoding occasionally.
I am running this on GB10 (DGX Spark). It works well at first, but the issue is that avg token generation goes down from 20 tg/s to even 5 tg/s when a session gets long (e.g. 250k). Have you seen the same issue?
Same. I tested it in deepseek harness and it dropped to 4-5 t/s after multiple compactions but this also happened to unsloth quants so I think it is about the model design. From more tests, This quant stop decoding randomly sometimes too, it just stop while decoding which not found with unsloth quants. The advantage of the quant is it is faster and smaller comparing to the unsloth one with the same quality. The disadvantage is it just stop decoding occasionally.
It seems many solo testers are developing solutions, one of which is here: https://github.com/hasso5703/dgx-spark-qwen38
After trying a few options, I am convinced to go back to adixArk/Qwen3.8-27B-NVFP4-BF16-LMHead, especially because I need 50GB headroom (to run another small model with at least 4 parallel requests) in my workflow. Perhaps give it a few days, and a new solution will come up. ;-)