Request for GGUF Update: Qwen 3.5 DeltaNet GPU Offloading Compatibility
First, thank you for your specialized MoE quants—the quality on your Qwen 3.5-35B-A3B sets has been awesome.
I wanted to check if you are planning to re-quantize or update the metadata for these models following the recent March 5th llama.cpp updates. It looks like the new Gated DeltaNet GPU kernels require specific metadata (specifically the delta_net_gpu_compat flag) to trigger full offloading.
On my local setup (RTX 5090 / CUDA 12.8), your current quants are still hitting the CPU very hard for the linear attention layers. A few other providers (like Unsloth) just pushed a "Final Update" on March 6th that resolved this, resulting in a ~30-40% speed boost and near-zero CPU usage.
Given how solid your imatrix calibrations are, I’d love to keep using your versions if they can be updated to support this new GPU offloading path.
Thanks for pointing me to those updates, I heard about them here and there but wasn't keeping track to see they've actually landed. I'll re-quant and upload over the next few days.
Actually looking at https://github.com/ggml-org/llama.cpp/pull/19504 I don't see anything about a delta_net_gpu_compat flag and there was no change to the convert_hf_to_gguf which would be needed to add any new KV metadata? Even doing grep on the master branch right now doesn't bring up anything about delta_net_gpu_compat.
Is there a way to see what Unsloth pushed the "Final Update" on March 6th that resolved this?
Someone else pointed out this PR to me: https://github.com/ggml-org/llama.cpp/pull/19139 and that does indeed change the convert_hf_to_gguf so that's probably the one OP meant. It needs a --fuse-gate-up-exp option but I'm not sure if it supports every model or just DS / GLM-4.7-Flash / GPT-OSS like the PR mentions. I'll test it later today.
Edit: looks like it should work with Qwen3.5: https://github.com/ggml-org/llama.cpp/pull/19139#issuecomment-3960623496
I've updated the quants in this repo and will be uploading the updated quants for the other two repos today as well.
after using your updated quant: Qwen3.5-35B-A3B-Q4_K_M,
I cannot use cpumoe, or n-cpumoe parameter, or it will cause cpu memory problems,
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 15670.80 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 16432024064
llama_model_load: error loading model: unable to allocate CUDA0 buffer
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'G:\lm-studio\models\AesSedai\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q4_K_M-00001-of-00002.gguf'
srv load_model: failed to load model, 'G:\lm-studio\models\AesSedai\Qwen3.5-35B-A3B-GGUF\Qwen3.5-35B-A3B-Q4_K_M-00001-of-00002.gguf'
srv operator(): operator(): cleaning up before exit...
main: exiting due to model loading error
but -fit on works.
however the generation speed drops from 40 to 20 tk/s, the prefill speed increases form 690 to 721 tk/s
My hardware: 4070 12gb with 64gb ddr 4 ram, win11, llama.cpp version precompiled b8263, cuda 12.4
@kironlau this is what I had measured on my system, but I wasn't trying n-cpumoe either. I wonder if the fused up+gate is just less flexible for CPU+GPU mixing? The speed drop is unexpected too. I think it might not be optimized well enough yet this is still a very new flag.
Maybe post your findings in this PR? https://github.com/ggml-org/llama.cpp/pull/19139
The only other thing I changed in the recipe was to bump the ssm_alpha and ssm_beta weights from Q8_0 to F32, the rest is identical.
Thanks AesSedai for your quants and thorough research and releasing data and methodology clearly, it is such a boon!
I wonder if the fused up+gate is just less flexible for CPU+GPU mixing?
In general I don't recommend splitting up|gate across devices, but I didn't think --n-cpu-moe would have done that. @kironlau one should expect to see 2-10% increase in PP and nominal TG using the pre-fused quants.
PSA: these new pre-fused quants by any quantizer don't run on ik_llama.cpp. UPDATE: it works now, ik added support! ik can do the fusion "on the fly" without baking it in a conversion time, but -sm graph is still better for 2+ GPUs there. Using ik's new -muge feature there is showing 2-6% increase in PP with -sm layer. I haven't tested using a single GPU.
Anyway, plenty of options, but it is getting confusing!
@kironlau I opened an issue in llama.cpp for this: https://github.com/ggml-org/llama.cpp/issues/20414
You should be able to get the same TG perf with manual tuning of the -ot flag but that's a bit of a manual process.
@kironlau @ddh0 put up a PR to fix the issue and it merged about an hour ago: https://github.com/ggml-org/llama.cpp/pull/20416
@kironlau @ddh0 put up a PR to fix the issue and it merged about an hour ago: https://github.com/ggml-org/llama.cpp/pull/20416
thankyou very much
I'm quite hestitate to report the bug, as I don't know how to benchmark the result
@kironlau @ddh0 put up a PR to fix the issue and it merged about an hour ago: https://github.com/ggml-org/llama.cpp/pull/20416
Just compiled of the latest PR, it works!!! the prefill speed is now 800 tk/s, increased from 700 tk/s.
thanks again!!!
Excellent! Glad to hear :)