Harsha001

/

Nanbeige4.3-33B-ROCmFPX

Like 3

Hey,

Thanks for reaching out! I'm already actively working on this. I recently updated my RocmFPX fork to support this workflow, and I'm aiming to get the quant ready within the next 2–3 days.

However, I can't guarantee a release yet, as it needs to meet a certain performance threshold before I publish it.

If everything works out, DeepSeek V4.1 Flash will be the first, followed by similar quantizations for other models.

I'll share an update once I have results.

Hey,

I've added two models with smaller quants. Please test them on your hardware and report back with the TPS (tokens per second) and output quality.

I've included the DeepSeek V4.1 Flash checkpoint you requested, along with MiniMax M3.1-Flash.

Looking forward to seeing your results!

1. Of course, will try! Btw. if there is 97gb in RAM and rest in SSD, what will happen if you'll do IQ4_XXS version? I do not know the logic behind this division and how the engine works, so just asking :D

2. And about benchmarking, I use it for local projects, so I'll have to setup sth, some auto tests, it may take a while (probably 2-3 weeks).

3. And you mention perplexity only, is it possible to do some benchmarking? Or I can do it (if you have some suggestions <3)

4. And is this GSQ-RCO quantization possible to do? Because it seems they managed to do it super efficiently, so do you base on their quants? Maybe you can add this to the model name ("Harsha001/DeepSeek-V4.1-Flash-1017-StrixHalo-GGUF" -> "Harsha001/DeepSeek-V4.1-Flash-1017-GSQ-RCO-StrixHalo-GGUF"). PS. I think you can write "ROCm" instead of "StrixHalo", or keep both, so final will be quite long, but eh "Harsha001/DeepSeek-V4.1-Flash-1017-GSQ-RCO-StrixHalo-ROCmFPX-GGUF"

@Arkovski Thanks a lot, and no rush on the tests! Answers in order:

1. IQ4 and the RAM/SSD split. Only the Engram table lives on the SSD. It is a lookup table, so each token reads just a few small rows from it, and the SSD handles that easily. The ~97 GB part is different: it's the attention layers plus the experts, and the GPU computes with them on every token, so they have to sit in memory.

At about 4.25 bits per weight (IQ4_XS; there is no IQ4_XXS type), all 384 experts come to roughly 275 GB. Only about 128 of the 384 experts per layer would fit in 97 GB, and cutting two thirds of the experts hurts quality much more than going down to ~2 bits. That's why the builds keep many experts at low bits instead.

You could also stream the experts from the SSD. Each token uses 6 experts per layer, about 4.5 GB of reads per token at 4 bits, so even a fast NVMe would give around 1 token per second or less.

2. Benchmarking. 2–3 weeks is totally fine. Even quick numbers before that would help a lot:
llama-bench -m <first shard> -ngl 99 -p 512 -n 128 (add -d 8192 for speed at longer context), plus how much memory it uses.

3. Real benchmarks. Agreed, perplexity is only a first check. We're now running GSM8K (math word problems) on both DeepSeek and MiniMax on our side and will put the scores on the model cards. For your own runs, an easy route: start llama-server and point lm-evaluation-harness at it, for example
lm_eval --model local-chat-completions --model_args model=ds,base_url=http://localhost:8080/v1/chat/completions --tasks gsm8k_cot --apply_chat_template --limit 200.
Tasks from your own projects are the most useful signal, though: same prompts, compared against the full model through an API.

4. GSQ-RCO. Yes, the DeepSeek build is based on IST-DASLab's GSQ-RCO release. Their quantized experts are copied unchanged; we chose which experts to keep (REAP ranking) and repackaged it to fit 128 GB. The card credits them. Running GSQ-RCO ourselves needs the full-precision model and a lot of GPU time; our own low-bit formats in that direction for AMD are the next step. The MiniMax build is based on Unsloth's UD quants, not GSQ-RCO.

On the name: done, it's now Harsha001/DeepSeek-V4.1-Flash-1017-GSQ-RCO-StrixHalo-GGUF (the old link redirects). I left out "ROCm"/"ROCmFPX", because these files use standard llama.cpp types and run on Vulkan, ROCm, CUDA and CPU. "Strix Halo" only means "sized to fit 128 GB". MiniMax is here: Harsha001/MiniMax-M3.1-Flash-StrixHalo-GGUF.