So what do you think of it so far?
I haven't had a chance to test your quant yet as I don't have the time to setup vLLM for it, and was wondering what are your impressions when using the model/quant so far?
I'm interested in quality not speed, because many seem to be chasing speed (max token output) for whatever reason.
vLLM doesn't support this model yet, so I can't give a final verdict on it. That said, from using it with llama.cpp, it's pretty good.
First, it reasons efficiently and doesn't overthink. Like Gemma-4 levels of efficient thinking.
Second, it quantizes well. Even at INT4 it holds up nicely.
Its KV cache efficiency is also very high. This is an agentic model, and for that use case you can never have enough context. I was able to fit 822,190 tokens of context, and even on 2x RTX 3090 it batches requests well at that length.
Besides that, if you don't have the time to test this model, you could just skip it and wait for Qwen3.8-27B to drop. That should be a clear step up in performance.
Maybe I'll try it with llama.cpp then but I've got a setup with vLLM that works for my needs that I'm hard pressed to change. Although, llama.cpp has had a lot of great improvements in the past quarter that could potentially displace vLLM for single user use.
It also seems to me that Meta did actually consider (tuned for) local users with 24GB of VRAM this time since this model is targetting that. To me that's kind of unprecedented because in general we get these models meant for DCs, and then the community has to figure out how to run it on "home" hardware.
That's why I'm also not holding my breath on Qwen3.8-27B because they could've tuned it for 24GB even in 3.5 and 3.6. It's not like they have some regulation over there ("..someone think about the children") preventing them from making models for local use.
I don't use llama.cpp either. I only use it when I need to offload the model to RAM or when vLLM doesn't support the model. At least for my use case, it's worse in many way except RAM offloading. It does have far better community support too, and it's the better option if you need to offload part of the model to RAM, but once you've experienced vLLM's speed it's really hard to switch back.
I agree that Meta built this one specifically for 24GB, and to be fair, that's not something Qwen has done well so far. 3.6 overthinks and its KV cache is heavy, which is exactly what hurts people at 24GB.
Where I'd push back a little is on reading that as a deliberate choice. Qwen has kept a ~30B dense slot in every generation, and they ship day-one support in llama.cpp and vLLM. Llama.cpp is not for DC customers, and neither are ~30B dense models. A DC operator would use a large MoE, which is by far the better option for them. So the intent to serve single-GPU users looks real to me.
What's been missing is the efficiency work, as it uses much more KV cache per token. It used to be pretty good next to the alternatives, but now we have Deepseek-V4-Flash and this very model, which uses far less VRAM for context. Given the fact that coding agents use a lot of context, I'd say the Qwen-Next architecture is showing its age. It's still better than Gemma-4 though.
I think Meta is realizing that they are not going to lead the race again (like they used to with llama) so being a champion of the people is better than nothing at all. Google could do it as well but neither companies have the mentality or the corporate etiquette needed. And so they continually get put to shame by teams and companies from the Far East, rightfully so.
I did not even look into DS-V4-Flash - a mention of DS to me immediately rings like 150GB VRAM needed. 😃