Can see, can touch but can’t use
Thank Qwen for your open models.
But 180B is too large for most people.
Hope that you guys will release smaller models, dense and Moe.
Thank so much.
But 180B is too large for most people.
Quantization is meant for that, but we will probably need to wait a few more days until the inference engines work well with it. A6B should run really fast though.
But 180B is too large for most people.
Quantization is meant for that, but we will probably need to wait a few more days until the inference engines work well with it. A6B should run really fast though.
Even at q4 you need more than 64 GB RAM. So OP is still right. Most systems have 16- 32 GB RAM and 4-8 GB VRAM. I hope we will see a 35B too on this new architecture, which can be run on consumer hardware.
50GB is max i can go. waiting for quants :)
comment sent twice sorry.
Even at q4 you need more than 64 GB RAM. So OP is still right. Most systems have 16- 32 GB RAM and 4-8 GB VRAM. I hope we will see a 35B too on this new architecture, which can be run on consumer hardware.
Theres always openrouter, runpod or other providers if hardware is too expensive, this model should be really cheap to run in the cloud. But yeah, i hope the models continue to become even more efficient in the future or the hardware prices cheaper.
51B n-gram embedding lives in CPU RAM, not VRAM. Check PLE offloading.
51B n-gram embedding lives in CPU RAM, not VRAM. Check PLE offloading.
Doesn't change a thing. 120B parameters still have to load in RAM, and not everyone has over 64 GB of RAM.
@DampfinchenPeople are currently working to get the ngrams to be streamed from nvme. If the idea takes off you will probably see moes like "6a 25b 200bngram" or something like that. But dont get your hopes up.
Qwen3-Coder-next is 80B and my 128GB of Unified RAM can run this one.
That's all they need to do. Between 50B and 80B.
For local usage with a single request, a 128B model is fine. But for serving a small number of users, a 35B-A3B model is much better. That is reason why many people needs a 35B A3B Moe model.