What's the point of this model?!
I love this 4B model and it's smart as hell, but its architecture doesn't give a benefit over using a decent 9B model! The VRAM savings aren't justified to use it; that's the main reason we use 4B models! For examples: a Q4 version with Q4 unified kv cache tops out at 64k context. I'd get more out of using qwen3.5-9b (no vision) or ornith-1.5-9b than doing that. Tell me if I'm wrong or I'm missing something.
not everyone may have as much vram as you, because you find more use for a 9b model that good for you, mobile local is also a thing so there is your answer
not everyone may have as much vram as you, because you find more use for a 9b model that good for you, mobile local is also a thing so there is your answer
that's a very weird answer, they're literally saying for the same amount of VRAM you can get more context out of a 9B model than this model, what's the point of this model having a 1M context size, when it uses 13GB of VRAM for the context KV cache (at Q8)?
I just feel like there's no point in using this model on a mobile device either because you can't run it with large context, there's better models for mobile, also on mobile the models don't run on VRAM, so you can actually run the bigger models, they would just be slow.
Its a trade, compute vs context length most android models are capped at 4096 and 8192 during conversion to mobile formats like LiteRT sadly so it comes down to compute at that point rather than just context length hence why 4b models are favored atm
Exactly V100. JNK33 is not understanding what we have said. JNK, read this: this 4B model uses MORE vram than any 4b model Ive seen because it's a context pig--more than most 9B models even. Yeah the initial load of this model is like a typical 4B but you can't use it on a phone because the context usage goes thru the roof immediately!
XHToken: My apologies. the full attention 1 of 4 layers was not being handled correctly by llama.cpp and causing a MASSIVE context spike. I am now successfully using context within the size ranges i would expect. I blamed you because this was the only model it was happening with, but it was because of llama.cpp.
Hi @NezTheNaughty , thanks for the update! We set up a fresh environment to investigate the reported memory usage.
We tested both upstream llama.cpp b10828 and the XHToken fork at 4a3635c, on an A100 40GB, using the official Q4_K_M GGUF, q4_0 for both K and V caches, Flash Attention enabled, a single sequence, and the default sliding-window cache behavior.
In both builds, at configured context sizes of 64K / 128K, KV cache allocation was approximately 678 / 1,326 MiB, and process GPU memory peaked at approximately 3.91 / 4.85 GiB, respectively. We also successfully processed an actual 60K-token prompt with the upstream build, without observing an abnormal memory spike.
So far, we haven’t reproduced the reported excessive memory usage under these settings. This doesn’t rule out an issue with an earlier build or a different configuration.
If the issue persists or recurs, could you share your hardware/OS, exact llama.cpp version or commit, any frontend used, GGUF source, launch command/settings, and startup logs showing KV cache allocation?
Since you mentioned it is now working, could you also share what changed between the failing and working setups? That would help us identify the affected version or configuration.
I believe I was using on llama.cpp v2.37 when I mentioned it wasn't working very well. Then updated to the llama.cpp release on 9/17 (v2.40?) and tested it and reported things seemed ok. I remember seeing someone else's diagnosis (from the llama.cpp community?) reporting that the full attention was being re-read on every prompt, causing that earlier problem with the memory spike, which seemed like what I was experiencing. That was solely the source of my frustration. I would top out at around 24 or 25k context which is why I complained, but like I said it must have been a llama.cpp problem because the llama.cpp update fixed the spike. You guys seem really on top of things, so my hats off to you!