Memory bandwidth saturation, KV-cache thermal throttling, and O(1) sequence scaling on edge devices
Hi Fantominsight team,
Quantizing a 3.62B Mistral-based model into Q8_0 for localized roleplay, multi-language processing, and agentic tasks is a great practical release. Being upfront about performance metrics (5-8 tok/s at 29 GB/s memory bandwidth) and realistic mobile limitations is very helpful for end users.
Looking at your runtime behavior and the thermal constraints on mobile hardware:
The memory bandwidth ceiling on 29 GB/s pipelines:
At 3.65 GB for the Q8_0 weights, the theoretical throughput ceiling on a 29 GB/s memory bus is roughly 29 / 3.65 = ~7.9 tokens/second during autoregressive decoding.
Your measured 5-8 tok/s means the engine is running at near 100% bus utilization on every generated token. On mobile SoCs (like Snapdragon 888 or 8 Gen 1 in Xiaomi 11 and 12 series), pegging the memory controller continuously draws substantial power and generates rapid heat buildup, which triggers hardware thermal throttling and drops clock speeds within minutes.KV-cache memory pressure across 16k contexts:
While the model supports up to 16k tokens, standard attention caches grow substantially as context accumulates during long roleplay sessions.
On devices with 6-8 GB total system RAM, combining the 3.65 GB base weights with OS background memory and a multi-hundred megabyte FP16 KV-cache puts the device right on the brink of Android low-memory killer (LMK) eviction. Furthermore, attending across thousands of past key-value vectors increases computational overhead on every decode step.Constant O(1) state generation in hybrid architectures:
In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we address long-context mobile execution using a 3:1 macro-topology (DGDA linear recurrence combined with MABA-SA latent sparse attention):
By updating 75% of the network via recurrent state transitions, decode latency remains strictly flat at O(1) regardless of sequence length, and the KV-cache footprint is compressed by nearly 40x.
For long multi-turn roleplay on edge devices, this prevents the memory expansion and computational creep that typically cause thermal throttling as conversations stretch past 4k or 8k tokens.
If you are exploring ways to improve efficiency on memory-constrained devices, taking a look at the hybrid scaling specifications in the Maba v2 repo might provide some useful architectural ideas.
Did you benchmark mid-range quants like Q4_K_M or IQ4_XS to ease bus saturation on 29 GB/s memory hardware?
Best,
Andrew
Hello, Andrew!
We have taken your advice into account. (We can may make mistakes in English).
We are very grateful to you for this.
About quant (Q4_K or more) - no, but we want).
Thanks. Best regards,
Fantominsight!
Hi Fantominsight team,
Your English is completely clear, no worries at all!
Exporting a Q4_K_M quant will definitely be a massive upgrade for mobile users. Here is what you can expect on that 29 GB/s bus:
Weight footprint drops from ~3.65 GB down to ~2.1 GB.
Memory bus ceiling immediately jumps from ~7.9 tok/s up to ~13-14 tok/s.
Drastically reduced memory controller saturation, which prevents the rapid thermal throttling and battery drain on Snapdragon chips.
Leaves plenty of headroom in Android RAM, avoiding out-of-memory app crashes during longer roleplay sessions.
Looking forward to testing the Q4_K_M release when you drop it! Keep up the great work.
Best,
Andrew