TINY BUT MIGHTY: A SOFTWARE-HARDWARE CO- DESIGN APPROACH FOR EFFICIENT MULTIMODAL IN- FERENCE ON BATTERY-POWERED SMALL DEVICES
Authors: Yilong Li
Yijing Zeng
Shuai Zhang
Hao Zhang
Xinmiao Xiong
Jingyu Liu
Pan Hu
Suman Banerjee
Venue: ICLR 2026 Poster (2026)
Keywords: On-device VLM
Efficient Inference
Software-Hardware Co-Design
Quantization
NPU
GPU
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: The smallest battery-powered device that can run VLMs in the world

Abstract: Large Multimodal Models (LMMs) are inherently modular, consisting of vision
and audio encoders, projectors, and large language models. Yet, they are almost
always executed monolithically, which underutilizes the heterogeneous accelera-
tors (NPUs, GPUs, DSPs) in modern SoCs and leads to high end-to-end latency.
In this paper, we present NANOMIND, a hardware–software co-design inference
framework for Large Multimodal Models (LMMs) that breaks large models into
modular “bricks” (vision, language, audio, etc.) and maps each to its ideal accelera-
tor. The key insight is that large models can be broken into modular components and
scheduled to run on the most appropriate compute units. It performs module-level
dynamic offloading across accelerators on unified-memory SoCs. By combining
customized hardware design, system-level scheduling, and optimized low-bit com-
putation kernels, we demonstrate our framework with a compact, battery-powered
device capable of running LMMs entirely on-device. This prototype functions as
a self-contained intelligent assistant that requires no network connectivity, while
achieving higher throughput and superior power efficiency under strict resource
constraints. The design further bypasses CPU bottlenecks and reduces redundant
memory usage through token-aware buffer management and module-level coordi-
nation. Our system outperforms existing implementations in resource efficiency,
cutting energy consumption by 42.3% and GPU memory usage by 11.2%. This
enables a battery-powered device to run LlaVA-OneVision-qwen2-05B with a
camera for nearly 20.8 hours.

Published as a conference paper at ICLR 2026
TINY BUT MIGHTY: A SOFTWARE-HARDWARE CO- DESIGN APPROACH FOR EFFICIENT MULTIMODAL IN- FERENCE ON BATTERY-POWERT
SMALL DEVICES
Yilong Lit, Shuai Zhang", Yijing Zeng'. Chengpo Yan', Hao Zhang', Xinmiao Xiong*.
Jingyu Liu Pan Hu Suman Banerjee+
Universsiyy of Wisconsin Madison, AAmazon Web Services AI, USA, Uber, USA
ABSTRACT
Large Multimodal Models (LMMs) are inherently modular, consisting of vision and audio encoders, projectors, and large language models. Yet, they are almost always executed monolithically, which underutilizes the heterogeneous accelera- tors (NPUs, GPUs, DSPs) in modern SoCs and leads high end-to-end latency. In this paper, we present NANOMIND, hardware-software co-design inference framework for Large Multimodal Models (LMMs) that breaks large models into modular "bricks" (vision, language, audio, etc.) and maps each to its ideal accelera- tor. The key insight is that large models can be broken into modular components and scheduled to run the most appropriate compute units. It performs module-level dynamic offloading across accelerators on unified-memory SoCs. By combining customized hardware design, system-leve scheduling, and optimized low-bit com- putation kernels, we demonstrate our framework with a compact, battery-powerts device capable of running LMMS entirely on-device. This prototype functions a self-contained intelligent assistant that requires no network connectivity, while achieving higher throughput and superior power efficiency under strict resource constraints. The design further bypasses CPU bottlenecks and reduces redundant memory usage through token-aware buffer management and module-level coordi- nation. Our system outperforms existing implementations in resource efficiency, cutting energy consumption by 42.3% and GPU memory usage by 11.2%. This enables poweeed device to run LlaVA-One WiioOniiiinno-nn22-555555 with a camera for nearly 20.8 hours.
INTRODUCTION
Large language models (LLMs), such as GPT-4/5 (OpenAl
2024
2025). Claude (Anthropic 2023)
and Gemini (Comanici et al. 2025). have shown exceptional proficiency in knowledge acquisition and application. Mcanwhile, Large Multimodal Models (LMMs) (Dubey et al. 2024 Liu et al.) 2023a
2024a Anthropic 2023 Bai al. 2023 Marafioti et al. 2025) have transformed various applications, including visual understanding and cross-modal reasoning, enabling more advanced Al-driven interactions. Running large multimodal models (LMMs/VLMM)) locally on edge devices is becoming increasingly important, as cloud-based deployment poses significant privacy pisks-personal data may be exposed misused in ways that difficult control, explored in prior studies (Kim et al. 2023 Hui et al., 2024). On-device LLMs enhance security by keeping data local and minimizing breach risks while enabling real-time intelligence and user privacy. Still, their practicality is limited by the tight power and compute budgets of compact systems. As demand for advanced models grows- erpeealsyy in offline or low-connectivity scenarios-ve we need solutions that balance resource efficiency with privacy. Deploying these models smartphones, desktops, and robots is increasingly common, enabling natural- natural-language interactions, real-time task execution, and stronger user privacy.
Significant efforts have been made to enable on-device AI, including the development of compact, parameter-efficien models like SmolLM (Allal et al.] 2024) and SmolVLM (Marafioti et al. 2025), Cemma-3-1B (Gemma Team 2025). and Phi-3 (Abdin et al. 2004).. advanced quantization techniques such as AWQ (Lin et al. 2024) and BitNet (Ma et al. (024), and deployment frameworks like llama.cpp (Gerganov 2023a) and MLC L.LM (MLC team 2023a) However, these approaches focus

---

Published as a conference paper at ICLR 2026
almost entirely software- or algorithm-leve' optimizations chiiiiattionsccceffyyyyll low-bit quantization and lack support for the fragmented diversity of mobile GPUs and emerging NPUs, nor do they adapt well across different hardware platforms. Most prior works also try to solve just one two aspects of the problem, but there is still no end-to-end solution. In particular, they often overlook the joint design of software and hardware. As a result, devices cannot fully use their available resources, and power consumption is rarely considered.
Modern LMMs integrate vision, text, and audio information. Although Vision-Language Models (VLMs) are typically trained as single unified models, their internal components are relatively independent, and many of them are fine-tuned separately rather than end-to-end. These loosely coupled components can be decoupled and executed independently, allowing each to run on the most suitable hardware. On edge and mobile devices, however, mapping the entire model onto one accelerator-Gii GPU, NPU, or use a unified memory architecture (UMA) with heterogeneous accelerators (NPU/GPU/DSP)), while common deployments still treat the model as monolith. Existing inference frameworks undermine overall inference efficiency on edge or small devices.
resources and increases latency. Yet today's s edge SoCs
Vision Encoder (o.g. SigLip) 8-bit
Unified Memory
LLM (e.g. Qwen2-0.58) W4A16
Leading ing buller
Onteedings BB M
Eneadeev
Ring Crpr
Terpo Peatre
Wi
Liver Aariter
Output
Seent coppert Ga/
"The piicn dhowe a tully white light ogange
Corrtel Ldbd
Video Frames
Mer 11ABP
Vaioo Encoder Offoadiigg
CPU Rontirw mannaged
Fe Inage?
Offinsing
FRNN
NPU
CPU
E
+
GPU
Witppeopp
Speech-to-Text (e.g. Whisper)
Schockiiig
FP16
Boheddalo
Scheckte
Workflow of NANOMIND: VLM Offloading to NPU/GPU with Zero-Copy Embedding Transfer via Ring Buffer.
Figure 1:
A key motivation for our work stems from two critical observations: First, LMMs are inherently mod- ular, often composed of distinct components such as vision encoders, embedding layers, a projector, and language decoders, each with unique computational characteristics. Second, different accelerators are designed with distinct strengths for example, NPUs outperform low-bit tensor operations (e.g.. INT4/INT8) but are inefficient for floating- ppint workloads due to high overhead, while GPUs are far better at large-scale parallel floating-poin computations. However, LMMs are often deployed as monolithic workloads match leads to underutilized hardware, increased latency, and inefficient inference. Without the ability to dynamically offload different components to the most suitable compute units, valuable resources remain idle. As we observed in our experiments (Sec. [ the NPU outperforms the GPU and CPU for vision-encoder inference, highlighting the importance of dynamic, module-level offloading. Finally, although many frameworks now support deploying LLMs on edge devices, most are adapted from server traditional PC architectures, where CPUs and GPUs operate with separate memory spaces. In contrast, modern edge devices-includiing including mobile phones use a unified memory architecture, where the CPU and GPU (or NPU) share the same physical DRAM. This fundamental difference makes many legacy designs inefficient when applied directly. Under unified memory, accelerators like the NPU and GPU lack DMA isolation and must coordinate access shared memory, requiring new system-level optimizations and careful redesign to ensure efficient operation.
a single accelerator, regardless of these architectural differences. This mis-
Existing approaches primarily focus on software-level techniques- as low-bit quantization and model scaling-to reduce memory usage. However, they often overlook essential hardware-level optimizations, including driver support for low-bit operations on mobile GPUs and NPUs, efficient power management, and enhanced cross-accelerator utilization. Additionally, naively deploying the entire model on a single accelerator frequently leads to high latency. As a result, these frameworks fail to fully exploit the limited compute resources available on edge and small-form-fatt factor devices.
2

---

Published as a conference paper at ICLR 2026
To overcome these challenges, introduce NANOMIND, an end-to-end on-device inference frame- work that decomposes large multimodal models into modular, independently executable components and dynamically offloads each to its optimal compute MPU- NPU, CPU. NANOMIND is built through a tightly integrated software-hardwar co-design. We demonstrate it using a custom battery- powered prototype device (Figure[ With this hardware platform and system-level implementation, NANOMIND outperforms mainstream frameworks running on commodity off-the-shell devices. Our contribution lies in a SW/HW co-design approach at the seerence-system level, where we develop a series of system and software optimizations rather than modifying model algorithms.
We also design an event driven On-Demand Cascade Inference Pipeline as shown in Fig. Only the minimal output needed suchh as text string or an embedding vector is retained and passed to the next stage. This results in lightweight, domino-like chain of execution.
Wert de
Whisper
Vision Encoder
Embeddings
Qwen2-0.5B-W4A16
Tox Prorge
Output
Mattmace
Altriinn
"The Bully
some light
Poad
vteddings
Load
1. Load
Release
1. Load
3. Palesse
NPU
NPU
CPU
GPU
TITl
T1TT
Exeouate
Exeovate
Execuate
Figure 2: Workflow of Low-Power On-Demand Cascade inference. Each modular model follows "load + execute + release" workflow: once it completes the inference and releases the hardware resources immediately.
As shown in Figure 3 and Figure 2 our framework enables efficient vision and voice inference on resource-constraiied hardware. To achieve this, we designed custom hardware, implemented system-level optimizations, and developed drivers and computation kernels for the built-in GPU and NPUs of a low-end SoC. Our key contributions are summarized as follows:
Cross-accelerator scheduling for modular VLMs. We decompose models into vision, fusion, and decoding modules and schedule each to the most suitable accelerator under UMA, improving utilization and end-to-end latency.
Custom Hardware-Software Co-Design. On the hardware side, we use a commodity RK3566 SoC with integrated GPU and NPU, improve effective memory bandwidth uti- lization with in-parallel LPDDR4X modules, and add a dedicated power management unit (PMU) for real-time energy monitoring. On the software side, we implement custom 2-bit, 4-bit, and 8-bit GEMM kernels tailored to our hardware, along with an offloading scheduler and drivers to accelerate quantized tensor operations on both GPU and NPU.
Dynamic workload Offloading. A lightweight ring buffer and buffer manager enable zero-copy token exchanges shared memory. Our layer-aware offloader makes per-layer decisions- on battery level, memory usage, and latency needs bypassing the CPU for buffer writes.
Battery-aware execution modes. Lightweight policies adapt placement and memory clocks to extend runtime under power constraints while preserving responsiveness.
By employing these efforts, tiny device can efficiently operate LLMs and LMMs (LlaVA Liu et al.) (2023b]a) a). Qwen2-VL series Bai et al.) (2023); Wang et al. (2024b) within constrained hard ware resources by directly offloading workloads to the on-device accelerators based on power and memory usage. This approach enhances inference performance and significantly reduces power consumption. Although our hardware prototype is implemented on RK3566 on RK3588 (Orange Pi 5)-our framework is not tied to any specific SoC. Modern edge and mobile SoCs (e.g., Apple Silicon, Qualcomm 6590, recent RK and MediaTek chips) typically integrate multiple accelerators, often including both an NPU and GPU, and sometimes DSP, making cross- accelerator execution directly applicable. Our work establishes a practical framework for efficient I.M deployment under tight power and memory constraints, enabling responsive and energy-efficien
we also tested key components
3

---

Published as a conference paper at ICLR 2026
multimodal inference soall-form-factii devices. This demonstrates the viability of running LMMs directly on edge hardware without cloud dependence.
RELATED WORK
Efforts to make large model inference more efficient edge, mobile, or small devices generally
fall into two directions: system-level optimizations to improve execution, and model compression
techniques. NANOMIND builds upon and is inspired by prior research and open-source efforts in
guantization Lin et al., 2024, Frantar et al,, 2022 Yang et al. 2024 Wang ct al. 2024a Dettmers
Zettleoyerr 2023 and efficient inference frameworks (Wei et al. 2024 Gerganoy 2023a MLC
team 2023a)
2.1 QUANTIZATION
Quantization reduces the bit-precision of models, which helps to reduce the model size and accelerate inference (Han et al. 2016).
Post-Training Quantization Post-training quantization (PTQ) compresses LLMs after training to produce smaller, inference-optimizee models, improving efficiency for storage and computation mobile and edge devices. Group-wise quantization Yang et al. (2024) divides weights into groups and quantizes each separately, while GGUF (ggml) in llama.cpp block-based method with per-sub-block scales and offsets. GPTQ Frantar et al. (2022) further reduces memory by compressing weights 3-4 bits. Activation-aware Weight Quantization (AWQ) Lin et al. (2024) preserves accuracy by identifying and retaining weights with high activation magnitudes. BitNet bl.38 (Ma et al. 2024 demonstrates a promising direction for reducing LLM inference costs with 1-bit quantization. Building on this, BitNet a4.8 (Wang et al. (0024)) introduces 4-bit activations and leverages hybrid quantization together with sparsification to further improve efficiency.
K-quant, block- and sub-
2.2 ON-DEVICE INFERENCE SYSTEMS AND FRAMEWORKS
Inference System In system-level optimization, recent work has leveraged heterogeneous accelerators in modern SoCs. For instance, Ilm.npu (Xu et al. 2025) restructures execution at the prompt, tensor, and block levels on NPUs, while offloading outliers and FP operations to CPU/GPU and reordering subgraphs to improve utilization uiilizationaaddresse aadressing the limitation that mobile NPUs only support static input shapes. PowerInfer-2 (Xue et al., (0244) proposes NPU-CPU collaborative framework that offloads LLM inference based on neuron activation density, enabling models larger than the device's memory to run on smartphones.
Open-source Frameworks MLC LLM (MLC team 2023a b uses TVM (Chen et al. 2018) to deploy LLMS natively on mobile and edge devices. However, TVM's heavy resource requirements make it impractical for routine on-device inference on small platforms, and it falls short in power and memory efficiency. llama.cpp (Gerganov) 2023a), developed by Georgi Gerganov with C++, lightweight and portable L.M inference frameworks. It supports multiple backends, including Vulkan, OpenCI., and CUDA, but struggles with efficiency on many mobile and edge GPUs. Our experiments show that on specific platforms, it often defaults CPU offloading and is even slower on GPU, limiting performance gains, as indicated in Tab []] Many existing inference frameworks are using llama.cpp as their backends, like LlamaEdge (LlamaEdge 2024) and Ollama Gross (2023).
Inefficiencies in laama.pp While llama.cpp provides layer-wise offloading, its workload distribution inefficient on small devices, particularly under unified memory. Although computation can be split between the CPU and GPU, GPU execution still depends on CPU-managed data transfers, increasing memory overhead during inference. Figure 10 in the Appendix illustrates this workflow, with further discussion in Section I. When a GPU is available, tensors can be assigned the GGML BACKEND GPU flag, allowing
() to offload computations to the GPU. This involves transferring key tensors from CPU memory, while the CPU must continuously write to buffers and maintain separate memory allocation, leading to additional overhead. This type of framework enables LLM deployment on edge devices but follows server-side designs with separate CPU and GPU memory. In contrast, modern edge devices use unified memory, where CPUs, GPUs, and NPUs share the same DRAM.
4

---

Published as a conference paper at ICLR 2026
Models
Layers GPU
CPU Usage
Memory Usage
GPU Usage
56%
2.9GB
0%
Llama-3-8B (2-bit)
10
38%
4.1GB
50%
38%
5.5GB
91%
50%
534MB
TinyL.lama-I (4-bit)
10
37%
734MB
75%
30
37%
818MB
99%
50%
80IMB
0%
Llama-3.2-3B (4-bit)
10
38%
1031MB
72%
30
38%
1091MB
99%
Table Resource utilization (CPU, GPU, and memory) when offloading model layers to the GPU in llama.cpp to illustrate it. Offloading more layers substantially increases memory usage relative to CPU-only inference.
DESIGN
In this section, we present the design of NANOMIND through "top-down approach", beginning with model decomposition and extending through noftwre-hhrdwwrd coordination, and hardware architecture- enabling efficient inference on heterogeneous SoCs. NANOMIND offloads vision encoding to the NPU and LLM decoding the GPU, employs custom Token-Aware Buffer Manager (TABM) for zero-copy data transfer, and uses a lightweight CPU scheduler that dynamically switches between performance and power-saving modes. Together, these components form a unified hardware- software Sardware-softwae co-design that optimizes inference under tight memory and power constraints.
Chamo Aeyrendk D'fasleg
Lange Lingage
Oedivesad Cardk Paclis
Falayy Schcklliy
D
Tex
Unitnize
Cacor
Speaisss Text
TTS
Eorel
liee Rendl
Frear Hasppooee
Qpead Whige
Wike Oape
ML
Puditi R200 Lrooote Maruo
(a) SW/HW Architecture
(b) Multimodal Inference
Figure 3: Architecture of NANOMIND: Enable Multimodal Inference via Hardware (SW/HW) Co-design.
3.1 MODEL.
We start with model decomposition. Because LMMs are inherently modular, we configure their components to run independently different accelerators, shown in Figure Figure We decomposed and converted several models for efficient on-device inference. Speech-to-text handled by a standalone Whisper-base model (Radford et al., (223) implemented with Whisper.cpp (Gerganov 2023b), while text-to-speech is provided by Piper (Rhasspy 2025). a lightweight C++ program that runs the CPU, both independently of the VLM. For vision, extract the encoder from VLMs such as VAaVOne LLaVA-OneVVision--Qaaaaa Vision-Qwenn-O..... al. 2024b), both of which adopt SigLip (Zhai al. 2023) their vision encoder. The SigLip encoder can be converted into the RKNN format using Rockchip's official toolkit (Linux 2025).
(Liu et al. 2024a) and Qwen2-VI. (Bai et al. 2023 Wang
enabling efficient deployment on NPUs. Following the LLa VA-One Vision architecture, we obtained the original weights in safetensors format from Hugging Face (Li et al. 2024 HF (025) and extracted the vision encoder with its projector, the multimodal embedding layer, and the Owen-2.0-0.55 base model.
3.2 SOFTWARE-HARDWARY COORDINATION
Here describe the system-level optimizations that adapt the modular components of LMMs, highlighting the inference backends across NPU and GPU accelerators, hybrid quantization, token- aware buffer management for zero-copy data transfer, and power-efficiency strategies. While the
5

---

Published as a conference paper at ICLR 2026
PMU
ROME RoC
Mamorry
(b) Front of PCB Design
US92 OTG
RK3566 SoC
WUMIC
Meroery
(a) Block diagram of hardware design.
(c) Back of PCB Design
Figure 4: NANOMIND hardware design and PCB layout. (a) Block diagram of hardware components: an RK3566 SoC, a PMU IC for power monitoring, and LPDDR4x memory modules in parallel: (b) front view of PCB design; (c) back view of PCB design.
deployment strategy is designed for our custom SoC, the framework remains flexible and can be applied to other mobile SoCs with different offloading policies.
NPU Most mobile NPUs only support static input shape, meaning that any change in input shape requires recompiling the firmware- impractical step for resource-constraine devices. To address this limitation, we offload the vision encoder to the NPU and pre-process all images by compressing and resizing them to fixed resolution, ensuring consistent input shapes during inference. Rockchip's RKNN driver (Linux 2025) and Qualcomm' 's AI Hub (Qualcomm) 2025) provide native support for models such CLIP (Radford al., 202I) and SigLip (Zhai et al. 2023). offering higher performance than open-source implementations. We deploy ViT (SigLip) on the NPU rather than the GPU for performance reasons: the official RKNN driver provides significantly more efficient execution environment. Inference of the vision encoder is much faster on Rn(kocchsspss NPU. In contrast, running the LLM on the NPU is impractical due its static-shape vary at runtime. Therefore, we offload only the Vision Encoder/ViT to the NPU. All input images are pre-processed to a fixed resolution of 448 736 (Qwen2-Vi) VL) or 384 x 384 (Llava-One Vision), and multi-frame inputs are merged using simple average temporal pooling operation. In addition to the official SDK, insights from community resources- such as technical blogs and forums Devices -eee instrumental in navigating RKNN conversion and optimizing operator mappings, helping us maximize NPU efficiency.
pqquirement-promyy lengths
GPU Our inference kernel GPU builds llama.cpp, retaining the ggml (GGUF) model format while extending with customized backend to support heterogeneous edge accelerators. Using GGUF unified format allows NANOMIND leverage wide range of open-source quantized models. To further improve efficiency on resource-cone constraind devices, we incorporate OpenCL- based GPU kernels enhanced with linear attention and fused dequant-GEMM operations for W4A16 quantization (4-bit weights, FP16 activations). To improve efficiency on memory-constraine GPUs, we replace standard quadratic attention with a kernelized linear attention variant that eliminates the explicit 7'x7' attention matrix construction. This reduces memory traffic and stabilizes decoding latency for longer sequences. In our evaluation, this modification introduces no statistically significant accuracy degradation across the tested benchmarks. We also implement a fused dequant-GE.MI OpenCL kernel that unpacks and rescales int4 weights in-register within the GEMM loop, followed immediately by FP16 FMAs. This fusion eliminates intermediate buffers and memory passes, turning each byte into useful MACs -articaatiylllrly beneficial on mobile GPUs lacking efficient low-bit tensor cores. The kernel uses tiled vector loads, scale tables in constant/LDS memory, and epilogue that can fuse bias and activation, with FP 16/FP32 accumulators for stability. Together, these optimizations reduce memory traffic and latency while preserving accuracy.
mobile
Quantization Model compression is essential for on-device LLM inference due to hardware con- straints. NANOMIND supports various quantization for both GPU and NPU bit packages, includ- ing 4-bit (GPTQ 4-bit (Frantar et al.] 2022). BitNet 1-bit (Ma et al.) 2024, Wang et al.] (2024)), ggml (GGUF) 2-bit/3-bit/4-bit (Gerganov 2023a)) conventional implementation. By decomposing

---

Published as a conference paper at ICLR 2026
LMMs into modular components, we can apply hybrid quantization using different quantizations for the vision encoder ( (ViT) and the base model (LLM). In our setup, SigLip vision encoders are deployed the NPU in RKNN format with FP16 or 8-bit precision, while GGUF-quamtized LLMs run on the GPU with 4-bit (W4A16) or lower-bit (2/3-bit) quantization. Higher precision in the vision encoder enhances image understanding, whereas 4-bit LLMS are sufficient for wearable and edge devices, where complex reasoning tasks are less common. Recent work confirms that 4-bit quantization offers the best balance between memory efficiency and accuracy Li al. (2025). Mobile GPUs rarely have fast INT8 tensor cores. Use weight-only quantization (INT8/INT4 weights, FP16 activations) with a fused EequanttGEMM, OpenCL kernel kerneluuppac and rescale in registers, then multiply. Avoid separate dequant passes to cut memory traffic and keep the pipeline saturated.
Power-efficiency Strategy NANOMIND leverages a dynamic, three-state power management strategy driven by real-time data from the on-board Power Management Unit (PMU). By monitoring the device's battery level
(B), this policy intelligently arbitrates the trade-off between performance and longevity. (i) Unconstrained Performance State (B' > Thigh): The system operates at full capacity, aggressively offloading workloads in parallel to accelerators. (ii) Proportional Throttling State (Tiw B S Thigh): The system enters state of graceful degradation, using scaling factor a= (B - (TTigh - Tlow) to linearly interpolate camera frame rate and memory read/write rate. (iii) Critical Conservation State (B < Tiow): To ensure mission-critical functionality, the system activates the On-Demand Cascade Inference model, suspending parallel execution in favor of a power-optimized, sequential workflow.
Low-Power On-Demand Cascade Inference In critical low-battery situations, the system switches to an event-triggered mode called "On-Demand Cascade Inference" designed to minimize peak memory usage and power consumption. In this "one-time inference" mode, the system remains in ultra-low- lora-llowppwwrr standby, with a single CPU core waiting for camera or microphone events. For example, the camera captures only single frame (disabling temporal pooling), and all accelerators operate once per trigger. When triggered by an event such as a wake word, the system runs a sequential inference pipeline. Each lifecycle: it loaded, performs its task, then is released, passing only the minimal output (e.g.. text or embeddings) to the next stage. This forms a lightweight, domino-like cascade that reduces memory and power usage, avoiding heavy memory usage and CPU waiting.
Whisper, ViT, or LLM
"load -> execute -> release"
Embeddings Zero-Copy Transfer in Unified Memory To support efficient token flow and zero-copy transfer across accelerators, NANOMIND introduces the Token-Aware Buffer Manager (TABM) a lightweight CPU runtime and the core of dynamic workload offloading (Figure 3). TABM manages a shared ring buffer pool in unified DRAM, coordinating tokens between the NPU (producer) and GPU (consumer) without redundant memory movement or blocking. It tracks buffer states (FREE, ALLOCATED
_RITE, READY __TTT_O____ RO_RED, ALLOCATED COCCATED_FOR... (OR_READ)) and signals availability via lightweight synchronization. The NPU encoder writes embeddings directly into buffer slot, which the GPU can immediately bind as LLM input, avoiding copies. This design reduces CPU load, lowers latency, smooths podducer-consumee mismatches, and sustains a high- thtouhhtuughput token pipeline.
3.3 HARDWARE DESIGN
To enable modular model components offloading and achieve better coordination across the accel- crators at the system level, we designed specialized hardware. The PCB design was adapted and modified from several open-source references to ensure compatibility with mainstream I/O interfaces. As illustrated in Figure the design is optimized for efficient on-device LLM inference. The built hardware demo is shown in Figure 11
RK3566 SoC: We adopt the RK3566 RRokchhpp a cost-effective and power-efficient SoC from Rockchip. It features quad-core Arm Cortex-A55 (up to 1.6GHz), an integrated NPU, Mali G52-2EE GPU, and external DDR support. With price point under $ $12, the RK3566 provides all core functionalities required for building compact device capable of local LLM inference.
To address the memory-bound nature of LLM workloads on small- form-factor devices, we enhance the effective bandwidth utilization of RK3566's LPDDR4x sub- system. Although the chip uses four DDR schedulers that multiplex into single bi- controller, our coordinated EPU--PP-NPO buffer management reduces contention and redundant transfers, improving overall memory efficiency for L.LM inference.
Parallel LPDDR4x Memory:
7

---

Published as a conference paper at ICLR 2026
Interfaces: To minimize power consumption and simplify the system, remove unnecessary componentss such as HDMI, Wi-Fi/Bluetooth. Instead, we use USB-OTG to support an audio jack hub for speaker and microphone input, enabling voice interaction. A MIPI CSI interface supports image capture from low-power camera. Available interfaces are shown in Figure 11 in the Appendix.
Power Management Unit (PMU): Unlike traditional mobile and edge platforms, our system includes a dedicated PMU for real-time energy monitoring and control for our power efficiency strategy.
EXPERIMENTS
In this section, we present the experimental evaluation of NANOMIND. Unlike the Design section (Section which followed a "top-down perspective", here we adopt reverse "bottom-uu approach" along three dimensions: (1) profiling resource usage across different platforms, (2)model accuracy across different offloading strategies, and (3) measuring power efficiency under different runtime conditions.
4.1 RESOURCE USAGE
In this section, we evaluate resource efficiency in VLM inference, focusing on response latency, hard- ware utilization (CPU, GPU, and memory), and energy efficiency, with an emphasis on multimodal task performance. We use datasets including InfoVQA (Mathew al. 2022). DoCVQA (Mathew jet al., 2021). MMBench (Liu et al. 2024b), and MME (Fu et al. (024). Details the measure- ment methodology and datasets covering memory usage and power efficiency-
provided in Section A.3 in the Appendix due to space limitations. We compare memory usage across several small-scale VLMs, including LLa -LiiVA--neiision0...558 (HF) 2025), Qwen2-VL-2BB (Wang al., 20245). and SmolVLM-55000 Orange Pi 5 Ultra (P.). and Nvidia Jetson Nano/ AGX, with Jetson AGX serving as upper-bound reference due to its higher performance. As shown in Figure 5) Ilama.cpp consistently consumes more memory across all platforms, whereas NANOMIND and NanoVLM Wiedmann al. (2025) on Jetson Nano/AGX use less. The reduced usage in NANOMIND can be attributed to TABM ' ring buffer, which optimizes shared memory, while NanoVLM is an efficient Jetson framework that we could not deploy on the Rockchip SoCs.
(Marafioti et al. (025). on four hardware platforms: NANOMIND,
Mertry thopstit))
Nutcoand Elanacpe trace ACX (Noo Neseeee ilws)
Dongo Fi Fichoaags
ceus (MM
Messg Bogeee
PaanEzz (laza cap H
dbas Nam (Dumr)) Mi]
Jetcc haso Jammesps
Hele hame (Beaass!
Jeban Numa
tlama opt
(a) Llava onevision-O.. 0.5B
(b) Qwen2-VL. VL-2B
(c) EnnolVMM-500X
Figure 5: Memory utilization (GB) across different hardware platforms and LLM frameworks:
Llava-onevision-O..55 Qwen2-VL. -LL-2---L-B----R Insttrcct, and SmolVLM-500M.
Figure 6 reports throughput (tokens/s) and end-to-end latency (s) for Qwen2-VI. ----2B- Instruct with 4- bit quantization across different hardware platforms. (NANOMIND hardware with llama.cpp exceeded the runtime limit, so results are omitted.) Despite being less powerful than the Orange Pi 5 Ultra (RK3588 (Devices)) and Jetson Nano, NANOMIND achieves throughput comparable to Jetson Nano running LLM with CUDA (35.7 tok/s), while reducing end-to-end latency by 36.2% compared to the Orange Pi 5 Ultra using the official rkllm (Linux 2025)
Thronghput (Toles)
Throughpat (Tokis)
End to- End Latency (4)
End-te-En Eed Latency
30
48
30
36
20
24
12
O.P5(Ilma.cog opp)
O.Pi(ckllm)
Jetson Olama opp)
Jetson (NanoVLM)
Nanomind
Figure 6: Throughput (tokens/s) and end-to-end latency (s) for Qwen2-VL-2B-Instree on the In- foVQA Mathew et al. (2022) dataset across different hardware platforms. "O-Pi" refers to the Orange Pi 5 Ultra, and "Jetson" the NVIDIA Jetson Nano. NANOMIND uses cross-accelerator dynamic offloading, with FP16 for the vision encoder and W4A16 for the LLM.
8

---

Published as a conference paper at ICLR 2026
4.2 DIFFERENT COMBINATIONS OF HYBRID QUANTIZATION
To illustrate the trade-off between quantizations and performance, Figure 13 the Appendix compares various quantization strategies and module-decoupling configurations. Legend labels follow the format Hoduule-Qantizatioon where em- denotes the embedding layer, vis- the vision encoder (ViT), and dec- the language decoder (Qwen2-0.5B). fpl6 indicates 16-bit floating point, and q4f16 represents 4-bit weight quantization with fp16 activations. We evaluate these configurations on MMBench, MMLU, MME, and InA. As shown Figure [3] when the VLM is decomposed and each module (e.g.. the VIT and LLM) runs on different accelerators, accuracy on vision-related tasks is largely determined by the ViT's precision.
4.3 BREAKDOWN OF SYSTEM-LAVEL PERFORMANCE
To go beyond end-to-end results, we perform system-level breakdown quantify the contribution of each component in our design.
Zero-Copy TABM Architecture vs. Traditional Copy-Based Buffering TABM's ring-buffer and zero-copy design primarily reduce memory usage and CPU overhead by eliminating the need for CPU- managed buffer writes when transferring embeddings. We evaluated memory usage (GB) and CPU utilization (%) in Figure 7(a) Compared with llama.cpp's memory copy and layer-wise offloading, TABM achieves lower memory usage and significantly reduced CPU load during embedding transfer.
Visual Embedding Models: NPU vs. GPU vs. CPU We evaluate inference latency for two visual embedding models. SigLip (Zhai al. 2023) (from LLaVA- Vision- Vision-Qwen2 HHF 2025)) and
InsightFace (AreFace) (Deng et al. (022)), by measuring the per-image processing time (ms) on the NPU, GPU, and CPU under continuous input. All images used evaluate SigLip are resized to 384 x 384 to match its training resolution. For InsightFace (ArcFace), we use the MegaFace evaluation dataset (Nech & Kemelmacher-Shiie Shiiterr-aiiiimann 2017). which is also used during ArcFace training.
Memory Usage (GB)
Memory Footprint
C22 Utilization (%)
100
Time (ma) img
400
CPU
NPU
GPU
300
50
200
0.5
100
0
Cogy
BM
CPU Cops
TABM
Sigl-Ip
ArcPace
(a) Zepo-Copy TABM: Memory and CPU
(b) Single Image Encoding Latency
Fused dequant GEMM kernel GPU (Namomind) llama.cpp
MLC.-LIM Powerinfor-2
40
Throughput (Tokes (Tokes)
30
20
10
Orange Pi (RK3588)
RubikPi (Dragonwin QCS6490)
(c) Fused dequant GEMM kernel vs Baselines
Figure 7: System Breakdown Performance. Zero-Copy TABM Traditional Direct Copy and Offloading used on Ilama.cpp b) Visual Embedding Inference Latency (ms) Comparison: SigLip ViT encoder (from LLaVA- One Vision (Liu et al.) 2024a HF 2025)) and the ArcFace encoder (Deng et al.) 2022) across RKNN NPU, Mali GPU, and CPU. Throughput Comparison (Tok/ss): NANOMIND 's custom GEMM kernels (GPU-only LLM decoding). llama.cpp. MLC-LLM, and PowerInfer-2 on the Orange Pi 5 (RK3588) and RubikPi (Dragonwin QCS6490) while running Qwen2- 1.5m2-1-1BBB-B ------B--....
Fused dequant-GEMM Kernel vs. Existing Frameworks The fused dequant-GEMM, kernel runs the SoC GPU for LLM decoding. Because PowerInfer-2 and MLC-LLM currently support only LLM, we compare all frameworks in text-only setting using the Qwen2-1.55 1.5B- W8A8 model. Figure (cc shows that our fused Dequant-GEMMn kernel achieves the highest throughput (tok/s), with PowerInfer-2 close behind. MLC-LLM on the RubikPi (Qualcomm QCS6490) performs worse, likely due to weaker OpenCI. support Qualcomm GPUs rather than limitations of MI.C-

---

Published as a conference paper at ICLR 2026
LLM or the hardware. This experiment evaluates only GPU-side decoding and does not involve cross-accelerator inference or buffer management.
Mik Parformince Farallal Officiaing
1) Unconstriined Siste
(4) Proportonal Throtting (1 4 Taga)
Average Power Comsumption
(80 Cittcal Consenvaion (9 as Tawl
Energy alrncy
(Dagraddaaaaaalonnn interieterppisatinn
Battery Depleting
decredtes)
Mak Efficincy Onmandd Inferance
150
End-to-End Latency (seconds)
Figure 8: Energy-Latency Trade-off Across Three Power Modes. The curve illustrates how the system adapts to the battery level (B). (1) In the Unconstrained State, parallel acceleration delivers low latency at higher power. (2) In the Proportional State, the system linearly throttles frame rate and memory bandwidth as B decreases, producing a continuous latency-power trade-off trajectory. (3) In the Critical State, the system transitions to the low-power On-Demand Cascade pipeline.
4.4 QUANTITATIVE POWER CONSUMPTION ANALYSIS
To explore the energy latenyy trade-offss across power modes during long- rongrruning workloads, Figure shows how NANOMIND adapts to different battery levels in a real-world smart headband deployment. As illustrated in Figure 12 in the Appendix, we built pattery-powere prototype and conducted week-long study with five users, collecting extensive traces that were used to produce Figure g
Figure 9 reports power consumption and estimated runtime of NANOMIND when powered by standard 2000 mAh COTS battery pack. Thanks to software-hardware co-design, NANOMIND consumes less power by reducing resource usage. In the low-power power mode, the on-demand cascade pipeline consumes an average of 0.375 under event tring-triggeed execution. Assuming a standard 2000 mAh battery, this translates up to 20.8 hours of operation under low-duty-cycle workloads, where the system remains in standby and performs inference only when triggered.
Namomind (laama-cpp)
Nanomind (Own)
Orange (llama.cpp)
Jesson Nano (Naa
50
Orange (ckllon)
letson AGX (NaooVLL))
Jetson Nano (Ilama. epp)
Nanomind (On-Demand)
40
Jetsen AGX (llema cpp)
30
24.BW|
20
18.8W
13.5W
10
4.9V
4.71
G4h
Power
Hours to Use
Figure 9: Power consumption (W) and estimated operating hours of NANOMIND when connected to standard commercially available 2000 mAh power bank.
5 CONCLUSION
In this paper, we introduced NANOMIND, a hardware-software co-design framework for efffccinnt on-device inference of large multimodal models. By decomposing models into modular components and dynamically offloading tasks across heterogeneous accelerators, our evaluations show that matches outperforms existing frameworks edge devices, while enabling over 20 hours of nattery-poweree multimodal inference in low low-power mode. This work demonstrates practical path toward democratizing private, responsive, and energy-efficient multimodal Al everyday devices.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.142)) 2024.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. SmolLM blazingly fast and remarkably powerful, 2024.
Anthropic. Introducing the next generation of Claude. https//www..athpppcccccooowwwwes Family 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308, 00008..2966 12966, 2023.
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang. Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In In Proc. of the 13th OSDI, 2018.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626/ 2025.
Jankann Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):5962-597, October 2022. ISSN 1939-3539. doi: 10.1 109/ tpami. 20l1.i2021...8881. 33887709. URL http://dx..oi.orgg//// Tttp///0.1199PAMMIIIMM 100/TPPMM.2021.00887777 13087799
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit Inference Scaling Laws, 2023.
Tiny Devices. RK3588 Reverse engineering the RKNN (Rockchip Neural Processing Unit). http:///ssssiiei.kooooooooo/oooo
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. 2024. URL https://axii.orrr//////777715.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers. arXiv preprint arXiv:2210.17333, 2022.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, 2024. URL URL UtLLippps//aariioorgb2222...... 13394
Gemma Team. Gemma 3 Technical Report, 2025. URL http:://arriv.oo///////555.11111
Georgi Gerganov. llama.cpp. http:///wwhhhh..oomm///re/////// 2023a.
Georgi Gerganov. whisper.epp. http::/:/hhggghh..oooo//oooooggg 2023b.
Jesse Gross. Ollama. https///ihhbbboomm/ceeesss////uu 2023.
Song Han, Huizi Mao, and William Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. International Conference on Learning Representations (ICLR), 2016.
LLaVA HF. Llava-onevision qwen2-0.5h https//huggngguue..//laaaaaaaa htw.://wwwwwwwwaoeeeieeeoeooneee 0f 2025. Accessed: 2025-09-24.

---

Published as a conference paper at ICLR 2026
Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. PLeak: Prompt Leaking Attacks against Large Language Model Applications. In In Proc. of ACM CCS 2024, CCS '24, 2024.
Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. ProPILE: probing privacy leakage in large language models. In Proc. of the 37th NeurlPS, 2023.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. ALaVA-OneVissonn Easy Visual Task Transfer, 2024. URL
https://axixv.rr/bbb/448....88 033322
Yilong Li, Jingyu Liu, Hao Zhang, M Badri Narayanan, Utkarsh Sharma, Shuai Zhang, Yijing Zeng, Jayaram Raghuram, and Suman Banerjee. PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS, In The Thirteenth International Conference Learning Representations, 2025.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for IIm compression and acceleration. In MISys, 2024.
Rockchip Linux. Rknn toolkit2. hutps//ghhhkkooooooocciil//////o 2025. Accessed: 2025-09-24.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://llaaa-
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?
In European conference on computer vision, pp. 216-233. Springer, 2024b.
LlamaEdge. LlamaEdge. https://lamacdde.com// 2024.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits, 2024.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. SmoIVLM: Redefining small and efficient multimodal models. arXiv preprint 1.Xiv:2554.05299 2025.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A Dataset for VQA on Document Images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2208, 2021. doi: 10. 20110//CVV4866002200 .00025..
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. Infographic VQA. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.
MLC team. Machine Learning Compilation (MLC)). https://lm.mlc.ai/docss 2023a.
MLC team. MLC-LLM Github Repo. http://github.ooo/ml-e-aee
2023b.
Aaron Nech and Ira waattnacchh--hhizenmaae Level Playing Field for Million Scale Face Recogni- tion. In In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, 2017.
OpenAL GPT-4 Technical Report, 2024.
OpenAl. Introducing GPT-5. http:://groa...ommmmeeee/mmoeoes 2025.
12

---

Published as a conference paper at ICLR 2026
Orange Pi. Orange Pi 5B. http://wwwaagggggghaaaaavooooocr detailsOOrange-Pi-55 Pi-55e.-mi-B.hmm
Qualcomm. Qualcomm ai hub. https//ahubbqualcomm.com 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In In Proc. of ICML 2021, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In In Proc. of 40th International Conference on Machine Learning (ICML). JoLR.org, 2023.
Rhasspy. Piper: A fast, local neural text-to-speech system. https//gihhb..com//haapyypppp 2025.
Accessed: 2025-09-25.
Rockchip. RK3566: high-performance and low power quad-core application processor. /www.rockchips.com/
https:
MSOON Technology. High voltage power monitor. http://wwww.mssoo..cc.hhhhg..gg monitor, 2025. Accessed: 2025-09-25.
power-
Hongyu Wang, Shuming Ma, and Furu Wei. BitNet a4.8: 4-bit Activations for 1-bit LLMs. arXiv preprint arXiv:2411.0490 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 2024b.
Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. T- MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge, 2024. URL
https:/aaxiv.oog/bbb2440....0000
Luis Wiedmann, Aritra Roy Gosthipaty, and Andrés Marafioti. nano VLM. https://gtthub.co//
huggingfacc/nano VLM, 2025.
Daliang Xu, Hao Zhang, Liming Yang. Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. Fast On-device LLM Inference with NPUs. In Proc. of the 30th ASPLOS, 2025.
Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. PowerInfer-2. Fast Large Language Model Inference Smartphone, 2024.
Jiaming Yang, Chenwei Tang, Caiyang Yu, and Jiancheng Lv. GWQ: Group-Wise Quantization Framework for Neural Networks. In Asian Conference Machine Learning. PMLR, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11975-1 1986, 2023.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 LLAMA.CPP LAYER OFFLOADING MECHANISM
Most of the open-source frameworks- as lpm...pp- were designed for desktops and servers with separate CPU-GPU memory, requiring repeated parameter copies from DRAM GPU memory. Although later adapted for edge devices, they inherit assumptions from these architectures. Modern mobile SoCs use unified memory, where CPU, GPU, and NPU share the same DRAM. Applying legacy designs directly leads inefficiencies, accelerators must coordinate access to shared memory, making new system-level optimizations necessary.
In llama.cpp, GPUs accelerate tensor operations such matrix multiplication through high paral- lelism. When a GPU backend (GGML BACKEND_ GPU) is enabled, @gml__cmputee cormpute_forwaroooe forward () offloads supported operators to the GPU. During execution, key tensors (c.g.. K, Q. V) are transferred from host memory to GPU memory, where the associated computations are performed while the CPU orchestrates control flow. Intermediate results stay in GPU memory, and only the final output tensor copied back to CPU memory once the operation completes.
CPU Memory
K
ggml graph Btw___rrapppppseeeeee compute()
Results
forward()
Copy GPU
Offload computation to GPU
Copy results back to CPU memory
Memory
KQ scaled
KQ_ masked
KQ softmax
Results
GPU Memory
Figure 10:
The model layer offloading mechanism of llama.cpp Gerganov 2023a), which requires
CPU to frequently write data memory and use extra memory space.
A.2 NANOMIND DEMO WITH HARDWARE
Speaker
demera UBL
CB 3.0 Type C
Audio Adanter
Microphone
Lithium Battery
(a) Audio, Vision connections.
(b) Battery
Figure 11: NANOMIND hardware design and device interfaces. (a) multimodal connections (an carphone, microphone, and an RGB camera); (b) battery power module..
14

---

Published as a conference paper at ICLR 2026
A.3 MEASUREMENT AND DATASETS
Power Measurement: We employed professional USB-based power measurement instruments from Klein Tools monitor the power consumption of each tested device, along with the High Voltage Power Monitor from MSOON (Technology 2025).
Datasets: We use datasets including Info (Mathew al. 2022). DoCVQA (Mathew al. 2021), MMBench (Liu et al. 2024b), and MME (Fu ct al. (024) along three dimensions: (I) profiling resource usage
different platforms, m2)m accuracy across different offloading strategies, and (3) measuring power efficiency under different runtime conditions.
End-to-End Latency: The latency we report is end-to-end, measured as the total time from providing the input image and prompt to receiving the final response.
A.4 REAL-WORLD DEPLOYMENT OF NANOMIND ON A HEADBAND
To study user experience, energy efficiency, and latency under real-world usage with variable interac- tion patterns, we built headband- based demo device running NANOMIND. Users wore the device and interacted with it through natural language.
Head Band with NANOMIND
(b) Headband with NANOMIND
Figure 12: NANOMIND hardware design and device interfaces. (a) multimodal connections (an earphone, microphone, and an RGB camera); (b) battery power module..
A.5 DIFFERENT COMBINATIONS HYBRID QUANTIZATION
The results show that when the VLM is decomposed and each module-such as the ViT and LLM-is executed independently on different accelerators, the accuracy on vision-related tasks is predominantly determined by the precision of the VIT. This highlights the importance of allocating higher bitwidth or computational resources to the vision encoder when optimizing for multimodal performance under constrained hardware.
A.6 ADAPTATION ACROSS DIFFERENT SoCs
Table 2 summarizes the theoretical deployment support of NANOMIND across different SoCs. Our current implementation is fully realized only on custom RK3566 and RK3588 hardware with inte- grated PMU. Support for the Qualcomm Dragonwing QCS6490 is still under development, and our evaluations for this platform are conducted on the Rubik Pi
15

---

Published as a conference paper at ICLR 2026
buseline 0.5B
embiin88 visqá decqdf16
100
embfp16 visfp1ó decg4f16
embint88 visq4f16 6
embfp16 visq4 dcco4f16
embq4f16 visfp1 decq4f16
Accuracy Score (%)
80
embfp16 visq4f16 decq4f16
embq4f16 visq4 decq4f16
embint8 visfpl6 docg4f16
embg4f16 visg4f1 decq4f16
60
MMBench
MME
MMMU
InfoVQA
Figure 13: Comparison of different quantization strategies and module decoupling configurations. Each legend label follows the format Qonuul--uantizzttion Specifically, "em-" denotes the em- bedding layer, "vis-" refers to the vision encoder (ViT), and "dec-" indicates the language decoder (Qwen2-0.5B). "fp16" represents 16-bit floating-pont precision, while "q4f16" indicates 4-bit weight quantization with fp16 activations.
SoC
NPU
GPU
DSP
Supported?
Notes
RK3588
Directly supported
QCS6490
bSP-based NPU
In-Progress
Directly supported
Apple M2
Partial
Closed-source
Mali-only SoC
Very Limited
CPU-GPU Coordinate
Table SoC support.
16


Reviews:
[{"review_id": "2p11q4DQ7r", "reviewer": "Reviewer_7bMY", "summary": "This paper proposes NanoMind, a hardware-software co-design framework for efficient on-device inference of LMMs. The key idea is to decompose LMMs into modular components and dynamically offloads them to the most suitable heterogeneous accelerators including NPU, GPU, and CPUs. This paper proposes a token-aware buffer manager for zero-copy data transfer and a dynamic power management strategy. The framework is implemented on a custom-designed device based on the RK3566 SoC. Evaluations demonstrate reduced memory usage, lower latency, and improved power efficiency.", "questions": "- How can the proposed framework be adapted to other edge SoCs with different accelerator configurations (different NPU/GPU capabilities)? What modifications or adjustments would be required?\n\n- Can you provide more compelling quantitative evidence or statistical analysis to demonstrate that NanoMind offers a significant improvement in key metrics like latency reduction, memory efficiency, or accuracy preservation?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem is well-motivated, addressing the critical challenge of efficient LMM deployment on resource-constrained edge devices.\n\n- The paper conducts practical system implementation and evaluation on real hardware.", "weaknesses": "- The paper claims to be a system-algorithm co-design method; however, the algorithm-level innovations appear to be minor, primarily leveraging existing quantization and model decomposition ideas rather than introducing novel algorithmic contributions.\n\n- The framework's adaptability to different devices with varying computational resource budgets is not explored. The experiments are conducted only on RK3566 SoC, limiting the generalizability of the proposed framework.\n\n- Lack of strong quantitative metrics demonstrating the method effectiveness. As seen in Figures 5, 6, and 7, the improvements in memory usage, latency, and accuracy over baselines appear marginal, lacking of quantitative evidence for a substantial performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d0ZaWxCyCy", "reviewer": "Reviewer_NZpb", "summary": "This paper proposes NANOMIND, a novel software–hardware co-design framework for enabling efficient on-device inference of large multimodal models (LMMs) on resource-constrained, battery-powered devices. It leverages the modularity of LMMs by decomposing them into “bricks” and dynamically offloading modules to optimal compute units (NPU/GPU/CPU). The paper introduces a custom inference pipeline, quantization strategies, token-aware buffer management, and custom hardware design, achieving notable energy and memory efficiency.", "questions": "- A key architectural component is the use of a token-aware ring buffer to facilitate zero-copy data flow between heterogeneous compute units (e.g., NPU and GPU). While this design significantly optimizes memory bandwidth and latency, how does it manage the Key–Value (KV) caches?\n\n**Minor Comments**:\n\n- The references in the current version of the paper are incomplete in formatting and lack hyperlinking.\n\n- The information presented in the paragraph starting at line 354 overlaps considerably with the earlier section “Token-Aware Buffer Management” (beginning at line 327). The two sections convey similar ideas and could be merged into a single, more concise paragraph to improve the paper’s flow and avoid redundancy.\n\nI am not deeply familiar with prior work on on-device inference frameworks, so I am unsure whether other closely related studies exist. I would be happy to discuss this further during the rebuttal and consider increasing the score accordingly.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The software–hardware co-design for efficient on-device inference of LMMs is very interesting and impactful.\n\n- Clear research motivation: tackling latency and energy inefficiency of LMMs on edge devices.\n\n- Practical systems-level contribution, combining model decomposition, dynamic workload scheduling, and embedded hardware design.", "weaknesses": "- While the proposed deployment strategy is evaluated on the authors’ custom SoC, the paper does not provide validation on other commercial or widely available SoCs.\n\n- In Figure 6, Throughput (tokens/s) and Latency appear to vary according to the device’s power state, which is controlled by the proposed Power-Efficiency Strategy. If I understand correctly, the battery level determines whether the system operates in a high-performance parallel mode or in the low-power “On-Demand Cascade Inference” mode. However, the paper does not clearly explain how throughput and latency are measured or computed under different power levels.\n\n- While Figure 7 compares different quantization strategies, the paper does not include an accuracy comparison between the proposed system and existing implementations. Such a comparison is necessary to clearly demonstrate the performance advantage of the proposed design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "51U4gh4qtN", "reviewer": "Reviewer_VTbL", "summary": "The paper introduces NANOMIND, a hardware–software co-design framework targeting efficient on-device inference of Large Multimodal Models (LMMs) on battery-powered systems. The authors propose decomposing multimodal models into modular components (e.g., vision encoders, LLM decoders) and dynamically mapping each module to the optimal heterogeneous accelerator (GPU, NPU, CPU) under a unified memory architecture.", "questions": "- How does the scheduling handle rapid mode switching (camera and audio streams) under burst workloads without degrading responsiveness?\n\n- Can the system support multi-image or temporal encoder models given the static-shape NPU limitations?\n\n- What is the quantitative energy-latency trade-off curve of three power modes over long-running workflows?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- On-device LMM/VLM execution is increasingly important for privacy, latency, and offline use. The focus on battery-powered compact devices differentiates this work from existing edge-accelerator papers.\n\n- The modular execution model, accelerator-aware scheduling, and token-aware buffer manager offer practical and technical novelty, especially under UMA constraints.\n\n- The combination of FP16 encoders with W4A16 LLMs, fused dequant-GEMM OpenCL kernels, and linear attention demonstrates solid engineering toward performance and power efficiency.", "weaknesses": "- Baseline gaps: lacks rigorous, same-hardware comparisons against state-of-the-art mobile stacks such as MLC-LLM, llm.npu, and PowerInfer-2, weakening external validity.\n\n- Incomplete utility evidence: limited end-task accuracy and qualitative results for multimodal workloads leave the user-perceived quality underexplored.\n\n- Scalability uncertainties: generalization to larger models and to NPUs with static-shape constraints is not demonstrated, and ablations isolating hardware choices are minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2xKAOYBLQ", "reviewer": "Reviewer_PgXN", "summary": "This paper presents NANOMIND, a hardware-software co-design framework for efficient on-device LMM inference. The key idea is decomposing LMMs into modular components and dynamically scheduling each to the most suitable accelerator (NPU/GPU/CPU) on unified-memory SoCs. The authors build a custom battery-powered device with RK3566 SoC, achieving 42.3% energy reduction and enabling 20.8 hours of operation. The system features Token-Aware Buffer Manager for zero-copy transfer, battery-aware execution modes, and custom low-bit quantization kernels.", "questions": "1. **Portability strategy**: How would NANOMIND adapt to SoCs without dedicated NPUs (e.g., Mali-only systems) or with different NPU architectures (e.g., Qualcomm HTP)? What abstraction layer could make the framework hardware-agnostic?\n\n2. **Static shape limitation**: You mention NPUs require static input shapes, which you address by fixing image resolution. How does this impact accuracy on datasets with varying native resolutions? Did you experiment with multiple fixed resolutions or dynamic resolution selection?\n\n3. **Real-world deployment**: Your experiments use controlled benchmarks (MMBench, InfoVQA, etc.). How does the system perform in real-world usage with variable user interaction patterns, thermal throttling over extended use, and battery degradation over time?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **Holistic system design**: The paper presents a rare end-to-end co-design spanning algorithm (model decomposition, quantization), system (scheduling, memory management), and hardware (custom PCB with PMU, parallel memory). This comprehensive approach addresses multiple bottlenecks simultaneously rather than optimizing in isolation.\n\n2. **Practical hardware validation**: Unlike many systems papers that only simulate or use off-the-shelf platforms, the authors designed and fabricated custom hardware, providing concrete evidence of feasibility and real power measurements through integrated PMU.\n\n3. **Novel heterogeneous scheduling**: The insight of mapping modular LMM components to different accelerators based on their computational characteristics (NPU for low-bit vision encoding, GPU for FP16 LLM decoding) is well-motivated and demonstrates clear benefits in the unified memory architecture context.\n\n4. **Strong empirical results**: The battery life improvements (20.8 hours for voice interaction) and energy efficiency gains (42.3% reduction) are substantial and practically meaningful for edge deployment scenarios.", "weaknesses": "1. **Venue mismatch concern**: This work is fundamentally a hardware systems paper with custom PCB design, power management circuitry, and hardware-specific optimizations. While it has ML applications, the core contributions (hardware architecture, cross-accelerator scheduling, unified memory optimization) align more naturally with hardware/systems venues like HPCA, ISCA, or MICRO rather than ICLR, which focuses on machine learning methods and representations. The ML community may lack the expertise to properly evaluate the hardware contributions, and the authors would likely receive more targeted feedback from hardware systems reviewers.\n\n2. **Limited generalizability**: The framework is tightly coupled to the RK3566 SoC and Rockchip's RKNN ecosystem. Key components (NPU offloading, RKNN model conversion, specific driver optimizations) may not transfer to other mobile SoCs (Qualcomm Snapdragon, MediaTek Dimensity, Apple Silicon). The paper lacks discussion of how the design principles would adapt to different hardware platforms or what abstractions could enable portability.\n\n3. **Insufficient ablation studies**: While the paper shows end-to-end improvements, it doesn't systematically isolate individual contributions. What is the specific gain from: (a) parallel LPDDR4x vs. standard configuration? (b) zero-copy TABM vs. traditional buffer management? (c) NPU vs. GPU for vision encoding? (d) custom GEMM kernels vs. existing implementations? This makes it difficult to understand which design decisions are most impactful.\n\n4. **Weak baseline comparisons**: The comparisons are primarily against llama.cpp on various platforms, but the paper doesn't compare against other recent edge inference frameworks (PowerInfer-2, llm.npu) on the same hardware. The NanoVLM comparison is limited to Jetson platforms. Additionally, the claim that llama.cpp is inefficient on unified memory architectures needs more rigorous support—is the inefficiency inherent to the framework or the specific platform/configuration?\n\n5. **Missing accuracy-efficiency tradeoffs**: Figure 7 shows accuracy across quantization strategies but doesn't correlate these with latency, throughput, or power consumption. What accuracy degradation is acceptable for different battery levels? How do users navigate the performance-accuracy-power tradeoff space?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)