Title: GaLe: memory-efficient Global Approximate and Local Exact features
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Memory-efficient deep learning, Global-local feature map approximation, TinyML deployment, Resource-efficient CNN-transformer design, Low-memory inference

Abstract: Embedded devices and Microcontroller units (MCUs) generally offer only a fraction of the memory and computational power available on machines equipped with general-purpose GPUs. Existing approaches for memory-efficient inference on these devices rely either on patch-based inference, which causes significant computational overhead, or approximation-based methods, leading to substantial accuracy degradation.
In this work, we propose \emph{GaLe}, a novel memory-efficient approximation technique that enables the deployment of pretrained deep neural networks on tiny, resource-constrained devices without the need for retraining. Our method introduces a feature map partitioning strategy that approximates layer outputs using two complementary representations: (i) a local exact ($L_E$) component that preserves fine-grained details and (ii) a global approximate ($G_A$) component that retains long-range dependencies. Differently from available tiling approaches, GaLe maintains compatibility with architectures with global receptive field operations and attention mechanisms, such as modern hybrid CNN-transformer models, while significantly reducing memory usage and computational overhead.
We validate our approach on ImageNet classification, demonstrating performance comparable to exact inference methods while drastically reducing memory consumption and compute costs, achieving up to $65$% speedup on a Cortex-M33 core for a $90$% RAM reduction compared to patch-based inference. Beyond efficient deployment, GaLe offers a general recipe for feature map decomposition, enabling the design of novel, resource-efficient convolutional and attention modules and potentially guiding memory-aware architecture search. We further demonstrate its versatility across classification, detection, and diffusion models, highlighting its potential as a foundation for future research on memory-efficient architectures. GaLe also benefits general-purpose GPUs, reducing the memory usage of diffusion models under 200MB (from 6GB) for high-resolution outputs.

TL;DR: GaLe is a memory-efficient approximation technique that decomposes feature maps into local and global components, enabling pretrained deep models to run on tiny devices with up to 90% less RAM and 65% faster inference without retraining.