# Overlay-ML: Unioning Memory and Storage Space for On-Device AI on Mobile Devices
## 1 Introduction  
Nowadays, many researchers and engineers have been leveraging various machine learning (ML) techniques to improve the performance of image processing and pattern recognition [1, 2, 3, 4]. Especially, companies developing image sensors have started to introduce their new products to ensure the best image detection and performance because image sensors are considered the eyes of IoT devices [5]. In addition, the trends of image-sensing technologies and the corresponding ML techniques have shifted from low-end devices that just collect raw data from sensor devices to high-end and smart devices, which include more computation power and functions. For example, some studies focused on deep learning (DL) technologies that enable automated image classification in vehicles [4]. In fact, these are possible because the computing power of embedded devices (e.g., RISC-V [6], Raspberry Pi [7], smartphones, etc.) is growing to the point where it is sufficient to run ML techniques. Some researchers focused on optimizing on-device ML technologies [8, 9, 10]. Unfortunately, they never focused on the limitations of mobile devices while running ML models and did not utilize real-world embedded devices (e.g., smartphones or tablets) for their experiments. Meanwhile, the emergence of on-device ML [11, 12, 13] blurs the boundary ML technique on embedded devices; on-device ML executes training and inference procedure with the software help of ML instructions. For example, Google recently announced a new powerful package, called _ML Kit_, for mobile developers using Google machine learning [14]. It provides easy-to-use APIs that are supposed to detect barcodes, text, faces, and objects invideo and image files on mobile devices. TensorFlow Lite was designed to allow enabling on-device ML using transfer learning (TL), which reuses knowledge of a pre-trained model for intelligent services on embedded devices; the pre-trained model deals with full-scale datasets [12]. Thanks to these useful efforts and advances, we believe that it will be able to run more and more ML in mobile environments, such as data sensing on IoT platforms. However, there are some challenges in using ML on embedded devices. For example, the capacity issue of memory on embedded devices still remains because of limited space where memory chips are placed; an application running ML technologies on embedded devices still suffers from memory pressure. Therefore, an application developed with TensorFlow Lite [12] is unintentionally killed when it requires more memory space than the available memory space per application. The reason behind this is Android terminates an application that uses memory space over the pre-defined threshold (i.e., 256 MB) because of the fairness of memory resources. Unfortunately, such a killing issue can appear and is unavoidable for general applications on any embedded devices running the Android operating system (OS). A promising alternative to solve this problem guided by Google is turning on \"_largeHeap_\" option that extends the pre-defined threshold to 512 MB. However, the expanded memory footprint is too small to adequately run on-device ML because such an application commonly requires gigabyte volumes of memory. In this paper, we propose Overlay-ML which includes a new management policy to run on-device ML for image processing and pattern recognition even with memory pressure and limited memory space. To the best of our knowledge, it is the first solution to handle the memory issue for supporting applications with on-device ML. The key design concept of Overlay-ML is that it simultaneously handles tensors both on memory and storage media on embedded devices using _application-level swapping_ and performs _multi-layer shuffling_ to avoid overfitting issues of learning workloads. In this paper, we make the following contributions:  
* In order to understand the memory pressure issue, we briefly introduce the internal procedure of TensorFlow Lite [15] which is widely used for on-device ML.