SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hong-Sheng Zheng, Chen-Fong Hsu, Yu-Yuan Liu, Tsung Tai Yeh
National Yang-Ming Chiao Tung University
{hszheng.cs08，fonghsu.cs08，yyliu.cs11，ttyeh14}@nycu.edu.tw
With the emerging Tiny Machine Learning (TinyML) inference applications, there is a growing interest when deploying TinyML models on the low-power Microcontroller Unit (MCU). However, deploying TinyML models on MCUs reveals several challenges due to the MCU's resource constraints, such as small flash memory, tight SRAM memory budget, and slow CPU performance. Unlike typical layer-wise inference, patch-based inference reduces the peak usage of SRAM memory on MCUs by saving small patches rather than the entire tensor in the SRAM memory. However, the processing of patch-based inference tremendously increases the amount of MACs against the layer-wise method. Thus, this notoriously computational overhead makes patch-based inference undesirable on MCUs. This work designs StreamNet that employs the stream buffer to eliminate the redundant computation of patch-based inference. StreamNet uses 1D and 2D streaming processing and provides an parameter selection algorithm that automatically improves the performance of patch-based inference with minimal requirements on the MCU's SRAM memory space. In 10 TinyML models, StreamNet-2D achieves a geometric mean of 7.3X speedup and saves  $81\%$  of MACs over the state-of-the-art patch-based inference.
With the emerging Tiny Machine Learning (TinyML) applications, challenges to successfully deploy TinyML models on the resource-constrained microcontroller (MCU) are becoming critical. Unlike the high-performance CPU, the MCU is basically composed of a small flash memory (several MBs), an SRAM memory (hundreds of KBs), and a slow CPU (hundreds of MHz). Since the SRAM memory has better performance on reading and writing data than the flash memory, most TinyML system frameworks often place the input/output tensors on the SRAM and store weights and filters on the flash memory. In addition, the size of memory on MCUs is always small because of their constrained budgets. However, recent TinyML models tend to use large input images and intermediate tensors to raise their training accuracy (1; 2). Such TinyML models expose grand challenges when deploying these models on MCUs with the stringent memory constraint.
Tensor memory planner of TinyML system frameworks determines the usage of the SRAM memory. An MCU is often responsible for Machine Learning (ML) inference and often uses a single batch to facilitate the response time of ML inference tasks. Thus, the static tensor memory planner schedules intermediate tensors to meet the SRAM memory constraint. Conventionally, the compiler/interpreter employs layer-wise scheduling, pushes data of a layer in the SRAM memory, and replaces them after finishing its operator (3; 4). However, such layer-wise inference will overuse the SRAM memory space when TinyML models have large intermediate tensors. To overcome this shortcoming, patch-
based inference (2) divides the entire tensors into multiple overlapping patches. Then, patch-based inference places a patch in the SRAM memory space rather than the entire tensor to repair the out-of-memory problem. However, the redundant computation of patch-based inference occurs when proceeding with a large number of overlapping patches and tremendously increases the execution time of TinyML models. Furthermore, this situation becomes worse in particular when significantly shrinking the usage of SRAM memory space. Hence, patch-based inference is becoming undesirable with such notoriously high performance overhead on MCUs.
This work designs StreamNet that eliminates the significant amount of redundant computations while reaching minimal usage of the SRAM memory space for TinyML models on MCUs. StreamNet creates the stream buffer to reuse tensor data in the patch-based inference. In addition, StreamNet designs 1D and 2D stream processing to improve the performance of patch-based inference with different SRAM memory requirements. To facilitate the performance of StreamNet further, StreamNet also removes the unnecessary computation on the padding data. At last, StreamNet's parameter selection framework automatically constructs a new patch-based inference that mixes 1D and 2D stream processing to meet the memory constraint while resulting in the best performance for TinyML models.
The main contributions of this paper are shown below:
1. StreamNet achieves a geometric mean of 3.4X (1D) to 7.3X (2D) speedup over the existing patch-based inference without increasing memory usage significantly.
2. Developing output padding bypass kernels to enhance the performance of patch-based inference.
3. The StreamNet parameter selection framework automatically selects an optimal configuration for StreamNet-2D operations under the SRAM memory constraint of an MCU.
2 Understanding Performance Bottleneck of Memory Allocation on TinyML
Memory-efficient patch-based inference: Patch-based inference aims to reduce the peak SRAM memory usage of TinyML models on MCUs. This patch-based approach divides a large input feature map into multiple small patches. Then, the MCU's SRAM buffer reduces its space usage by only storing a small patch each time instead of the entire input feature map. For instance, Figure 1(a) demonstrates the peak SRAM memory usage of the proxyless-w0.3 TinyML model (5) and other TinyML models have the similar trend. Traditionally, ML system frameworks such as TFLM (3) place input and output tensors of a layer or an operator in the SRAM buffer. After finishing the work of a layer, TFLM (3) removes the obsolete input data of the completed layer and allocates another space to put the output data of the next layer. However, given an MCU and its size of SRAM memory is 128 KB, Figure 1 demonstrates that the memory usage of multiple layers in the proxyless-w0.3 TinyML model (5) is larger than the size of the SRAM memory on an MCU. As a result, the layer-wise memory allocation raises significant challenges when deploying TinyML models on MCUs. Unlike the layer-wise inference, Figure 1 presents patch-based inference that reduces the total SRAM memory usage of TinyML models and enables models to be run on MCUs with small memory size.
(a) The peak memory usage of cascade policy over the baseline
(b) The amount of MAC in cascade policy over the baseline
Figure 1: Comparing peak memory usage and the amount of MACs on patch-based and layer-wise inference
Challenges and opportunities: Patch-based inference creates a large amount of redundant computation that occurs among its adjacent patches. To reduce the peak SRAM memory usage on MCUs,
patch-based inference creates many small patches that overlap receptive fields. These small patches tremendously raise the amount of computation on a TinyML model. The inefficient patch-based inference is becoming undesirable on MCUs even if it saves the peak SRAM memory usage of a TinyML model significantly. Figure 1(b) demonstrates that patch-based inference takes  $33\mathrm{X}$  more MACs than the layer-wise one at the first layer. Furthermore, the amount of MACs in patch-based inference also rapidly grows with the reduction of peak SRAM memory usage. Hence, our work stores repeated results of patch-based inference in the stream buffer and skips these computations to eliminate the performance bottleneck of patch-based inference.
3 StreamNet: Efficient Patch-based Inference for TinyML Models
This section details the design and implementation of StreamNet that removes the redundant computation of patch-based inference with the minimal usage of the SRAM memory space on MCUs.
Figure 2: Patch-based inference example
3.1 Inefficient Patch-based Inference
To accelerate patch-based inference, this work proposes StreamNet that removes most redundant computations of patch-based inference by putting its repetitive results in the stream buffer. Figure 2 demonstrates operations of patch-based inference. In Figure 2, there are two operators where each of them has its own input and output tensor and the computed block means the completed output element. Conventionally, in the convolution processing, the  $6 \times 6$  input with the  $3 \times 3$  filter will yield the  $4 \times 4$  output. This calculation takes 144 ( $16 \times 9$ ) MACs where the number 16 is the size of the output and the number 9 comes from the  $3 \times 3$  filter. Unlike conventional convolution, in Figure 2, patch-based inference divides a  $6 \times 6$  input into four  $5 \times 5$  patches. Each  $5 \times 5$  patch computes the convolution with the  $3 \times 3$  filter to finally yield a  $4 \times 4$  output. As illustrated in Figure 2(a), Patch 0 and Patch 1 generate a  $3 \times 3$  output patch, respectively. Each of them takes  $9 \times 9$  MACs where the first number 9 is the size of output and the second number 9 is associated with the size of the filter. Thus, patch-based inference uses 4 patches to create a  $4 \times 4$  output and takes 324 MACs ( $9 \times 9 \times 4$ ). The amount of MACs of patch-based inference is 2.25X (324/144) more than the layer-wise inference because patch-based inference contains an enormous amount of repetitive computations. For instance, in Figure 2, each value in the overlapping area represents the repetition count of the output tensor element. This redundant calculation increases the execution time of a TinyML model significantly. In addition, patch-based inference tends to use the fine-grained patch to lower the usage of the SRAM memory. However, the amount of redundant computation increases as the patch size becomes small.
3.2 StreamNet 1D Streaming Processing
Figure 3(a) illustrates the steps of 1D streaming processing on StreamNet inference. In Figure 3(a), two patches proceed one after the other as the data stream during the computation of the convolution. In Figure 3, the computed block represents the completed element of the tensor, and the computing block indicates the elements of the tensor in the execution. At first, in Figure 3(a), StreamNet-1D performs the CONV0 operator on the  $5 \times 5$  patch and yields a  $3 \times 3$  output tile (①). Second, StreamNet-1D stores the overlapping area of this  $3 \times 3$  output tile in the stream buffer (②). Then, StreamNet-1D uses this  $3 \times 3$  tile to generate the  $1 \times 1$  output data after finishing the CONV1 operator (③). Unlike patch-based inference, StreamNet uses the results preserved in the stream buffer during the computation of Patch 1. Thus, StreamNet-1D only fetches a  $3 \times 5$  tile from Patch 1 to yield a
Figure 3: The operation of StreamNet-1D
$3 \times 1$  output data (4). Then, StreamNet-1D combines this  $3 \times 1$  output with the data of the stream buffer to work out a  $3 \times 3$  output tile as illustrated in Figure 3(a) (5). At last, the  $3 \times 3$  output of Patch 1 will yield a  $1 \times 1$  output after finishing the CONV1 operator (6). In the computation of Patch 0 and Patch 1, StreamNet-1D takes 108 MACs ( $12 \times 9$ ) where the number 12 is the sum of the calculated output size in (1) ( $3 \times 3$ ) and (4) ( $3 \times 1$ ) and the number 9 is associated with the size of the  $3 \times 3$  filter. Since StreamNet-1D also needs to use four patches to create a  $4 \times 4$  output in total, the total amount of MACs in 1D streaming processing of StreamNet-1D is 216 ( $12 \times 9 \times 2$ ). Unlike patch-based inference in Figure 3(b), StreamNet-1D in Figure 3(c) decreases the repetition counts in each element of the output tensor. Hence, StreamNet-1D reduces roughly  $60\%$  of MAC overhead over patch-based inference in Figure 3, because the overhead of StreamNet-1D and patch-based inference is 8 (24 - 16) and 20 (36 - 16) where  $16(4 \times 4)$  is the number of output elements of the layer-wise inference.
Figure 4: The StreamNet-2D operations
3.3 StreamNet 2D Streaming Processing
StreamNet uses the stream buffer to preserve temporary results that will be used multiple times across patches. After finishing an operator, StreamNet will place a number of output tensors in the stream buffer. Since StreamNet-1D does not completely remove the redundant computation overhead, StreamNet designs StreamNet-2D to eliminate all redundant MACs in patch-based inference. StreamNet-2D is composed of stream_x and stream_y processing. The execution order of
stream_y patches is firstly moving downward. For instance, patch-based inference in Figure 4 moves its execution vertically for Patch 1, then back to Patch 2, and finally moves downward for Patch 3. Unlike stream_y, the execution order of stream_x patch firstly moves toward the horizontal direction. At first, in Figure 4, the  $5 \times 5$  Patch 0 computes CONV0 with the  $3 \times 3$  filter and yields a  $3 \times 3$  output. Then, StreamNet-2D stores a  $3 \times 2$  tile and  $2 \times 3$  tile of the  $3 \times 3$  output in the Y and X stream buffer (①)(②), respectively. Second, StreamNet-2D tackles Patch 1 and fetches a  $5 \times 3$  tile in Patch 1 to yield a  $3 \times 1$  output. After finishing Patch 1, StreamNet stores a  $2 \times 1$  tile of its output tensor to the XY stream buffer (④) and cleans the Y stream buffer. Third, StreamNet-2D handles Patch 2 and reuses results in the X stream buffer. After completing Patch 2, StreamNet-2D stores a  $3 \times 2$  tile of Patch 2's output in the Y stream buffer (⑤)). Finally, StreamNet-2D reuses the data within the Y and XY stream buffer to proceed with Patch 3. StreamNet-2D takes 144 MACs  $((9 + 3 + 3 + 1) \times 9)$  where each number of the first bracket represents the amount of calculation in each patch and the second number 9 means the size of the  $3 \times 3$  filter. Hence, StreamNet-2D completely eliminates the total amount of the redundant computation shown in patch-based inference. However, StreamNet-2D has a larger stream buffer to store the temporary results of each patch over StreamNet-1D. As illustrated in Figure 4, the stream buffer of StreamNet-2D is 14 where the  $2 \times 3$  tile is from stream_x, the  $3 \times 2$  tile is from stream_y, and the  $2 \times 1$  tile is from the XY stream buffer. A large stream buffer increases the requirement of the SRAM memory space. To balance the consumption of the MACs and the SRAM memory space, StreamNet presents an parameter selection framework that mixes StreamNet-1D and StreamNet-2D. As a result, StreamNet can satisfy the constraint of the SRAM memory size on MCUs while achieving efficient execution on TinyML models.
Algorithm 1: The StreamNet Parameter Selection Algorithm
Input: model and constraint  
Output: selected StreamNet parameter under constraint  
1 patch.params = { (split_idx, n_batch) | split_idx \in ValidSplitIndexes(Model), n_batch \in OutShapeFactors(Layers[split_idx])}  
2 candidates = []  
3 for param in patch.params do  
4 performance = (mem, MACs) = ProfileParameter_PatchBased(param)  
5 if mem meets constraint then  
6 candidates.append(param, performance)  
7 Filter obtained records and only keep the ones that have performance on the Pareto frontier  
8 for candidate in candidates do  
9 for stream_x_level in [0, 1, ..., record.split_idx] do  
10 param = (candidate.params, max_STREAM_y_level, stream_x_level)  
11 performance = (mem, MACs) = ProfileParameter/stretchNet(param)  
12 if mem meets constraint then  
13 new_candidate.append(param, performance)  
14 if there is only one param that has the smallest MACs then  
15 return param as the best parameter.  
16 else  
17 return the param that has the largest output patch size as the best parameter.
3.4 StreamNet Parameter Selection Framework
Searching for the best parameter that satisfies memory constraints while minimizing the latency of a TinyML model manually is exceedingly tedious and challenging. To address this problem, StreamNet designs an parameter selection framework that determines the best parameter candidate to meet the constraint of the SRAM memory across different MCUs. Hence, StreamNet parameter selection framework aims to automatically find the parameter composition of StreamNet and mixes StreamNet-1D and StreamNet-2D to fit the limited amount of SRAM memory on the MCU. To minimize the runtime overhead, StreamNet refers to the MAC count to choose the best candidate of StreamNet parameters through its compiler pass.
Algorithm 1 demonstrates StreamNet parameter selection framework that infers parameters of StreamNet to fit an MCU at the compile time. The patch parameter in line 1 of Algorithm 1 includes the number of the patch (n_batch) and the number of layers that proceed with patch-based
computation (splitidx). The OutShapeFactor function returns the valid number of the patch in patch-based inference. Next, StreamNet calculates the amount of MACs with different combinations of these two parameters and picks out candidates that meet the constraint of the SRAM memory space of an MCU from line 2 - 7 of the Algorithm 1. Then, StreamNet examines different values of stream_x_level and stream_y_level to determine the number of stream-wise layers while having a minimum amount of MACs in a model and returns this combination from line 8 - 15 of Algorithm 1. Otherwise, StreamNet tends to choose the one with the smallest patch count when both candidates have the same amount of MACs, since a large output patch has bigger computation granularity, which usually yields better latency.
Figure 5: Output padding data bypassing on StreamNet
3.5 Output Padding Bypassing
The padding data of each tensor contributes the performance overhead in the patch-based inference. To further the performance of patch-based inference, StreamNet intelligently skips the computation of the padding data. Figure 5 presents a patch-based inference neural network that consists of 3 operators and 4 tensors. Padding is always added to a tensor to generate an output with the specified size. Parameters of conventional neural networks such as the size of the stride and the filter result in different amounts of padding data in each tensor. Unlike common neural networks, in Figure 5, the patch size of the current tensor is obtained from the configuration of the next tensor. For instance, patch-based inference refers to the patch size of Tensor C to figure out the patch size of Tensor B. The padding data in Tensor B appends the one in Tensor C. Thus, the amount of the padding data in patch-based inference can be more than the one in the layer-wise inference. Since the value of padding data is always assigned to be zero, the padding data won't change the expected results. In Figure 5, patch-based inference proceeds with a  $7 \times 7$  patch in Tensor A including padding data to generate an output  $5 \times 5$  patch. Next, this  $5 \times 5$  patch in Tensor B is used to create a  $3 \times 3$  patch in Tensor C. Hence, each operator computes its padding data that contributes additional MACs during the execution of TinyML models. StreamNet removes the performance overhead shown in the padding data by skipping the computation of each output padding data. In Figure 5, patch-based inference reads the data of a  $5 \times 5$  patch in Tensor B to generate a  $3 \times 3$  patch in Tensor B. Unlike patch-based inference, StreamNet uses the size of the padding data known after finishing the previous layer to distinguish the padding data from the real one. Next, StreamNet only considers the real data in the calculation of an operator. Furthermore, StreamNet only writes the zero value into output padding data directly without consuming any MACs. StreamNet changes the implemented kernel of patch-based inference to skip the computation of the output padding data during the execution of each operator.
4 StreamNet System Architecture
The system architecture of StreamNet contains the frontend and backend processing. In the frontend system of StreamNet, the input of StreamNet is the TensorFlow Lite model (.tflite file). Then, StreamNet uses the parameters of the TinyML model to create layers of patch-based inference (2). Second, the memory scheduler allocates SRAM memory space for tensors on the static arena memory space and then StreamNet performs the 1D and 2D stream processing with its stream buffers. In StreamNet's backend system, StreamNet translates tensors and operators of the patch-based inference
Figure 6: The system architecture of StreamNet
into C source code and links kernel libraries such as ARM CMSIS-NN (6), TinyEngine (1; 2), and our bypass padding kernels. Finally, StreamNet deploys TinyML models on the MCU by using Mbed CLI SDK (7).
5.1 Experiment Environment Setup
System Implementation: We compare the performance and memory consumption of StreamNet to MCUNetV2 (2), which performs patch-based inference on TinyML models. Since existing TinyEngine kernels (1) used by MCUNetV2 (2) do not support  $5 \times 5$  and  $7 \times 7$  filters, we fix this issue to enable our evaluation to work on the depthwise convolution function with more filter combinations. In addition, we also insert our output padding bypassing implementation within MCUNetV2 (2) as MCUNetV2+. StreamNet modifies MCUNetV2's memory scheduler and code generator to identify streamable area, allocate stream buffer, generate codes with different streaming processing, and insert the stream buffer operation between operator kernel invocation in StreamNet.
Benchmark: We evaluate 10 TinyML models from MCUNetV2 model zoo (5)such as mbv2-w0.35(MB2), proxyless-w0.3(PL), mcunet-vww0(MV0), mcunet-vww1(MV1), mcunet-vww2(MV2), mcunet-in0(MI0), mcunet-in1(MI1), mcunet-in2(MI2), mcunet-in3(MI3), mcunet-in4(MI4). All models use int8 quantized mode. These TinyML models have different requirements on the SRAM memory space and their execution time also differs.
On-device Experiment: We deploy TinyML models through ARM Mbed CLI (7) on the ARM Cortex-M7 CPU. The MCU used in our evaluation is STM32F767ZI (8) that includes an ARM Cortex-M7 CPU at 216 MHz, a 512KB SRAM, and a 2MB Flash
5.2 Performance Analysis when Consuming Nearly Equal Memory
Table 1 compares the performance of StreamNet and MCUNetv2 when both of them use nearly the same SRAM memory space. Table 1 presents the data of StreamNet-2D. Then, we scan through the value of n_batch and splitidx parameters to search for the one where their SRAM memory usage is closest to StreamNet-2D. In Table 1, StreamNet achieves a geometric mean of 5.11X speedup over MCUNetv2. The SRAM memory consumption varies with the change of n_batch and splitidx parameters on MCUNetv2. However, the amount of MACs increases rapidly with the reduction of SRAM memory usage on MCUNetv2. As a result, StreamNet has better performance than MCUNetv2 when consuming nearly equal SRAM memory.
5.3 The Variation of MAC Counts
Table 2 compares the amount of MACs used in each TinyML model. In Table 2, the value of the n_batch and splitidx determines the number of patches and the number of patch-based layers used in a model, respectively. StreamNet scans through all combinations of the n_batch and splitidx and
Table 1: Performance results of MCUNetv2 and StreamNet-2D when consuming nearly equal memory. Param (n_batch, splitidx), Memory (KB), MAC (millions), Latency (ms)
Model	StreamNet-2D	MCUNetV2	Speedup
Param	Mem	MAC	Latency	Param	Mem	MAC	Latency	MAC	Latency
MB2	(18,13)	66	24	417	(18,12)	63	77	1,513	3.29	3.63
PL	(22,13)	95	38	676	(11,21)	89	549	9,519	14.31	14.08
MI0	(12,06)	30	6	99	(04,14)	29	19	274,441	2.99	2.77
MI1	(12,11)	47	13	188	(03,12)	48	24	366	1.84	1.95
MI2	(20,21)	169	67	1,168	(05,25)	169	463	7,055	6.87	6.04
MI3	(22,21)	208	82	1,444	(11,32)	215	2,777	43,812	33.89	30.35
MI4	(20,17)	236	126	1,762	(04,21)	214	366	4,613	2.91	2.62
MV0	(08,13)	29	6	101	(08,11)	30	17	296	2.83	2.93
MV1	(20,14)	44	12	225	(05,17)	45	47	810	4.04	3.61
MV2	(18,21)	143	56	961	(06,29)	144	518	8,311	9.27	8.65
GMEAN	5.33	5.11