StreamNet: Memory-Efficient Streaming Tiny Deep Learning Inference on the Microcontroller
Authors: Hong Sheng Zheng
Yu-Yuan Liu
Chen-Fong Hsu
Tsung Tai Yeh
Venue: NeurIPS 2023 poster (2023)
Keywords: TinyML models
edge AIs
Microcontroller
TL;DR: Maximizing performance of patch-based inference with minimal SRAM memory usage for TinyML models on MCUs

Abstract: With the emerging Tiny Machine Learning (TinyML) inference applications, there is a growing interest when deploying TinyML models on the low-power Microcontroller Unit (MCU). However, deploying TinyML models on MCUs reveals several challenges due to the MCU’s resource constraints, such as small flash memory, tight SRAM memory budget, and slow CPU performance. Unlike typical layer-wise inference, patch-based inference reduces the peak usage of SRAM memory on MCUs by saving small patches rather than the entire tensor in the SRAM memory. However, the processing of patch-based inference tremendously increases the amount of MACs against the layer-wise method. Thus, this notoriously computational overhead makes patch-based inference undesirable on MCUs. This work designs StreamNet that employs the stream buffer to eliminate the redundant computation of patch-based inference. StreamNet uses 1D and 2D streaming processing and provides an parameter selection algorithm that automatically improve the performance of patch-based inference with minimal requirements on the MCU’s SRAM memory space. In 10 TinyML models, StreamNet-2D achieves a geometric mean of 7.3X speedup and saves 81\% of MACs over the state-of-the-art patch-based inference.

StreamNet: Memory-Eficient Streaming Tiny Deep Learning Inference on the Microcontroller
Hong-Sheng Zheng, Chen-Fong Hsu, Yu-Yuan Liu, Tsung Tai Yeh
Department of Computer Science
National Yang-Ming Chiao Tung University
Hsinchu, Taiwan yyliu.cs11, ttyeh14) edd.u..duuuuuuu....... edu. tw
(hszheng. cs08,
Abstract
With the emerging Tiny Machine Learning (TinyML) inference applications, there is growing interest when deploying TinyML models the low-power Microcon- troller Unit (MCU). However, deploying TinyML models on MCUs reveals several challenges due the MCU's resource constraints, such small flash memory, tight SRAM memory budget, and slow CPU performance. Unlike typical layer-wise in- ference, patch-based inference reduces the peak usage of SRAM memory MCUs by saving small patches rather than the entire tensor in the SRAM memory. How- ever, the processing of patch-based inference tremendously increases the amount of MACs against the layer-wise method. Thus, this notoriously computational overhead makes patch-based inference undesirable on MCUs. This work designs StreamNet that employs the stream buffer to eliminate the redundant computation of patch-based inference. StreamNet uses ID and 2D streaming processing and provides an parameter selection algorithm that automatically improve the perfor- mance of patch-based inference with minimal requirements on the MCU's SRAM memory space. In 10 TinyMI. models, StreamNet-2D achieves geometric mean 7.3X speedup and saves 81 % of MACs over the state-of-the-art patch-based inference.
Introduction
With the emerging Tiny Machine Learning (TinyML) applications. challenges to successfully deploy MinyML models on the resource-constraing microcontroller (MCU) are becoming critical. Unlike the high-performance CPU, the MCU is basically composed of a small flash memory (several MBs), an SRAM memory (hundreds of KBs), and slow CPU (hundreds of MHz). Since the SRAM memory has better performance on reading and writing data than the flash memory, most TinyMI. system frameworks often place the input/output tensors on the SRAM and store weights and filters on the flash memory. In addition, the size of memory MCUs is always small because of their constrained budgets. However, recent TinyML models tend to use large input images and intermediate tensors to raise their training accuracy (2.) Such TLnyML models expose grand challenges when deploying these models on MCUs with the stringent memory constraint.
Tensor memory planner of ML system frameworks determines the usage of the SRAM memory. An MCU is often responsible for Machine Learning (ML) inference and often uses a single batch to facilitate the response time of MI inference tasks. Thus, the static tensor memory planner schedules intermediate tensors to meet the SRAM memory constraint. Conventionally, the compiler/interpreeer employs wise scheduling, pushes data of layer in the SRAM memory, and replaces them after finishing its operator (3 4). However, such layer-wise inference will overuse the SRAM memory space when TinyML models have large intermediate tensors. To overcome this shortcoming, patch-
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

based inference (2) divides the entire tensors into multiple overlapping patches. Then, patch- based inference places a patch in the SRAM memory space rather than the entire tensor to repair the out-of-memory problem. However, the redundant computation of patch-based inference occurs when proceeding with a large number of overlapping patches and tremendously increases the execution time of MInyML. models. Furthermore, this situation becomes worse in particular when significantly shrinking the usage of SRAM memory space. Hence, patch-based inference is becoming undesirable with such notoriously high performance overhead on MCUs.
patch-based
This work designs StreamNer that eliminates the significant amount of redundant computations while reaching minimal usage of the SRAM memory space for TinyML models on MCUs. StreamNet creates the stream buffer to reuse tensor data in the patch-based inference. In addition, StreamNet designs ID and 2D stream processing to improve the performance of patch-based based inference with different SRAM memory requirements. To facilitate the performance of StreamNet further, StreamNet also removes the unnecessary computation on the padding data. At last, StreamNet's parameter selection framework automatically constructs a new patch-based inference that mixes ID and 21) stream processing to meet the memory constraint while resulting in the best performance for TinyML models.
The main contributions of this paper are shown below:
1. StreamNet achieves geometric mean of 3.4X (ID) to 7.3X (2D) speedup over the existing patch-based inference without increasing memory usage significantly.
2. Developing output padding bypass kernels enhance the performance of ratch-based inference.
3. The StreamNet parameter selection framework automatically selects optimal configura- tion for StreamNet-2D operations under the SRAM memory constraint an MCU.
2 Understanding Performance Bottleneck of Memory Allocation on TinyML
Memoor--fficient patch-based inference: Patch-based inference aims to reduce the peak SRAM memory usage of TinyML models on MCUs. This patch-based approach divides large input feature map into multiple small patches. Then, the MCU's SRAM buffer reduces its space usage by only storing small patch each time instead of the entire input feature map. For instance. Figure [ aa) demonstrates the peak SRAM memory usage of the proxyless-wO.3 TinyML model (5) and other TL models have the similar trend. Traditionally, ML system frameworks such as TFLM 3 place input and output tensors of a layer or operator in the SRAM buffer. After finishing the work of layer, TFLM (3) removes the obsolete input data of the completed layer and allocates another space to put the output data of the next layer. However, given an MCU and its size of SRAM memory is 128 KB, Figure I demonstrates that the memory usage of multiple layers in the proxyless-wO.3 ML model (5) is larger than the size of the SRAM memory on an MCU. As a result, the layer-wise memory allocation raises significant challenges when deploying MI, models MCUs. Unlike the layer-wise inference, Figure Satureeppeeeettee presents patch-based inference that reduces the total SRAM memory usage of TinyML models and enables models to be run on MCUs with small memory size.
242kB
The en MADs (M)
Pask Memary Usage (48)
250
33x
200
Per-layer Peak Memory
Per-leyer Memorr
Per-patch Memory
100
Peak MAC
Per-patch
Pen-layer MAC
Per-petich MAC
150
128KB
75
100
0
3
8
9
10
11
2
3
7
10
11
Layer Index
Layer Index
(8) peak memory usage cascade policy over the baseline
(b) The amount of MAC in cascade pollcy over the baseline
Figure I: Comparing peak memory usage and the amount of MACs on patch-based and layer-wiss wise inference
Challenges and opportunities: Pateh-based inference creates a large amount redundant computa- tion that occurs among its adjacent patches. To reduce the peak SRAM memory usage MCUs,

---

patch-based inference creates many small patches that overlap receptive fields. These small patches tremendously raise the amount of computation on TinyML model. The inefficient patch- based inference is becoming undesirable on MCUs even if it saves the peak SRAM memory usage of TinyML. model significantly. Figure (T)) demonstrates that patch-based inference takes 33X more MACs than the layer-wise one at the first layer. Furthermore, the amount of MACs in aatch-based inference also rapidly grows with the reduction of peak SRAM memory usage. Hence, our work stores repeated results of patch-based inference in the stream buffer and skips these computations to eliminate the performance bottleneck of patch-based inference.
3 StreamNet: Efficient Patch-based Inference for TinyML Models
This section details the design and implementation of StreamNet that removes the redundant compu- tation of patch-based inference with the minimal usage of the SRAM memory space on MCUs.
Operator
Computed
Patch 0
Patch 1 CONV
Patch 2
Patch 3
[Ovelaapping area
CONVO CONV1
CONV1
CONV
CONV1
(a) Step & 2
(b) Step 3
(c) Step
Figure 2: Patch-based inference example
3.1 Inefficient Patch-based Inference
To accelerate patch-based inference, this work proposes StreamNet that removes most redundant computations of patch-based inference by putting its repetitive results in the stream buffer. Figure demonstrates operations of patch- based inference. In Figure there are two operators where each of them has its own input and output tensor and the computed block means the completed output element. Conventionally, in the convolution processing, the 6 6 input with the 3 3 filter will yield the 4 x output. This calculation takes 144 (16 x 9) MACs where the number 16 is the size of the output and the number comes from the 3 filter. Unlike conventional convolution, in Figure 2 pateh-based inference divides a x 6 input into four 5 x 5 patches. Each 5 x 5 patch computes the convolution with the x 3 filter finally yield 4 4 output. As illustrated in Figure 2 a),, Patch and Patch generate a 3 x 3 output patch, respectively. Each of them takes 9 x 9 MACs where the first number 9 is the size of output and the second number 9 is associated with the size of the filter. Thus, patch-based inference uses 4 patches to create a 4 x 4 output and takes 324 MACs (9 x 9 x 4). The amount of MACs of patch-based inference is 2.25X (324/146 44) more than the layer-wise inference because patch- basch-based inference contains an enormous amount of repetitive computations. For instance, in Figure 2, each value in the overlapping area represents the repetition count of the output tensor element. This redundant calculation increases the execution time of a TinyML. model significantly. In addition, patch-based inference tends to use the fine-grained patch to lower the usage of the SRAM memory. However, the amount of redundant computation increases as the patch size becomes small.
3.2 StreamNet ID Streaming Processing
Figure 3(a) (a) illustrates the steps of ID streaming processing on StreamNet inference. In Figure 3 a). two patches proceed one after the other the data stream during the computation of the convolution. In Figure 3 the computed block represents the completed element of the tensor, and the computing block indicates the elements of the tensor in the execution. At first, in Vigure(Jja), a)), StreamNet-ID performs the CONVO operator on the 5 X 5 patch and yields a 3 x 3 output tile StreamNet- ID stores the overlapping area of this x 3 output tile in the stream buffer StreamNet-1D 1D uses this 3 x 3 tile to generate the 1 x output data after finishing the CONVI operator Unlike patch-based inference, StreamNet uses the results preserved the stream buffer during the computation of Patch 1. Thus, StreamNet- ID only fetches 3 x 5 tile from Patch to yield
Second, Then,
3

---

Patch 0
Patch 1
Computed
Computing
Operator
Overlapping
CONVO
CONV
(b) Patch-based inference
Stream
StreamNet-1D
Buffer
Inplace Concat
(a) StreamNet-10 streaming processing
(c) Stream result
Figure 3: The operation StreamNet-ID
3 x output data Then, StreamNet- ID combines this 3 1 output with the data of the stream buffer to work out a 3 x 3 output tile as illustrated Figure 3 a) ( At last, the 3 x 3 output of In the computation of Patch 0 and Patch 1, StreamNet-1D takes 108 MACs (12 x 9) where the number 12 the sum the calculated output size in (3 3) and (3 x 1) and the number is associated with the size of the 3 x filter. Since StreamNet- ID also needs to use four patches to create 4 4 output in total, the total amount of MACs in ID streaming processing of StreamNet-ID is 216 (12
Patch will yield 1 output after finishing the CONVI operator
2). Unlike patch-based inference in Figure b), StreamNet-ID in Figure 3(c) decreases the repetition counts in each element of the output tensor. Hence, StreamNet-IDD reduces roughly 60% of MAC overhead over patch-based inference in Figure 3, because the overhead of StreamNet- -D and patch-based inference (24 16) and 20 (36- 16) where 16 (4 is the number of output elements of the layer-wise inference.
Operator
Patch 0
Patch 1
Patch
Patch 3
width
CONVO
Stream Buffer
CONV
height
Inplace
Conca
Stream Buffer
Stream
Stream Buffer
Inplace
Concat
Stream Buffer
Inplace Concat
stream
stream + stream
Figure 4: The StreamNet-2D operations
3.3 StreamNet 2D Streaming Processing
StreamNet uses the stream buffer to preserve temporary results that will be used multiple times across patches. After finishing an operator, StreamNet will place number of output tensors in the stream buffer. Since StreamNet- DD does not completely remove the redundant computation overhead, StreamNet designs StreamNet-2D to eliminate all redundant MACs in patch-based infer- ence. StreamNet- 2D is composed of stream and stream_y processing. The exccution order of

---

stream patches is firstly moving downward. For instance, antch-based inference in Figure moves its execution vertically for Patch 1, then back to Patch 2, and finally moves downward for Patch 3. Unlike stream_y. the execution order of stream patch firstly moves toward the horizontal direction. At first, in Figure[4] the x Patch computes CONVO with the x 3 filter and yields x output. Then, StreamNet-2D stores 3 x 2 tile and 2 x 3 tile of the 3 x 3 output in the Y and X stream buffer O respectively. Second, StreamNet-2DD tackles Patch and fetches a 5 x tile in Patch I to yield a x 1 output. After finishing Patch 1, StreamNet stores a 2 X 1 tile of its output tensor to the XY stream buffer
and cleans the stream buffer. Third, StreamNeet-2D handles Patch 2 and reuses results in the X stream buffer. After completing Patch 2, StreamNet-2D stores 3 2 tile of Patch 2's output in the Y stream buffer (8) ). Finally, StreamNet-22 reuses the data within the Y and XY stream buffer to proceed with Patch 3. StreamNet-22D takes 144 MACs ((9 (9+3++ 3++3++ 1) x 9) where each number of the first bracket represents the amount of calculation in each patch and the second number 9 means the size of the 3 x filter. Hence, StreamNet-2DD completely eliminates the total amount of the redundant computation shown in patch-based inference. However, Stream 2D has a larger stream buffer store the temporary results of each patch over StreamNet-1D. As illustrated in Figure 4 the stream buffer of StreamNet-2D is 14 where the 2 x 3 tile is from stream_ X, the 3 x 2 tile from stream y, and the 2 X tile is from the XY stream buffer. large stream buffer increases the requirement of the SRAM memory space. To balance the consumption of the MACs and the SRAM memory space. StreamNet presents an parameter selection framework that mixes StreamNet- ID and StreamNet-22D As result, StreamNet can satisfy the constraint of the SRAM memory size on MCUs while achieving efficient execution on TLLyML models.
Algorithm 1:
The StreamNet Parameter Selection Algorithm
Input: model and constraint
Output: selected StreamNet parameter under constraint
patch_pprarn { {splil__idx, (l_pcch) splil_idr jix ValidSppial (Moees)
"_patch E OutSSapee (Layers
candidates []
3 for param in paathh_paraai do per formance (mem, M Aacchhheoooiiierromttrt/ttdddood if' mem meets constraint then candidates. apppead (param, per p formance)
Filter obtained records and only keep the ones that have per formance on the Pareto frontier
8 for candidate in candidates do for stream_x_level in 10, ..., record.splüt_ idx] do param = (candidate.parant per formance (mem, MAC's) PrrofileParamete Sarerreeeeeeeeeeeaaeeeeee Strream-et((prran) if mem meets constraint then
10
mar ma.r_stream level, stream_r x_level)
11
12
13
new www.idates candidates..apendd pppenddppram, per f formance)
if there only one param that has the smallest M AC's then return param as the best parameter.
15
else
17
return the param that has the largest output patch size as the best parameter.
3.4 StreamNet Parameter Selection Framework
Searching for the best parameter that satisfies memory constraints while minimizing the latency of a TinyML model manually is exceedingly tedious and challenging. To address this problem, StreamNet designs parameter selection framework that determines the best parameter candidate to meet the constraint of the SRAM memory across different MCUs. Hence, StreamNet parameter selection framework aims to automatically find the parameter composition of StreamNet and mixes StreamNet- ID and StreamNet-2D to fit the limited amount of SRAM memory on the MCU. To minimize the runtime overhead, StreamNet refers to the MAC count to choose the best candidate of StreamNet parameters through its compiler pass.
Algorithm | demonstrates StreamNet parameter selection framework that infers parameters of StreamNet to fit an MCU at the compile time. The patch parameter line 1 of Algorithm [] includes the number of the patch (n patchtch) and the number of layers that proceed with batch-
5

---

computation (split idx). The OutShapeFactor function returns the valid number of the patch in patch-based inference. Next, StreamNett calculates the amount of MACs with different combinations of these two parameters and picks out candidates that meet the constraint of the SRAM memory space of an MCU from line 7 of the Algorithm[]] [ Then, StreamNet examines different values of stream_ level and stream @eeaam___leee _y_level to determine the number of stream- werse layers while having minimum amount of MACs in a model and returns this combination from line 15 of Algorithm Otherwise, StreamNet tends to choose the one with the smallest patch count when both candidates have the same amount MACs, since a large output patch has bigger computation granularity, which usually yields better latency.
Operator
Tensor area CONV
Patch area
Padding area
StreamNet skipping area
CONV1
CONV2
Tensor A
Tensor B
Tensor C
Tensor D
Figure 5: Output padding data bypassing on StreamNet
3.5 Output Padding Bypassing
The padding data of each tensor contributes the performance overhead in the patch- inference. To further the performance of patch-based inference, StreamNet intelligently skips the computation of the padding data. Figure 5 presents a patch-based inference neural network that consists of operators and 4 tensors. Padding is always added to a tensor to generate an output with the specified size. Parameters conventional neural networks such as the size of the stride and the filter result in different amounts of padding data in each tensor. Unlike common neural networks, in Figure 5 the patch size of the current tensor is obtained from the configuration of the next tensor. For instance, patch-based inference refers to the patch size of Tensor C to figure out the patch size of Tensor B. The padding data in Tensor B appends the one in Tensor C. Thus, the amount of the padding data in batch-based inference can be more than the one in the layer- layer-wise inference. Since the value of padding data is always assigned to be zero, the padding data won't t change the expected results. In Figure 5] patch- based inference proceeds with 7 x patch in Tensor A including padding data to generate an output x patch. Next, this 5 x 5 patch in Tensor is used to create a 3 x 3 patch in Tensor C. Hence, each operator computes its padding data that contributes additional MACs during the execution of Mi models. StreamNet removes the performance overhead shown in the padding data by skipping the computation of each output padding data. In Figure patch-based inference reads the data 5 x 5 patch in Tensor B to generate 3 x 3 patch in Tensor B. Unlike patch-based inference. StreamNet uses the size of the padding data known after finishing the previous layer to distinguish the padding data from the real one. Next, StreamNet only considers the real data in the calculation of an operator. Furthermore, StreamNet only writes the zero value into output padding data directly without consuming any MACs. StreamNet changes the implemented kernel of aatch-based inference skip the computation of the output padding data during the execution of each operator.
4 StreamNet System Architecture
The system architecture of StreamNet contains the frontend and backend processing. In the frontend system of StreamNet, the input of StreamNet is the TensorFlow Lite model (.tflite file). Then, StreamNet uses the parameters of the TinyML model to create layers of patch-based inference (2). Second, the memory scheduler allocates SRAM memory space for tensors on the static arena memory space and then StreamNet performs the ID and 2D stream processing with its stream buffers. In StreamNet's s backend system, StreamNet translates tensors and operators of the patch-based inference

---

TE Kernels: TinyEngine Kernels
StreamNet
BP Kernels: Bypass padding Kernels
Unchanged
Kernel Library
Stream patch Generation +
CMMSIS-NN
TE Kernels
BP Kernels
.tflite
Patch generation
patched
Target Platform
Mbed CLI SDK
ARM Cortex- M7
patch params (n_patch, split_ _idx)
stream params (stream _ level,
Stream Patch Code Gen
yreeamyyleevl))
+
Patch Code Gen
Patched Layers
code
Figure 6: The system architecture of StreamNet
into source code and links kernel libraries such ARM CMSIS-NN G), TinyEngine (1. and our bypass padding kernels. Finally, StreamNet deploys TLnyML models on the MCU by using Mbed CLI SDK 7..
5
Experiments
5.1 Experiment Environment Setup
System Implementation: We compare the performance and memory consumption of StreamNet to MCUNetV2 (2). which performs patch-based inference on TinyML models. Since existing TinyEngine kernels ① used by MCUNetV2 (2) do not support 5 and 7 filters, fix this issue to enable our evaluation to work on the depthwise convolution function with more filter combinations. In addition, we also insert our output padding bypassing implementation within MCUNetV2 (2) as MCUNetV2+. StreamNet modifies MCUNetV2's 's memory scheduler and code generator to identify streamable area, allocate stream buffer, generate codes with different streaming processing, and insert the stream buffer operation between operator kernel invocation in StreamNet.
Benchmark: We evaluate 10 MInyML models from MCUNetV2 model (5succ mbv2- wO.35(MBB). proxyless-w0.3(PM) mnunet-vwww((MVO) mcunet-vww (iNee--Vwww(MV1))) (MV1), mcunet-vwW((MV22) mcunet-inO(MIO)) mcunet-in (MII), mcunet-in2(M112), mcunet-in3(MII3) mcunet-in((MI4), All models use int8 quantized mode. These TinyML models have different requirements on the SRAM memory space and their execution time also differs.
On-device Experiment: We deploy TinyML models through ARM Mbed CLI 7 on the ARM M7 CPU. The MCU used in our evaluation is STM32F767ZI (8) that includes ARM Co CPU at 216 MHz, a 512KB SRAM, and a 2MB Flash
Performance Analysis when Consuming Nearly Equal Memory
Table I compares the performance of StreamNet and MCUNetv2 when both of them use nearly the same SRAM memory space. Table presents the data of StreamNet-2D. Then, we scan through value of n_patch and split_ids parameters to search for the one where their SRAM memory usage closest to StreamNet-225 In Table [ StreamNet achieves a geometric mean of 5.1 speedup over 2CUNetv2. The SRAM memory consumption varies with the change of n patch and split idx parameters on MCUNetv2. However, the amount of MACs increases rapidly with the reduction of SRAM memory usage MCUNetv2. As result, StreamNet has better performance than MCUNetv2 when consuming nearly equal SRAM memory.
5.3 The Variation of MAC Counts
Table compares the amount of MACs used in each MI model. In Table 2 the value of the n_ p_paach and split_idx idx determines the number of patches and the number of patch-based layers used in model, respectively. StreamNet scans through all combinations of the patch and split_id idx and

---

Table 1: Performance results of MCUNetv2 and StreamNet-2D when consuming nearly equal
memory. Param (n (n_patch, split_idx), Memory (KB), MAC (millions), Latency (ms)
StreamNet-2U
MCUNetV2
Speedup
Model
Param
Mem
MAC
Latency
Param
Mem
MAC
Latency
MAC
Latency
MB2
(38,13)
66
24
417
(18,12)
63
77
1,513
3.29
3.63
PI.
(22,13)
95
38
676
(11,21)
89
549
9,519
14.31
14.08
MIO
(12,06)
30
99
(04,14)
29
19
274.441
2.99
2.77
MII
(12,11)
47
188
(03,12)
48
24
366
1.84
1.95
MI2
(20,21)
169
67
1.168
(05,25)
169
463
7,055
6,87
6.04
MI3
(22,21)
208
82
1,444
(11.32)
215
2,777
43,812
33.89
30.35
MI4
(20,17)
236
1,762
(04,21)
214
366
4,613
2.62
MVO
(08,13)
29
101
(08,11)
30
17
296
2.83
2.93
MVI
(20,14)
44
12
225
(05,17)
45
47
810
4.04
3.61
MV2
(18,21)
143
961
(06,29)
144
518
8,311
9.27
8.65
GMEAN
5.33
5.11
Table 2: The amount of MACs (Millions) used in StreamNet and MCUNetv2
Speedup over MNv2
Model
patch
split idx
MNv2
MNv2+
SN-1D
SN-2D
MNv2+
SN-1D
SN-2D
MB2
18
13
77.4
73.1
36.9
23.5
1.06
2.10
3.29
PL
349.6
326.5
88.6
38.4
1.07
3.95
9.11
MIO
12
06
15.5
14.4
8.5
6.4
1.08
1.82
2.43
MII
12
11
27.0
25.4
16.6
12.8
1.06
1.62
2.11
MT2
20
21
998.8
920.2
189.2
67.4
1.09
5.28
14.82
MI3
21
1,784.7
1,650.7
271.7
81.9
1.08
6.57
21.78
MI4
20
17
1,942.1
1,787.8
353.8
126.0
1.09
5.49
15.41
MVO
08
13
23.1
20.0
9.7
6.0
1.15
2.39
3.87
MVI
89.1
24.2
11.6
1.08
3.97
8.28
MV2
18
21
826.8
754.3
153.3
55.9
1.10
5.39
14.79
GMEAN
1.08
3.45
7.18
finds the one that can result in the minimal usage of SRAM memory space in patch-based inference. Thus, results in Table show the amount of MACs used in the TinyMI. model with the lowest peak SRAM memory usage through all possible combinations of p_patch and split_idx. The amount of the MACS in patch-based inference increases with the reduction of the usage the SRAM memory. Therefore, MCUNetv2 (MNv2) obtains a significant amount of the MACs over the other alternatives Table 2 In addition. MCUNetv2+ (MNv2+) reduces roughly 8% of MACs on average over the baseline MCUNetv2. The output padding bypassing of MCUNetv2+ removes overhead on the padding data and achieves such a reduction on the amount of MACs. Moreover, StreamNet- 1D (SN- ID) performs ID stream processing and saves the redundant computation through its stream buffer. Thus, StreamNet- ID reduces 65% of MACs on average over the baseline. To completely remove the redundant computation on patch-based inference (MCUNetv2). StreamNet-2D (SN-2D) performs 2D streaming processing to increase more data reuse than StreamNer- ID. Hence, StreamNet-2D reduces 83% of MACs on average over MCUNetv2 across 10 TinyML models. StreamNet-2D saves 93% and 95% of MACs on MI2 and MI3, since these models consist of many patch- layers with the large split_ idx value and the amount of MACs on the redundant computation is large.
5.4 Reveal the Latency of StreamNet
This evaluation aims figure out if StreamNet can shorten the execution time of TinyML models by reducing the amount of their MACs. Table compares the latency of each TinyML model across different implementations on MCUNerv2 and StreamNet. The n_patch and split_ _dittidxx____ in Table
and and Table use the same value. Table shows the execution time when TinyML. models use minimum SRAM memory space through patch-based inference. MCUNetv2+ (MNv2+), StreamNet- ID (SN-ID), and StreamNet-2D (SN-2D) achieves geometric mean of 1.IX, 3.4X, 7.3X speedup

---

Table 3: The latency (ms) of TinyML models on StreamNet and MCUNetv2
Speedup over MNv2
Model
MNv2
MNv2+
SN-1D
SN-2D
MNv2+
SN-1D
SN-2D
MB2
1,514
1,391
687
417
1.09
2.20
3.63
PL
6,296
5,795
1,610
676
1.09
3.91
9.31
MIO
238
219
131
99
1.09
1.81
2.40
MI1
449
413
258
188
1.09
1.74
2.39
MI2
16,545
15,055
3,281
1,168
1.10
5.04
14.17
MI3
29,700
27,331
4,858
1,444
1.09
6.11
20.57
MI4
26,816
24,747
5,153
1,762
1.08
5.20
15.22
MVO
415
355
169
101
1.17
2.45
4.10
MVI
1,817
1,667
473
225
1.09
3.84
8,09
MV2
14,106
13,040
2,778
961
1.08
5.08
14.68
GMEAN
1.10
3.40
7.28
over the baseline MCUNetv2. The speedup Table presents the consistent results over the amount of MACs in Table 2 Hence, this result provides StreamNet the evidence to exploit the MAC count obtained compile time infer the final latency in StreamNet auto-tuning framework.
Layer-wise MCUNetv2 ( StreamNet- 1D StreamNett-2 2D
mem- 0%
mem-5%
mem+100%
2 mem-15%
Peak Memory Usaga (KB)
416.1
Performance Spocduu
over MCLNNN2 N
MB2
MI0
MIl
MI2
MI3
MI MVO MV1 MV2MEAN
M62
MIC
MIl
MI2
MI3
MI
MVG MV1
MV2 GMEAN
TinyML Models
TinyMI Models
(a) SRAM memory consumption
(b) Auto-tuning framework
Figure 7: Results of SRAM Memory consumption and StreamNet's auto-tuning framework
5.5 Analysis of Peak SRAM Memory Usage
Figure 7 a) presents the peak SRAM memory usage of TinyML models different implementations of MCUNetv2 (2) and StreamNet. MCUNetv2 (2). StreamNet-1D, and StreamNer-2D save 66%, 62%, and 47% SRAM memory usage on average Unlike pateh-based inference, StreamNett ID reuses the data within the stream buffer that occupies the SRAM memory space. Thus, StreamNet- ID consumes additional 4% SRAM memory space on average to store the stream buffer over the MCUNetv2 (2). To increase the ratio of the data reuse, StreamNet-2D1 requires about 22% extra SRAM memory space on average for the stream buffer to cover more data reuse than StreamNet- ID. StreamNet provides trade-off between the performance and the peak usage the SRAM memory. Unlike patch-bases inference, StreamNet improves the performance of ML models without sacrificing too much SRAM memory space.
the layer-wise inference baseline, respectively,
5.6 Results of StreamNet Auto-tuning Framework
This evaluation aims to validate if StreamNet auto-tuning framework can create case that satisfies its memory constraint while achieving the best performance. Figure ( presents the performance speedup results when adding 0% 15% memory space on each TinyML model based on the minimal SRAM memory usage of StreamNet- ID. In addition, we also design auto-tuning framework for MCUNetV2 (2) that scans through its n a_paatch and split idx parameters to obtain case with the best performance while meeting the memory constraint. Unlike MCUNetV2 (2) auto-tuning framework, StreamNet searches for the combination of StreamNet- and StreamNet-2D network to fit memory constraint while achieving the best performance. As illustrated in Figure/7bb StreamNet achieves a geometric mean 1.63X, 1.61X, 1.8X speedup over MCUNetV2 (2) when the memory constraint the base memory space plus 5%, 10%. and 15%, respectively. In Figure 7b)) MIO

---

achieves only a speedup of 1.07X to 1.09X over MCUNetV2 (2) where the value of the split idx and n patches is small, leading to minor recomputation in MCUNetV2 (2). Unlike MI0, MI3 uses larger split_ide idx and n o_patches values to meet the memory constraints. The large value of split_idx and n patches allows StreamNet eliminate the significant amount of redundant computation.
Related Work
Tiny Machine Learning (TinyMI.) TinyMI. has gained significant attention in recent years.
Techniques such as model compression (9 (0), pruning 12 13 14 (5) and quantization (II
16
17
18
19
20 21 22), employed reduce model size and complexity while maintaining
accuracy. Existing frameworks like TFLM (3) and microTVM (4) optimize the execution time of
TinyML models on resource-constrained devices by utilizing CMSIS-NN (6) and CMix-NN (23) libraries for kernel implementation. MCUNet (I) further optimizes kernel implementation through vectorization, loop reordering, loop unrolling, and data layout transformation, resulting in more efficient performance.
Memory-Efficient Inference The peak usage of SRAM memory is critical factor in deploying TinyML models on microcontrollers due to the tight SRAM constraints in MCUs. However, existing works 3 24 25) employ per-layer inference, which restricts model deployment on severely memory- constrained MCUs. Several approaches have been proposed to address this issue in different ways. Some methods rearrange operators (26 27 or use tensor memory planners (28) to save memory usage. Nonetheless, a single intermediate tensor may still be too large for the MCUs. As an alternative, models can be rewritten to reduce memory requirements (29) 30). However, this requires network modifications and may affect accuracy, reducing adaptability. MCUNetV2 (2) and TVM's cascade scheduling (3I) address this problem by fusing multiple layers and introducing redundant computation. Prior work (32 33 34 35; 4 also tackles similar problems as MCUNetv2 (2) in the ML inference. However, too many fused layers increase the amount of redundant computation.
Remove Recomputations To address recomputations introduced by the aforementioned approach, the main idea is to cache the computed result and reuse it when necessary. Halide (35) introduces sliding window optimization to remove recomputation, but it lacks flexibility in adjusting temporary buffers to reduce memory requirements across different dimensions. Prior work such as Fused-layer 40 41) offer solutions to reduce execution time in the data streaming manner, but none of them provides a method to mix ID and 2D data streaming. Hence, our proposed StreamNet builds upon and extends existing streaming approaches by introducing a hybrid method that mixes different dimensional streaming to form ID and 2D streaming techniques, skips the redundant computation shown in patch-based inference (2), and offers an auto-tuning framework to select StreamNet configurations.
CNN accelerators (36) and streaming methods (37 38 39
7 Conclusion
Deploying TinyML. models exposes challenges on MCUs with the stringent memory and power constraints. Unlike layer-wise inference, patch-based inference significantly decreases the usage of the SRAM memory and helps the deployment of MinyML models on MCUs. However, the redundant computation of patch-based inference enormously raises the latency of Machine Learning inference. Thus, this work proposes StreamNet that aims to eliminate such a performance bottleneck of patch- based inference. StreamNet designs ID and 2D streaming processing significantly remove the performance overhead of patch-based inference. Thus, StreamNet-21 achieves geometric- -ean speedup of 7.3X over patch-based inference across 10 Ti models. Furthermore, StreamNet also skips the unnecessary computation for the padding data. Finally, StreamNet provides an auto-tuning framework automatically search for the network that can satisfy the memory constraint of MCUs while achieving better performance over the existing patch batch-based inference.
8 Acknowledgement
The authors gratefully acknowledge the support from the National Science and Technology Council in Taiwan under grant number
10

---

References
[1] J. Lin, W.-M. Chen, Y. Lin, C. Gan, and S. Han, "Meunet: Tiny deep learning on iot devices," Advances in Neural Information Processing Systems, vol. 33, 2020.
[2] J. Lin, W.-M Chen, H. Cai, C. Gan, and S. Han, "Mcunetv2: Memory-efficient patch-based inference for tiny deep learning," in Anmual Conference on Neural Information Processing Systems (NeuralPS), 2021.
[3] R. David, J. Duke, A. Jain, V. Janapa Reddi, Jeffries, Li, N. Kreeger, Nappier, M. Natraj. T. Wang, et al., "Tensorflow lite micro: Embedded machine learning for tinyml systems," Proceedings of Muchine Learning and Systems (MLSys), vol. 3, pp. 800-811, 2021.
[4] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, M. Cowan, H. Shen, L. Wang, Y. Hu, L. Ceze, et al., "Tvm: An automated end-to-end optimizing compiler for deep learning," " arXiv preprint arXiv:1802.04799 2018.
[5] Lin, Ji and Chen, Wei-Ming and Lin, Yujun and Gan, Chuang and Han, Song, "MCUNet Model Zoo." " [Online]. Available: https: //tpsbb com www/mitttmmeooeecccooeteeoooo hwwp/aaaeeeelloo/eeaaeaaeeee mcunet/model wttpe//tmmmeeeoooooeoooo zoo. py 2023. (accessed May. 16, 2023).
[6] L. Lai, N. Suda, and V. Chandra, "Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus," arXiv preprint 1xXiv:100.06600 2018.
[7] ARM Ltd., "Arm Mbed CLI." [Online]. Available: https://gitthbb com.Ri/bbe.om//mmbee.aa mbed- cli 2023. (accessed May. 16, 2023).
[8] STI Ltd., "STM32F7677I Datasheet." " [Online]. Available: mi aicrootttoleeerrccoopppooor////p html 2023).
commeoocoeeeeeeeee.e.ee en/ 2023. (accessed May. 16,
[9] Y. Hc, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, "Ame: Automl for model compression and acceleration on mobile devices," in Proceedings of the European conference on computer vision (ECCV), pp. 784 784-800, 2018.
[10] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, "Eie: Efficient inference engine on compressed deep neural network," in Proceedings of the International Symposium on Computer Architecture (ISCA), p. 243-254, 2016.
[11] Han, H. Mao, and W. Dally, "Deep compression: Compressing deep neural networks with pruming, trained quantization and huffman coding," arXiv preprint arXiv: 2015.
[12] Lin, Y. Rao, J. Lu, and Zhou, "Runtime neural pruning," Advances in neural information processing systems, vol. 30, 2017.
[13] Y. He, X. Zhang, and J. Sun, "Channel pruning for accelerating very deep neural networks," in Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
[14] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, "Learning efficient convolutional networks through network slimming." in Proceedings of the IEEE international conference computer vision, pp. 2736-2744, 2017.
[15] Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K.-T. Cheng, and J. Sun, "Metapruning: Meta learning for automatic neural network channel pruning," in Proceedings of the IEEF/CVF international conference on computer vision, pp. 3296-3305, 2019.
[16] H. F. Langroudi, V. Karia, T. Pandit, and D. Kudithipudi, "Tent: Efficient quantization of neural networks on the tiny edge with tapered fixed point." arXiv preprint arXiv:2104.022. 00433003.,,,,... 2021.
[17] M. Rusci, A. Capotondi, and L.. Benini, "Mrvory-driven mixed low precision quantization for enabling deep network inference on microcontrollerrs" Proceedings of Machine Learning and Systems, vol. 2, pp. 326-335, 2020.
11

---

[18] J. Choi, Z. Wang, S. Venkataramani, P. I.-J Chuang, V. Srinivasan, and K. Gopalakrish- nan, "Pact: Parameterized clipping activation for quantized neural networks," " arXiv preprint arXiv: arKiv:1805.000 1805.0008, 2018.
[19] K. Wang. Z. Liu, Y. Lin, J. Lin, and S. Han, "Haq: Hardware-aware automated quantization with mixed precision," " in Proceedings of the IEEE/CVF Conference on Computer Vision and 2019.
Pattern Recognition, pp.
[20] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, "Xnor-net: Imagenet classification using binary convolutional neural networks," in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pp. 525- 542, Springer, 2016.
[21] S. Zhou, Y. Wu, Z. Ni. X. Zhou, H. Wen, and Y. Zou, "Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients," " arXiv preprint arXiv:1606.0611 2016.
[22] C. Zhu, S. Han, H. Mao, and W. J. Dally, "Trained ternary quantization," arXiv preprint 1611v:1612.01066 2016.
[23] A. Capotondi, M. Rusci, M. Fariselli, and Benini, "Cmix-nn: Mixed low-precision enn library for memory-constrained edge devices," IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 67, no. 5, pp. 371-875, 2020.
[24] N. Rotem, J. Fix, S. Abdulrasool, G. Catron, S. Deng, R. Dzhabarov, N. Gibson, J. Hegeman, M. Lele, R. Levenstein, et al., "Glow: Graph lowering compiler techniques for neural networks," arXiv preprint arXiv: 00997, 2018.
[25] H.-L C. Liu, M. Brehler, M. Ravishankar, N. Vasilache, B. Vanik, and S. Laurenzo, "Tinyiree: An ml execution environment for embedded systems from compilation to deployment," arXiv preprint arXiv: 14479, 2022.
[26] E. Liberis and N. D. Lane, "Neural networks on microcontrollers: saving memory at inference via operator reordering," arXiv preprint arXiv:1910.051 10, 2019.
[27] B. H. Ahn, J. Lee, J. M. Lin, H.-P. Cheng. Hou, and H. Esmaeilzadeh. "Ordering chaos: Memory-aware scheduling of irregularly wired neural networks for edge devices," Proceedings of Machine Learning and Systems, vol. pp. 44-57, 2020.
[28] Y. Pisarchyk and J. Lee, "Efficient memory management for deep neural net inference," arXiv preprint arXiv:2001.00 1032888 2020.
[29] T. Jin and S. Hong, "Split-cnn: Splitting window-based operations in convolutional neural networks for memory system optimization," in Proceedings of the International Conference on Architectural Support for Programming Languages and Operation Systems (ASPLOS), pp. 835 847, 2019.
[30] O. Saha, A. Kusupati, H. V. Simhadri, M. Varma, and P. Jain, "Rnopool: Efficient non-linear pooling for ram constrained inference," Advances in Neural Information Processing Systems, vol. 33, pp. 20473-20484, 2020.
[31] ARM Itd, "Miniaturizing Models for microNPUs: a Cascading Scheduler." [Online Available:] https: ://::///wwwww/ www.ww...ubb.... www.yyouu.oooo/tttttttt http:///////aaattt///aooooaoa/a
2022. (accessed Jan. 20, 2023). [32] J. Zhao and A. Cohen, "Flextended tiles: A flexible extension of overlapped tiles for polyhedral compilation." ACM Trunsactions on Architecture and Code Optimization (TACO), vol. 16, no. 4, pp. 1-25, 2019.
[33] X. Zhou, J.-P. Giacalone, M. J. Garzarán, R. H. Kuhn, Y. Ni, and D. Padua, "Hierarchical overlapped tiling," in Proceedings of the Tenth International Symposium on Code Generation and Optimization, pp. 207-218, 2012.
[34] K. Goetschalckx and M. Verhelst, "Breaking high-resolution enn bandwidth barriers with enhanced depth-first execution," IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9, no. 2, pp. 323-331, 2019.
12

---

[35] J. Bagan-Kelley C. Barnes, A. Adams, S. Paris, F. Durand, and S. Amarasinghe, "Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines," Acm Sigplan Notices, vol. 48, no. 6, pp. 519-530, 2013.
[36] M. Alwani, H. Chen, M. Ferdman, and P. Milder, "Fused-layer enn accelerators." in 2016 49th Annual JEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 1-12, IEEE, 2016.
[37] O. Rybakov, N. Kononenko, N. Subrahmanya, M. Visontai, and S. Laurenzo, "Streaming keyword spotting on mobile devices," " arXiv preprint wrXiv:2005.06720, 2020.
[38] A. Coucke, M. Chlich, T. Gisselbrecht, D. Leroy, M. Poumeyrol, and T. Lavril, "Efficient key- word spotting using dilated convolutions and gating." in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1551-6355, IEEE, 2019.
[39] Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Wu, R. Pang, et al., "Streaming end-to-end speech recognition for mobile devices," " in ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 638 1-385,,,,, IEEE, 2019.
[40] X. Chen, Y. Wu, Z. Wang. S. Liu, and J. Li, "Developing real-time streaming transformer trans- ducer for speech recognition large-scale dataset," in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9904-5908, IEFE, 2021.
[41] Y. Wang, H. Lv, D. Povey, L. Xie, and S. Khudanpur; "Wake word detection with streaming transformers," ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5864-5868, IEEE, 2021.
13


Reviews:
[{"review_id": "8vtTmKVzgQ", "reviewer": "Reviewer_bp8j", "summary": "The processing of patch-based inference for MCUs induce a large number of redundant MACs against the layer-wise processing because of the overlapped processing. In order to address this problem,  this work designs StreamNet that employs the stream buffer to eliminate the redundant computation of patch-based inference. StreamNet uses 1D and 2D streaming processing and an auto-tuning framework to significantly improve the performance of patch-based inference with minimal requirements on the MCU’s SRAM memory space. ", "questions": "1) According to the experiments, we notice that the performance speedup is sensitive to the different neural network architectures say kernel sizes and feature map sizes. While the benchmarks are mostly obtained from NAS and it is a bit difficult to evaluate the representation of these models. Could you provide more details about the models such as the number of layers, model sizes, and accuracy?\n\n2) Transformer models are increasingly utilized, will the proposed framework be applicable to transformer models?", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1) The proposed streamnet removes the computing redundancy in prior patch-based DNN processing framework and improves DNN inference performance significantly. \n\n2) The paper is well organized and easy to follow. The experiments are sufficient and to the point.", "weaknesses": "StreamNet essentially introduces additional buffers to explore the redundant computing results induced by patched DNN processing without compromising the memory requirements. The major contribution will be the DNN computing system or implementation optimization and the novelty is relatively limited. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you all for the valuable comments!  In our revised version, we  addressed all of the reviewer comments. The details of our reply and the changes are presented in the following.\n\nA.1 Could you provide more details about the models such as the number of layers, model sizes, and accuracy?\n\nOur benchmark models were obtained from the MCUNet model zoo and were used to perform the real-world applications such as visual wake word (VWW) and image classification on MCUs. The MCUNet uses the Neural Architecture Search (NAS) to fine tune the accuracy, memory usage, and the amount of MACs of TinyML models to meet the requirements of MCUs. In the following table, each TinyML model comprises multiple layers. The model size and the top-1 accuracy of the MCUNet models are presented in Table 1. These MCUNet models can serve as representatives to real-world TinyML applications on MCUs, which faithfully reflect the performance improvements of our StreamNet.  \n\n| Benchmark            | number of layers  | model sizes (MB)  | Top-1 Accuracy  |\n|----------------------|-------------------|-------------------|-----------------|\n| mcunet-vww0 (MV0)    | 55                | 0.37              | 87.3%           |\n| mcunet-vww1 (MV1)    | 51                | 0.43              | 88.9%           |\n| mcunet-vww2 (MV2)    | 83                | 0.64              | 91.8%           |\n| mcunet-in0 (MI0)     | 51                | 0.75              | 40.4%           |\n| mcunet-in1 (MI1)     | 55                | 0.64              | 49.9%           |\n| mcunet-in2 (MI2)     | 67                | 0.73              | 60.3%           |\n| mcunet-in3 (MI3)     | 67                | 0.74              | 61.8%           |\n| mcunet-in4 (MI4)     | 63                | 1.73              | 68.0%           |\n| mbv2-w0.35 (MB2)     | 64                | 0.75              | 49.0%           |\n| proxyless-w0.3 (PL)  | 76                | 0.75              | 56.2%           |\n\nA.2 Will the proposed framework be applicable to transformer models?\n\nSince the receptive field in the convolution operation works in the sliding window manner, the patch-based inference can divide the entire input into multiple small tiles and reduces the peak SRAM memory usage by only storing one of the small patches. Unlike the convolution operation, in the transformer model, the receptive field covers the entire input. Thus, the tiling method used by the patch-based inference does not work in transformer models. StreamNet aims to decrease the amount of the recomputation caused by the overlapping patches shown on the patch-based inference, which is not applicable to transformer models. Hence, we need to redesign our StreamNet to decrease the SRAM memory usage on transformer models. \n"}, {"review_id": "aYWaLLA5Xv", "reviewer": "Reviewer_XmQf", "summary": "The patch-based inference is widely employed for TinyML models on resource-constrained microcontroller units (MCUs), which significantly reduces memory requirements compared to layer-based inference. However, path-based inference can lead to a substantial increase in Multiply-Accumulates (MACs), as it introduces a great deal of redundant computation among adjacent patches. This paper introduces StreamNet, a solution that curtails repeated computation by memorization. Furthermore, StreamNet can auto-tune the area to be skipped during path-based inference, enhancing overall efficiency.", "questions": "I enjoyed reading the paper, and the idea is well present, and the gains are intuitively making sense. My only question is that what is the reuse distance of patches? Is there any case that the stream buffer is not sufficient to capture all the reuses? ", "limitations": "not applicable", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper provides a comprehensive explanation of the benefits and drawbacks of patch-based inference.\n2. The proposed method is clearly articulated and easy to understand.\n3. StreamNet not only reduces MACs by leveraging the memory of the overlapped computation area, but it also introduces an auto-tuning framework to streamline these areas.\n4. StreamNet outperforms the state-of-the-art methods at a relatively minor expense of additional memory space.\n5. The auto-tuning framework possesses the ability to further accelerate inference automatically if more memory space is made available.", "weaknesses": "1. It would be better to list the potential overlapped area percentage for several commonly used tinyML models.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you all for the valuable comments!  In our revised version, we  addressed all of the reviewer comments. The details of our reply and the changes are presented in the following.\n\nB.1 What is the reuse distance of patches?\n\nThe reuse distance of the StreamNet means the distance of the data in a patch reused by another patch. For example, in Figure 3 of the StreamNet paper, the data of the patch 0 will be reused by the patch 1 in the StreamNet-1D. Thus, the reuse distance of the StreamNet-1D is 1. In addition, as illustrated in Figure 4, the data of patch 1 will be reused by patch 3 in the horizontal direction of the StreamNet-2D. Therefore, the reuse distance of StreamNet-2D in the horizontal direction is 2 because the StreamNet-2D will reuse the  data after passing two patches. That means the reuse distance in the horizontal direction of the StreamNet-2D is the value of the n_patch. \n\nB.2 Is there any case that the stream buffer is not sufficient to capture all the reuses?\n\nThe StreamNet-2D typically captures all the reuses of the patch-based inference in the 2D convolution and the depthwise convolution. However, the StreamNet-2D needs more SRAM memory space to store its 2D stream buffer than the StreamNet-1D. Therefore, the StreamNet-2D will increase the peak usage of the SRAM memory and may run out of the SRAM memory space on MCUs. To address this challenge, our StreamNet mixes the StreamNet-1D and the StreamNet-2D to reduce the peak usage of the SRAM memory on MCUs. Hence, the StreamNet does not capture all the reuses when using both the StreamNet-2D and the StreamNet-1D in a Convolutional Neural Network (CNN) model. \n"}, {"review_id": "YOyOeWxxeh", "reviewer": "Reviewer_xxe7", "summary": "This paper introduces StreamNet, a novel approach designed to eliminate the performance bottleneck associated with patch-based inference, which incurs additional computational overheads due to overlapping patches. StreamNet comprises two techniques, StreamNet-1D and StreamNet-2D, each offering a different trade-off between memory overhead for buffering intermediate results and computational overhead reduction. The authors also propose an auto-tuning framework to derive an inference schedule, which is a composition of the two techniques, given the memory constraints of a Microcontroller Unit (MCU). Evaluation results indicate that the proposed approach significantly improves upon prior work in terms of memory usage and latency.", "questions": "1. Could you provide more insight into how the current experimental setup mirrors different user scenarios, particularly those that commonly employ patch-based inference?\n2. Could you provide the original latency and memory usage of the models without patch-based inference as reference data points?\n3. How would varying the input resolution and the number of patches impacts the performance of StreamNet?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear and comprehensive illustration of the method: The paper provides good illustrations of the proposed methods, making them easy to understand. The examples used to explain the methods are also helpful in allowing readers to understand the advantages of the proposed methods and the trade-off between design elements.\n- Intuitive and straightforward methods: The proposed StreamNet-1D and StreamNet-2D techniques are intuitive and straightforward, offering a clear path to reducing computational overheads by buffering tensors shared between patches.\n- Practicality and on-device evaluation: The paper demonstrates the practicality of the proposed methods through on-device evaluation, which is a strength. However, more comprehensive testing is needed.", "weaknesses": "- Limited experimental settings: The experimental setting does not seem to cover the common user scenario of employing patch-based inference. This limits the generalizability of the results and makes it difficult to assess the full potential of the proposed methods. Future work should include more diverse testing scenarios to fully evaluate the effectiveness and applicability of StreamNet. In my opinion, two control variables, input resolution and number of patches, are missing in the experiments. \n    * Patch-based inference is favorable in settings with large input resolution where the overhead of re-computation can be amortized by the large patch size. For instance, the re-computation overhead reported in MCUNetV2 is only 10% for MobileNetV2, but such overhead appears significantly higher in this paper. What causes the discrepancy requires further discussion.\n    * The number of patches used also significantly influences the re-computation overheads and memory usage. It would be intriguing to observe the trade-off between latency and memory usage achieved by the proposed method in comparison to the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you all for the valuable comments!  In our revised version, we  addressed all of the reviewer comments. The details of our reply and the changes are presented in the following.\n\nC.1 The re-computation overhead reported in MCUNetV2 is only 10% for MobileNetV2, but such overhead appears significantly higher in this paper. What causes the discrepancy requires further discussion.\n\nThe MCUNet-v2 model zoo does not include the MobileNet-v2 model used in the MCUNet-v2 paper. Thus, we cannot directly compare the result reported in the MCUNet-v2 paper with that in the StreamNet paper. In addition, the re-computation overhead of the patch-based inference (MCUNet-v2) becomes worse when decreasing the peak usage of SRAM memory. The MCUNet-v2 uses the patch-based inference to increase 10% of the MACs while decreasing 87.5% of the peak SRAM memory usage compared to the layer-wise tensor memory allocation. Although the MCUNet-v2 can further decrease the SRAM memory usage of TinyML models on MCUs.  MCUNet-v2 only presents the SRAM memory usage of TinyML models when increasing 10% of the MACs.  This is because the amount of recomputation in the MCUNet-v2 will significantly increase when the SRAM memory usage of TinyML models is minimized. Unlike MCUNet-v2, the StreamNet minimizes the usage of the SRAM memory without significantly increasing the amount of MACs on TinyML models.  \n\nC.2 Could you provide more insight into how the current experimental setup mirrors different user scenarios, particularly those that commonly employ patch-based inference?\n\nThe usage of the SRAM memory can drop significantly in the patch-based inference by only increasing a few MACs. For instance, the original PL model contains 38.3 M of the MACs and uses 259 KB SRAM memory space. Then, the patch-based inference increases 8.8% of the MACs (41 M) and decreases the usage of the SRAM memory on the PL model to 158 KB. Unlike MCUNet-v2, the amount of MACs in the StreamNet is 39.1 M when using 92 KB SRAM memory space. As a result, the StreamNet needs fewer MACs and SRAM memory space than the patch-based inference, since the StreamNet removes the recomputation overhead of the patch-based inference while using small SRAM memory space.\n\nC.3 Could you provide the original latency and memory usage of the models without patch-based inference as reference data points?\n\nThe original latency was obtained from the MCUNet-v1 that does not use the patch-based inference. To achieve a fair comparison of the latency on TinyML models, the StreamNet and the MCUNet-v1 use the same back-end library. The MCUNet-v1 reduces the peak SRAM memory usage on TinyML models by performing the in-place depthwise operation instead of using the patch-based method. Table 2 presents the StreamNet-2D gains about 25% runtime overhead when processing streaming buffers over the MCUNet-v1. Moreover, the StreamNet-2D saves about 42.3% SRAM memory usage over the MCUNet-v1 by using the patch-based inference with 2D streaming buffers. \n\n|        | Original Latency        |               | StreamNet-2D            |               |\n|--------|-------------------------|---------------|-------------------------|---------------|\n| Model  | SRAM Memory Usage (KB)  | latency (ms)  | SRAM Memory Usage (KB)  | latency (ms)  |\n| MB2    | 295                     | 367           | 66                      | 417           |\n| PL     | 259                     | 542           | 95                      | 676           |\n| MI0    | 49                      | 82            | 30                      | 99            |\n| MI1    | 96                      | 169           | 47                      | 188           |\n| MI2    | 215                     | 869           | 169                     | 1,168         |\n| MI3    | 260                     | 1,048         | 208                     | 1,444         |\n| MI4    | 416                     | 1,371         | 236                     | 1762          |\n| MV0    | 59                      | 86            | 29                      | 101           |\n| MV1    | 92                      | 159           | 44                      | 225           |\n| MV2    | 174                     | 718           | 143                     | 961           |\n\nC.4 How would varying the input resolution and the number of patches impact the performance of StreamNet?\n\nThe following two figures present the latency and memory variations of the StreamNet-2D when changing the number of patches in the PL model. As illustrated in Figure I, the execution time of the PL model does not significantly fluctuate when varying the number of patches on the STM32F767 MCU. In Figure II, the SRAM memory usage is sensitive to the value of the split_idx. For instance, in Figure I, the SRAM memory usage on the setting of the (2, 12) is 50.5% smaller than the (2, 2). In addition, the SRAM memory usage also drops when increasing the number of patches. For example, in Figure II, the SRAM memory usage of the (44, 4) is 14.4% smaller than the (2, 4). Furthermore, the size of the patches will increase with the growth of the input resolution without changing the value of the n_patch and the split_idx. Hence, the StreamNet-2D requires more stream buffer space to store tensor data and the recomputation overhead of the StreamNet-2D does not increase with the growth of the input resolution. \n"}, {"review_id": "vwMvXS9bNf", "reviewer": "Reviewer_TjRD", "summary": "This paper presents methods for speeding up patch-based inference on microcontrollers. The method creates a buffer to selectively save intermediate values that are traditionally discarded in patch-based computation. This allows StreamNet to balance latency and memory consumption. There are 1d and 2d variants of this optimization.\n\nThe paper additionally proposes a method for skipping the computation of padding data and a framework for auto-tuning the hyperparameters of StreamNet. The framework searches for patch hyperparameters that meet certain memory constraints while minimizing latency.", "questions": "- Is this method purely a runtime/compiler optimization, or are there any model architecture implications (e.g. on accuracy)?\n\n- How might one redesign MCU class models to maximize the benefits of StreamNet? If StreamNet more easily optimizes certain layer configurations, how does that impact exiting Pareto curves?\n\n- Will StreamNet be open-sourced?", "limitations": "How long does it take to run the auto-tuning framework? How does it compare to black-box search methods?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Pure runtime optimization targeted at the most important MCU metrics (latency and memory)\n- State-of-the-art performance vs. a strong baseline (MCUNetv2) \n- Enables a new dimension for latency-memory tradeoffs\n- Includes an algorithm for automatically searching the newly created search space.\n", "weaknesses": "- Given the tradeoff between latency and memory consumption that StreamNet unlocks, it would be easier to interpret results as points on a latency-memory Pareto curve rather than cross-referencing tables and charts. The table in the appendix that gave the speedup at nearly equal memory was the most informative in understanding the benefits, but it was buried. \n\n- Only compares against MCUNetv2 and not the other patch-based optimization mentioned in the related work.\n\n- Relies heavily on existing work (MCUNetv2 and TinyEngine) \n\n- The contributions in the intro should be reworded to be consistent", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you all for the valuable comments!  In our revised version, we  addressed all of the reviewer comments. The details of our reply and the changes are presented in the following.\n\nD.1 Is this method purely a runtime/compiler optimization, or are there any model architecture implications (e.g. on accuracy)?\n\nYes, StreamNet is purely a runtime/compiler solution. The codegen of StreamNet yields the streaming buffer used when transferring data at the runtime without changing the original model structure. As a result, StreamNet has the same output and accuracy compared to the original model. \n\nD.2 How might one redesign MCU class models to maximize the benefits of StreamNet?\n\nIt is possible to use the Neural Architecture Search (NAS) method to discover new parameters of DNN models to meet the given memory budget on MCUs. Unlike the NAS method, the StreamNet can be used for existing DNN models without having to change the structure of the original model. The StreamNet also eliminates the model training processing used by the NAS. Furthermore, the StreamNet compiler also automatically analyzes the structure of the DNN model to create streaming buffers and reduces the amount of unnecessary recomputations shown on the patch-based inference. As a result, the StreamNet decreases the additional burden to manually craft a new DNN model to meet the requirement of the MCUs.   \n\nD.3 Will StreamNet be open-sourced?\n\nStreamNet will be open-sourced after the paper is published. \n\nD.4 How long does it take to run the auto-tuning framework? How does it compare to black-box search methods?\n\nThe StreamNet auto-tuning method takes about one minute to discover the best parameters used to reorganize the structure of the streaming buffer and fit a DNN model in the MCU. The black-box (brute-force) method firstly enumerates all combinations of the n_patch and the split_idx parameters. Second, the black-box method compiles combinations of the n_patch and the split_idx parameters with all feasible configurations of streaming levels (x or y streaming). Finally, the black-box method works out parameters that will consume the minimum amount of MACs while meeting the SRAM memory budget of the MCU. Unlike the black-box method, StreamNet leverages the Pareto Front optimization method to reduce the size of the search space. As a result, the StreamNet auto-tuning achieves a geometric mean of 1.29X speedup over the black-box method on the Apple Silicon M1 CPU by using single thread. \n\n| Benchmark            | StreamNet Auto tuning  | Brute Force  | Speedup  |\n|----------------------|------------------------|--------------|----------|\n| mcunet-vww0 (MV0)    | 51.17s                 | 83.00s       | 1.62X    |\n| mcunet-vww1 (MV1)    | 46.47s                 | 77.86s       | 1.68X    |\n| mcunet-vww2 (MV2)    | 95.42s                 | 107.44s      | 1.13X    |\n| mcunet-in0 (MI0)     | 91.68s                 | 134.30s      | 1.46X    |\n| mcunet-in1 (MI1)     | 59.17s                 | 93.85s       | 1.59X    |\n| mcunet-in2 (MI2)     | 90.87s                 | 92.39s       | 1.02X    |\n| mcunet-in3 (MI3)     | 61.11s                 | 64.95s       | 1.06X    |\n| mcunet-in4 (MI4)     | 124.40s                | 126.61s      | 1.02X    |\n| mbv2-w0.35 (MB2)     | 73.28s                 | 101.98s      | 1.39X    |\n| proxyless-w0.3 (PL)  | 65.90s                 | 79.89s       | 1.21X    |\n"}]

Author rebuttal:
Thank you all for the valuable comments!  In our revised version, we  addressed all of the reviewer comments. The details of our reply and the changes are presented in the following.

C.1 The re-computation overhead reported in MCUNetV2 is only 10% for MobileNetV2, but such overhead appears significantly higher in this paper. What causes the discrepancy requires further discussion.

The MCUNet-v2 model zoo does not include the MobileNet-v2 model used in the MCUNet-v2 paper. Thus, we cannot directly compare the result reported in the MCUNet-v2 paper with that in the StreamNet paper. In addition, the re-computation overhead of the patch-based inference (MCUNet-v2) becomes worse when decreasing the peak usage of SRAM memory. The MCUNet-v2 uses the patch-based inference to increase 10% of the MACs while decreasing 87.5% of the peak SRAM memory usage compared to the layer-wise tensor memory allocation. Although the MCUNet-v2 can further decrease the SRAM memory usage of TinyML models on MCUs.  MCUNet-v2 only presents the SRAM memory usage of TinyML models when increasing 10% of the MACs.  This is because the amount of recomputation in the MCUNet-v2 will significantly increase when the SRAM memory usage of TinyML models is minimized. Unlike MCUNet-v2, the StreamNet minimizes the usage of the SRAM memory without significantly increasing the amount of MACs on TinyML models.  

C.2 Could you provide more insight into how the current experimental setup mirrors different user scenarios, particularly those that commonly employ patch-based inference?

The usage of the SRAM memory can drop significantly in the patch-based inference by only increasing a few MACs. For instance, the original PL model contains 38.3 M of the MACs and uses 259 KB SRAM memory space. Then, the patch-based inference increases 8.8% of the MACs (41 M) and decreases the usage of the SRAM memory on the PL model to 158 KB. Unlike MCUNet-v2, the amount of MACs in the StreamNet is 39.1 M when using 92 KB SRAM memory space. As a result, the StreamNet needs fewer MACs and SRAM memory space than the patch-based inference, since the StreamNet removes the recomputation overhead of the patch-based inference while using small SRAM memory space.

Unlike the MCU used in the StreamNet paper, there is another user scenario when using the STM32L412 MCU that contains 256 KB SRAM memory and 1MB flash memory. The peak SRAM memory usage of the Ml4 model is 426 KB, and the amount of MACs is 126 M when using the Google TensorFlow Lite for Microcontroller (TFLM) with the layer-wise tensor memory allocation. The TFLM will cause the out-of-memory exception on the STM32L412 MCU, since the SRAM memory usage of the Ml4 model is larger than the SRAM memory space on the STM32L412 MCU. The MCUNet-v2 performs the patch-based inference to reduce the use of the SRAM memory on the MCU. Thus, in the MCUNet-v2, the Ml4 model only uses 129 KB SRAM memory, but the amount of MACs becomes 1942 M when using the patch-based inference. To fit the SRAM memory constraint, in the StreamNet-2D, the Ml4 model uses 241 KB SRAM memory space with 126M MACs. The StreamNet-2D reduces 15.41X MACs by capturing all reuses in the Ml4 model. In addition, in the StreamNet-1D, the amount of MACs is 354 M, and the peak SRAM memory usage is 150 KB. The outcome of the StreamNet-1D is similar to the result of the MCUNet-v2 in the Ml4 model. Typically, we can adjust parameters of the MCUnet-v2 to increase the usage of the SRAM memory and decrease the amount of the recomputation on overlapping patches. For instance, to meet the size of the SRAM memory on the STM32L412 MCU, we choose that value of the (5, 19) in the (n_patch, split_idx) parameter that yields the usage of the SRAM memory closest to the memory constraint on the STM32L412 MCU. Thus, in such MCUNet-v2 parameters, the amount of MACs in the Ml4 model becomes 458 M, and the peak SRAM memory usage is 239 KB. Therefore, it is possible to adjust values of the patch-based inference parameters to meet the memory constraint of the MCUs. However, the patch-based inference exposes the extremely high recomputation overhead when lowering the peak SRAM memory usage or using the MCU with a small SRAM memory. Unlike the patch-based inference, the StreamNet eliminates the large portion of the recomputation overhead shown in the patch-based inference without significantly increasing the usage of the SRAM memory.  

C.4 How would varying the input resolution and the number of patches impact the performance of StreamNet?

The following two figures present the latency and memory variations of the StreamNet-2D when changing the number of patches in the PL model. As illustrated in Figure I, the execution time of the PL model does not significantly fluctuate when varying the number of patches on the STM32F767 MCU. In Figure II, the SRAM memory usage is sensitive to the value of the split_idx. For instance, in Figure I, the SRAM memory usage on the setting of the (2, 12) is 50.5% smaller than the (2, 2). In addition, the SRAM memory usage also drops when increasing the number of patches. For example, in Figure II, the SRAM memory usage of the (44, 4) is 14.4% smaller than the (2, 4). Furthermore, the size of the patches will increase with the growth of the input resolution without changing the value of the n_patch and the split_idx. Hence, the StreamNet-2D requires more stream buffer space to store tensor data and the recomputation overhead of the StreamNet-2D does not increase with the growth of the input resolution.







Meta-review: The submitted paper presents StreamNet, a framework designed to optimize patch-based DNN inference on microcontroller units (MCUs) by reducing computational redundancy. StreamNet achieves this using stream buffers and offers an auto-tuning framework for further optimization. The paper aims to balance memory and latency, providing a significant improvement over previous methods. The paper has garnered four "Weak Accept" ratings from the reviewers, with scores generally indicating good presentation and contribution.

Given the above considerations and the "Weak Accept" consensus among the reviewers, the paper is a candidate for acceptance. The reviewers agree on the paper's strengths in presentation and practical contributions. It is advised that the authors take into consideration the feedback on the experimental setup and general applicability for future revisions or presentations. All reviewers have moderate to high confidence in their assessments, adding weight to the decision.
Decision: Accept (poster)