Title: StreamNet: Memory-Efficient Streaming Tiny Deep Learning Inference on the Microcontroller
Venue: NeurIPS 2023
Accepted: yes
Keywords: TinyML models

edge AIs

Microcontroller
Abstract: With the emerging Tiny Machine Learning (TinyML) inference applications, there is a growing interest when deploying TinyML models on the low-power Microcontroller Unit (MCU). However, deploying TinyML models on MCUs reveals several challenges due to the MCU’s resource constraints, such as small flash memory, tight SRAM memory budget, and slow CPU performance. Unlike typical layer-wise inference, patch-based inference reduces the peak usage of SRAM memory on MCUs by saving small patches rather than the entire tensor in the SRAM memory. However, the processing of patch-based inference tremendously increases the amount of MACs against the layer-wise method. Thus, this notoriously computational overhead makes patch-based inference undesirable on MCUs. This work designs StreamNet that employs the stream buffer to eliminate the redundant computation of patch-based inference. StreamNet uses 1D and 2D streaming processing and provides an parameter selection algorithm that automatically improve the performance of patch-based inference with minimal requirements on the MCU’s SRAM memory space. In 10 TinyML models, StreamNet-2D achieves a geometric mean of 7.3X speedup and saves 81\% of MACs over the state-of-the-art patch-based inference.

StreamNet: Memory-Efficient Streaming Tiny Deep Learning Inference on the Microcontroller

Abstract

With the emerging Tiny Machine Learning (TinyML) inference applications, there is a growing interest when deploying TinyML models on the low-power Microcontroller Unit (MCU). However, deploying TinyML models on MCUs reveals several challenges due to the MCU's resource constraints, such as small flash memory, tight SRAM memory budget, and slow CPU performance. Unlike typical layer-wise inference, patch-based inference reduces the peak usage of SRAM memory on MCUs by saving small patches rather than the entire tensor in the SRAM memory. However, the processing of patch-based inference tremendously increases the amount of MACs against the layer-wise method. Thus, this notoriously computational overhead makes patch-based inference undesirable on MCUs. This work designs StreamNet that employs the stream buffer to eliminate the redundant computation of patch-based inference. StreamNet uses 1D and 2D streaming processing and provides an parameter selection algorithm that automatically improve the performance of patch-based inference with minimal requirements on the MCU's SRAM memory space. In 10 TinyML models, StreamNet-2D achieves a geometric mean of 7.3X speedup and saves 81% of MACs over the state-of-the-art patch-based inference.

1 Introduction

With the emerging Tiny Machine Learning (TinyML) applications, challenges to successfully deploy TinyML models on the resource-constrained microcontroller (MCU) are becoming critical. Unlike the high-performance CPU, the MCU is basically composed of a small flash memory (several MBs), an SRAM memory (hundreds of KBs), and a slow CPU (hundreds of MHz). Since the SRAM memory has better performance on reading and writing data than the flash memory, most TinyML system frameworks often place the input/output tensors on the SRAM and store weights and filters on the flash memory. In addition, the size of memory on MCUs is always small because of their constrained budgets. However, recent TinyML models tend to use large input images and intermediate tensors to raise their training accuracy (1; 2). Such TinyML models expose grand challenges when deploying these models on MCUs with the stringent memory constraint.

Tensor memory planner of TinyML system frameworks determines the usage of the SRAM memory. An MCU is often responsible for Machine Learning (ML) inference and often uses a single batch to facilitate the response time of ML inference tasks. Thus, the static tensor memory planner schedules intermediate tensors to meet the SRAM memory constraint. Conventionally, the compiler/interpreter employs layer-wise scheduling, pushes data of a layer in the SRAM memory, and replaces them after finishing its operator (3; 4). However, such layer-wise inference will overuse the SRAM memory space when TinyML models have large intermediate tensors. To overcome this shortcoming, patch-

based inference (2) divides the entire tensors into multiple overlapping patches. Then, patch-based inference places a patch in the SRAM memory space rather than the entire tensor to repair the out-of-memory problem. However, the redundant computation of patch-based inference occurs when proceeding with a large number of overlapping patches and tremendously increases the execution time of TinyML models. Furthermore, this situation becomes worse in particular when significantly shrinking the usage of SRAM memory space. Hence, patch-based inference is becoming undesirable with such notoriously high performance overhead on MCUs.

This work designs StreamNet that eliminates the significant amount of redundant computations while reaching minimal usage of the SRAM memory space for TinyML models on MCUs. StreamNet creates the stream buffer to reuse tensor data in the patch-based inference. In addition, StreamNet designs 1D and 2D stream processing to improve the performance of patch-based inference with different SRAM memory requirements. To facilitate the performance of StreamNet further, StreamNet also removes the unnecessary computation on the padding data. At last, StreamNet's parameter selection framework automatically constructs a new patch-based inference that mixes 1D and 2D stream processing to meet the memory constraint while resulting in the best performance for TinyML models.

The main contributions of this paper are shown below:

StreamNet achieves a geometric mean of 3.4X (1D) to 7.3X (2D) speedup over the existing patch-based inference without increasing memory usage significantly.

Developing output padding bypass kernels to enhance the performance of patch-based inference.

The StreamNet parameter selection framework automatically selects an optimal configuration for StreamNet-2D operations under the SRAM memory constraint of an MCU.

2 Understanding Performance Bottleneck of Memory Allocation on TinyML

Memory-efficient patch-based inference : Patch-based inference aims to reduce the peak SRAM memory usage of TinyML models on MCUs. This patch-based approach divides a large input feature map into multiple small patches. Then, the MCU's SRAM buffer reduces its space usage by only storing a small patch each time instead of the entire input feature map. For instance, Figure 1(a) demonstrates the peak SRAM memory usage of the proxyless-w0.3 TinyML model (5) and other TinyML models have the similar trend. Traditionally, ML system frameworks such as TFLM (3) place input and output tensors of a layer or an operator in the SRAM buffer. After finishing the work of a layer, TFLM (3) removes the obsolete input data of the completed layer and allocates another space to put the output data of the next layer. However, given an MCU and its size of SRAM memory is 128 KB, Figure 1 demonstrates that the memory usage of multiple layers in the proxyless-w0.3 TinyML model (5) is larger than the size of the SRAM memory on an MCU. As a result, the layer-wise memory allocation raises significant challenges when deploying TinyML models on MCUs. Unlike the layer-wise inference, Figure 1 presents patch-based inference that reduces the total SRAM memory usage of TinyML models and enables models to be run on MCUs with small memory size.

PeakMemoryUsage

B

250

200

242kB

-Per-layer

PeakMemory

Per-layerMemory

Per-patchMemory

amountofMACs

125

100

33x

←Per-patch

PeakMAC

Per-layerMAC

Per-patchMAC

150

128KB

75

100

50

50

25

0

0

1

2

3

4

5

6

7

8

9

10

11

1

2

3

4

5

6

7

8

9

10

11

LayerIndex

LayerIndex

(a)Thepeakmemoryusageofcascadepolicyoverthebaseline

(b)TheamountofMACincascadepolicyoverthebaseline

Challenges and opportunities : Patch-based inference creates a large amount of redundant computation that occurs among its adjacent patches. To reduce the peak SRAM memory usage on MCUs,

patch-based inference creates many small patches that overlap receptive fields. These small patches tremendously raise the amount of computation on a TinyML model. The inefficient patch-based inference is becoming undesirable on MCUs even if it saves the peak SRAM memory usage of a TinyML model significantly. Figure 1(b) demonstrates that patch-based inference takes 33X more MACs than the layer-wise one at the first layer. Furthermore, the amount of MACs in patch-based inference also rapidly grows with the reduction of peak SRAM memory usage. Hence, our work stores repeated results of patch-based inference in the stream buffer and skips these computations to eliminate the performance bottleneck of patch-based inference.

3 StreamNet: Efficient Patch-based Inference for TinyML Models

This section details the design and implementation of StreamNet that removes the redundant computation of patch-based inference with the minimal usage of the SRAM memory space on MCUs.

(a) Step 1 & 2

଴

ଵ

1

2

1

1

2

2

2

1 1

2

2

1

(b) Step 3

଴

ଵ

1

2

1

3

2

1

3

2

1

3

1 1

3

2

1

(c) Step 4

଴

ଵ

1

2

2

4

2

1

4

2

2

4

1 2

4

2

2 1

Operator

Overlapping area

Computed

Patch 0

Patch 1

Patch 2

Patch 3

3.1 Inefficient Patch-based Inference

To accelerate patch-based inference, this work proposes StreamNet that removes most redundant computations of patch-based inference by putting its repetitive results in the stream buffer. Figure 2 demonstrates operations of patch-based inference. In Figure 2, there are two operators where each of them has its own input and output tensor and the computed block means the completed output element. Conventionally, in the convolution processing, the 6 × 6 input with the 3 × 3 filter will yield the 4 × 4 output. This calculation takes 144 ( 16 × 9 ) MACs where the number 16 is the size of the output and the number 9 comes from the 3 × 3 filter. Unlike conventional convolution, in Figure 2, patch-based inference divides a 6 × 6 input into four 5 × 5 patches. Each 5 × 5 patch computes the convolution with the 3 × 3 filter to finally yield a 4 × 4 output. As illustrated in Figure 2(a), Patch 0 and Patch 1 generate a 3 × 3 output patch, respectively. Each of them takes 9 × 9 MACs where the first number 9 is the size of output and the second number 9 is associated with the size of the filter. Thus, patch-based inference uses 4 patches to create a 4 × 4 output and takes 324 MACs ( 9 × 9 × 4 ). The amount of MACs of patch-based inference is 2.25X (324/144) more than the layer-wise inference because patch-based inference contains an enormous amount of repetitive computations. For instance, in Figure 2, each value in the overlapping area represents the repetition count of the output tensor element. This redundant calculation increases the execution time of a TinyML model significantly. In addition, patch-based inference tends to use the fine-grained patch to lower the usage of the SRAM memory. However, the amount of redundant computation increases as the patch size becomes small.

3.2 StreamNet 1D Streaming Processing

Figure 3(a) illustrates the steps of 1D streaming processing on StreamNet inference. In Figure 3(a), two patches proceed one after the other as the data stream during the computation of the convolution. In Figure 3, the computed block represents the completed element of the tensor, and the computing block indicates the elements of the tensor in the execution. At first, in Figure 3(a), StreamNet-1D performs the CONV0 operator on the 5 × 5 patch and yields a 3 × 3 output tile ( 1 ). Second, StreamNet-1D stores the overlapping area of this 3 × 3 output tile in the stream buffer ( 2 ). Then, StreamNet-1D uses this 3 × 3 tile to generate the 1 × 1 output data after finishing the CONV1 operator ( 3 ). Unlike patch-based inference, StreamNet uses the results preserved in the stream buffer during the computation of Patch 1. Thus, StreamNet-1D only fetches a 3 × 5 tile from Patch 1 to yield a

1

2

1

2

2

1

2

1

1

2

1 2

2

1

2 1

1

2

2

4

2

1

4

2

2

4

1 2

4

2

2 1

StreamNet-1D

(a) StreamNet-1D streaming processing

Stream

Buffer

Inplace

Concat

଴

ଵ

5

3

4

2

1

6

(c) StreamNet-1D result

(b) Patch-based inference

Patch 1

Patch 0

Operator

Overlapping area

Computing

Computed

8

3 × 1 output data ( 4 ). Then, StreamNet-1D combines this 3 × 1 output with the data of the stream buffer to work out a 3 × 3 output tile as illustrated in Figure 3(a) ( 5 ). At last, the 3 × 3 output of Patch 1 will yield a 1 × 1 output after finishing the CONV1 operator ( 6 ). In the computation of Patch 0 and Patch 1, StreamNet-1D takes 108 MACs ( 12 × 9 ) where the number 12 is the sum of the calculated output size in ( 1 ) ( 3 × 3 ) and ( 4 ) ( 3 × 1 ) and the number 9 is associated with the size of the 3 × 3 filter. Since StreamNet-1D also needs to use four patches to create a 4 × 4 output in total, the total amount of MACs in 1D streaming processing of StreamNet-1D is 216 ( 12 × 9 × 2 ). Unlike patch-based inference in Figure 3(b), StreamNet-1D in Figure 3(c) decreases the repetition counts in each element of the output tensor. Hence, StreamNet-1D reduces roughly 60% of MAC overhead over patch-based inference in Figure 3, because the overhead of StreamNet-1D and patch-based inference is 8 (24 - 16) and 20 (36 - 16) where 16 ( 4 × 4 ) is the number of output elements of the layer-wise inference.

଴

Patch 1

Patch 0

Operator

Patch 2

Patch 3

ଵ

height

width

Inplace

Concat

Inplace

Concat

X Stream

Buffer

Inplace

Concat

1

2

3

4

stream_x

stream_y

stream_x + stream_y

Y Stream

Buffer

Y Stream

Buffer

XY

Stream

Buffer

AIW

3.3 StreamNet 2D Streaming Processing

StreamNet uses the stream buffer to preserve temporary results that will be used multiple times across patches. After finishing an operator, StreamNet will place a number of output tensors in the stream buffer. Since StreamNet-1D does not completely remove the redundant computation overhead, StreamNet designs StreamNet-2D to eliminate all redundant MACs in patch-based inference. StreamNet-2D is composed of stream_x and stream_y processing. The execution order of

stream_y patches is firstly moving downward. For instance, patch-based inference in Figure 4 moves its execution vertically for Patch 1, then back to Patch 2, and finally moves downward for Patch 3. Unlike stream_y, the execution order of stream_x patch firstly moves toward the horizontal direction. At first, in Figure 4, the 5 × 5 Patch 0 computes CONV0 with the 3 × 3 filter and yields a 3 × 3 output. Then, StreamNet-2D stores a 3 × 2 tile and 2 × 3 tile of the 3 × 3 output in the Y and X stream buffer ( 1 )( 2 ), respectively. Second, StreamNet-2D tackles Patch 1 and fetches a 5 × 3 tile in Patch 1 to yield a 3 × 1 output. After finishing Patch 1, StreamNet stores a 2 × 1 tile of its output tensor to the XY stream buffer ( 4 ) and cleans the Y stream buffer. Third, StreamNet-2D handles Patch 2 and reuses results in the X stream buffer. After completing Patch 2, StreamNet-2D stores a 3 × 2 tile of Patch 2's output in the Y stream buffer ( 3 )). Finally, StreamNet-2D reuses the data within the Y and XYstream buffer to proceed with Patch 3. StreamNet-2D takes 144 MACs ( (9+3+3+1) × 9 ) where each number of the first bracket represents the amount of calculation in each patch and the second number 9 means the size of the 3 × 3 filter. Hence, StreamNet-2D completely eliminates the total amount of the redundant computation shown in patch-based inference. However, StreamNet-2D has a larger stream buffer to store the temporary results of each patch over StreamNet-1D. As illustrated in Figure 4, the stream buffer of StreamNet-2D is 14 where the 2 × 3 tile is from stream_x, the 3 × 2 tile is from stream_y, and the 2 × 1 tile is from the XY stream buffer. A large stream buffer increases the requirement of the SRAM memory space. To balance the consumption of the MACs and the SRAM memory space, StreamNet presents an parameter selection framework that mixes StreamNet-1D and StreamNet-2D. As a result, StreamNet can satisfy the constraint of the SRAM memory size on MCUs while achieving efficient execution on TinyML models.

Algorithm 1: The StreamNet Parameter Selection Algorithm

3.4 StreamNet Parameter Selection Framework

Searching for the best parameter that satisfies memory constraints while minimizing the latency of a TinyML model manually is exceedingly tedious and challenging. To address this problem, StreamNet designs an parameter selection framework that determines the best parameter candidate to meet the constraint of the SRAM memory across different MCUs. Hence, StreamNet parameter selection framework aims to automatically find the parameter composition of StreamNet and mixes StreamNet-1D and StreamNet-2D to fit the limited amount of SRAM memory on the MCU. To minimize the runtime overhead, StreamNet refers to the MAC count to choose the best candidate of StreamNet parameters through its compiler pass.

Algorithm 1 demonstrates StreamNet parameter selection framework that infers parameters of StreamNet to fit an MCU at the compile time. The patch parameter in line 1 of Algorithm 1 includes the number of the patch (n_patch) and the number of layers that proceed with patch-based

computation (split_idx). The OutShapeFactor function returns the valid number of the patch in patch-based inference. Next, StreamNet calculates the amount of MACs with different combinations of these two parameters and picks out candidates that meet the constraint of the SRAM memory space of an MCU from line 2 - 7 of the Algorithm 1. Then, StreamNet examines different values of stream_x_level and stream_y_level to determine the number of stream-wise layers while having a minimum amount of MACs in a model and returns this combination from line 8 - 15 of Algorithm 1. Otherwise, StreamNet tends to choose the one with the smallest patch count when both candidates have the same amount of MACs, since a large output patch has bigger computation granularity, which usually yields better latency.

Tensor D

଴

ଵ

ଶ

…

…

Tensor A

…

…

Tensor C

…

…

…

…

Tensor B

Padding area

Tensor area

Patch area

Operator

StreamNet

skipping area

3.5 Output Padding Bypassing

The padding data of each tensor contributes the performance overhead in the patch-based inference. To further the performance of patch-based inference, StreamNet intelligently skips the computation of the padding data. Figure 5 presents a patch-based inference neural network that consists of 3 operators and 4 tensors. Padding is always added to a tensor to generate an output with the specified size. Parameters of conventional neural networks such as the size of the stride and the filter result in different amounts of padding data in each tensor. Unlike common neural networks, in Figure 5, the patch size of the current tensor is obtained from the configuration of the next tensor. For instance, patch-based inference refers to the patch size of Tensor C to figure out the patch size of Tensor B. The padding data in Tensor B appends the one in Tensor C. Thus, the amount of the padding data in patch-based inference can be more than the one in the layer-wise inference. Since the value of padding data is always assigned to be zero, the padding data won't change the expected results. In Figure 5, patch-based inference proceeds with a 7 × 7 patch in Tensor A including padding data to generate an output 5 × 5 patch. Next, this 5 × 5 patch in Tensor B is used to create a 3 × 3 patch in Tensor C. Hence, each operator computes its padding data that contributes additional MACs during the execution of TinyML models. StreamNet removes the performance overhead shown in the padding data by skipping the computation of each output padding data. In Figure 5, patch-based inference reads the data of a 5 × 5 patch in Tensor B to generate a 3 × 3 patch in Tensor B. Unlike patch-based inference, StreamNet uses the size of the padding data known after finishing the previous layer to distinguish the padding data from the real one. Next, StreamNet only considers the real data in the calculation of an operator. Furthermore, StreamNet only writes the zero value into output padding data directly without consuming any MACs. StreamNet changes the implemented kernel of patch-based inference to skip the computation of the output padding data during the execution of each operator.

4 StreamNet System Architecture

The system architecture of StreamNet contains the frontend and backend processing. In the frontend system of StreamNet, the input of StreamNet is the TensorFlow Lite model (.tflite file). Then, StreamNet uses the parameters of the TinyML model to create layers of patch-based inference (2). Second, the memory scheduler allocates SRAM memory space for tensors on the static arena memory space and then StreamNet performs the 1D and 2D stream processing with its stream buffers. In StreamNet's backend system, StreamNet translates tensors and operators of the patch-based inference

StreamNet

Unchanged

TE Kernels: TinyEngine Kernels

BP Kernels: Bypass padding Kernels

StreamNet

Unchanged

TE Kernels: TinyEngine Kernels

BP Kernels: Bypass padding Kernels

Stream patch Generation

+

Patch generation

.tflite

patch params

(𝑛_𝑝𝑎𝑡𝑐ℎ, 𝑠𝑝𝑙𝑖𝑡_𝑖𝑑𝑥)

stream params

(𝑠𝑡𝑟𝑒𝑎𝑚_𝑥_𝑙𝑒𝑣𝑒𝑙,

𝑠𝑡𝑟𝑒𝑎𝑚_𝑦_𝑙𝑒𝑣𝑒𝑙)

Patched Layers

patched

Stream Patch Code Gen

+

code

*.h

*.c

CMSIS-NN

TE Kernels

BP Kernels

Mbed CLI SDK

Target Platform

ARM Cortex-M7

into C source code and links kernel libraries such as ARM CMSIS-NN (6), TinyEngine (1; 2), and our bypass padding kernels. Finally, StreamNet deploys TinyML models on the MCU by using Mbed CLI SDK (7).

5 Experiments

5.1 Experiment Environment Setup

System Implementation : We compare the performance and memory consumption of StreamNet to MCUNetV2 (2), which performs patch-based inference on TinyML models. Since existing TinyEngine kernels (1) used by MCUNetV2 (2) do not support 5 × 5 and 7 × 7 filters, we fix this issue to enable our evaluation to work on the depthwise convolution function with more filter combinations. In addition, we also insert our output padding bypassing implementation within MCUNetV2 (2) as MCUNetV2+. StreamNet modifies MCUNetV2's memory scheduler and code generator to identify streamable area, allocate stream buffer, generate codes with different streaming processing, and insert the stream buffer operation between operator kernel invocation in StreamNet.

Benchmark : We evaluate 10 TinyML models from MCUNetV2 model zoo (5)such as mbv2w0.35(MB2), proxyless-w0.3(PL), mcunet-vww0(MV0), mcunet-vww1(MV1), mcunet-vww2(MV2), mcunet-in0(MI0), mcunet-in1(MI1), mcunet-in2(MI2), mcunet-in3(MI3), mcunet-in4(MI4). All models use int8 quantized mode. These TinyML models have different requirements on the SRAM memory space and their execution time also differs.

On-device Experiment : We deploy TinyML models through ARM Mbed CLI (7) on the ARM Cortex-M7 CPU. The MCU used in our evaluation is STM32F767ZI (8) that includes an ARM Cortex-M7 CPU at 216 MHz, a 512KB SRAM, and a 2MB Flash

5.2 Performance Analysis when Consuming Nearly Equal Memory

Table 1 compares the performance of StreamNet and MCUNetv2 when both of them use nearly the same SRAM memory space. Table 1 presents the data of StreamNet-2D. Then, we scan through the value of n_patch and split_idx parameters to search for the one where their SRAM memory usage is closest to StreamNet-2D. In Table 1, StreamNet achieves a geometric mean of 5.11X speedup over MCUNetv2. The SRAM memory consumption varies with the change of n_patch and split_idx parameters on MCUNetv2. However, the amount of MACs increases rapidly with the reduction of SRAM memory usage on MCUNetv2. As a result, StreamNet has better performance than MCUNetv2 when consuming nearly equal SRAM memory.

5.3 The Variation of MAC Counts

Table 2 compares the amount of MACs used in each TinyML model. In Table 2, the value of the n_patch and split_idx determines the number of patches and the number of patch-based layers used in a model, respectively. StreamNet scans through all combinations of the n_patch and split_idx and

finds the one that can result in the minimal usage of SRAM memory space in patch-based inference. Thus, results in Table 2 show the amount of MACs used in the TinyML model with the lowest peak SRAM memory usage through all possible combinations of n_patch and split_idx. The amount of the MACs in patch-based inference increases with the reduction of the usage on the SRAM memory. Therefore, MCUNetv2 (MNv2) obtains a significant amount of the MACs over the other alternatives in Table 2. In addition, MCUNetv2+ (MNv2+) reduces roughly 8% of MACs on average over the baseline MCUNetv2. The output padding bypassing of MCUNetv2+ removes overhead on the padding data and achieves such a reduction on the amount of MACs. Moreover, StreamNet-1D (SN1D) performs 1D stream_y processing and saves the redundant computation through its stream buffer. Thus, StreamNet-1D reduces 65% of MACs on average over the baseline. To completely remove the redundant computation on patch-based inference (MCUNetv2), StreamNet-2D (SN-2D) performs 2D streaming processing to increase more data reuse than StreamNet-1D. Hence, StreamNet-2D reduces 83% of MACs on average over MCUNetv2 across 10 TinyML models. StreamNet-2D saves 93% and 95% of MACs on MI2 and MI3, since these models consist of many patch-based layers with the large split_idx value and the amount of MACs on the redundant computation is large.

5.4 Reveal the Latency of StreamNet

This evaluation aims to figure out if StreamNet can shorten the execution time of TinyML models by reducing the amount of their MACs. Table 3 compares the latency of each TinyML model across different implementations on MCUNetv2(MNv2) and StreamNet. The n_patch and split_idx in Table 3 and Table 2 use the same value. Table 3 shows the execution time when TinyML models use minimum SRAM memory space through patch-based inference. MCUNetv2+ (MNv2+), StreamNet1D (SN-1D), and StreamNet-2D (SN-2D) achieves a geometric mean of 1.1X, 3.4X, 7.3X speedup

over the baseline MCUNetv2. The speedup of Table 3 presents the consistent results over the amount of MACs in Table 2. Hence, this result provides StreamNet the evidence to exploit the MAC count obtained at compile time to infer the final latency in StreamNet auto-tuning framework.

GMEAN

0

1

2

3

4

MB2

PL

MI0

MI1

MI2

MI3

MI4

MV0 MV1 MV2

over MCUNetv2

TinyML Models

mem+0%

mem+5%

mem+10%

mem+15%

0

100

200

300

MB2

PL

MI0

MI1

MI2

MI3

Peak Memory Usage (kB)

TinyML Models

Layer-wise

MCUNetv2

StreamNet-1D

StreamNet-2D

416.1

(a) SRAM memory consumption

(b) Auto-tuning framework

5.5 Analysis of Peak SRAM Memory Usage

Figure 7(a) presents the peak SRAM memory usage of TinyML models on different implementations of MCUNetv2 (2) and StreamNet. MCUNetv2 (2), StreamNet-1D, and StreamNet-2D save 66%, 62%, and 47% SRAM memory usage on average over the layer-wise inference baseline, respectively. Unlike patch-based inference, StreamNet-1D reuses the data within the stream buffer that occupies the SRAM memory space. Thus, StreamNet-1D consumes additional 4% SRAM memory space on average to store the stream buffer over the MCUNetv2 (2). To increase the ratio of the data reuse, StreamNet-2D requires about 22% extra SRAM memory space on average for the stream buffer to cover more data reuse than StreamNet-1D. StreamNet provides a trade-off between the performance and the peak usage of the SRAM memory. Unlike patch-based inference, StreamNet improves the performance of TinyML models without sacrificing too much SRAM memory space.

5.6 Results of StreamNet Auto-tuning Framework

This evaluation aims to validate if StreamNet auto-tuning framework can create a case that satisfies its memory constraint while achieving the best performance. Figure 7(b) presents the performance speedup results when adding 0% to 15% memory space on each TinyML model based on the minimal SRAM memory usage of StreamNet-1D. In addition, we also design an auto-tuning framework for MCUNetV2 (2) that scans through its n_patch and split_idx parameters to obtain a case with the best performance while meeting the memory constraint. Unlike MCUNetV2 (2) auto-tuning framework, StreamNet searches for the combination of StreamNet-1D and StreamNet-2D network to fit the memory constraint while achieving the best performance. As illustrated in Figure 7(b), StreamNet achieves a geometric mean of 1.63X, 1.61X, 1.8X speedup over MCUNetV2 (2) when the memory constraint is the base memory space plus 5%, 10%, and 15%, respectively. In Figure 7(b), MI0

achieves only a speedup of 1.07X to 1.09X over MCUNetV2 (2), where the value of the split_idx and n_patches is small, leading to minor recomputation in MCUNetV2 (2). Unlike MI0, MI3 uses larger split_idx and n_patches values to meet the memory constraints. The large value of split_idx and n_patches allows StreamNet to eliminate the significant amount of redundant computation.

6 Related Work

Tiny Machine Learning (TinyML) TinyML has gained significant attention in recent years. Techniques such as model compression (9; 10), pruning (11; 12; 13; 14; 15) and quantization (11; 16; 17; 18; 19; 20; 21; 22), are employed to reduce model size and complexity while maintaining accuracy. Existing frameworks like TFLM (3) and microTVM (4) optimize the execution time of TinyML models on resource-constrained devices by utilizing CMSIS-NN (6) and CMix-NN (23) libraries for kernel implementation. MCUNet (1) further optimizes kernel implementation through vectorization, loop reordering, loop unrolling, and data layout transformation, resulting in more efficient performance.

Memory-Efficient Inference The peak usage of SRAM memory is a critical factor in deploying TinyML models on microcontrollers due to the tight SRAM constraints in MCUs. However, existing works (3; 24; 25) employ per-layer inference, which restricts model deployment on severely memoryconstrained MCUs. Several approaches have been proposed to address this issue in different ways. Some methods rearrange operators (26; 27) or use tensor memory planners (28) to save memory usage. Nonetheless, a single intermediate tensor may still be too large for the MCUs. As an alternative, models can be rewritten to reduce memory requirements (29; 30). However, this requires network modifications and may affect accuracy, reducing adaptability. MCUNetV2 (2) and TVM's cascade scheduling (31) address this problem by fusing multiple layers and introducing redundant computation. Prior work (32; 33; 34; 35; 4) also tackles similar problems as MCUNetv2 (2) in the ML inference. However, too many fused layers increase the amount of redundant computation.

Remove Recomputations To address recomputations introduced by the aforementioned approach, the main idea is to cache the computed result and reuse it when necessary. Halide (35) introduces sliding window optimization to remove recomputation, but it lacks flexibility in adjusting temporary buffers to reduce memory requirements across different dimensions. Prior work such as Fused-layer CNN accelerators (36) and streaming methods (37; 38; 39; 40; 41) offer solutions to reduce execution time in the data streaming manner, but none of them provides a method to mix 1D and 2D data streaming. Hence, our proposed StreamNet builds upon and extends existing streaming approaches by introducing a hybrid method that mixes different dimensional streaming to form 1D and 2D streaming techniques, skips the redundant computation shown in patch-based inference (2), and offers an auto-tuning framework to select StreamNet configurations.

7 Conclusion

Deploying TinyML models exposes challenges on MCUs with the stringent memory and power constraints. Unlike layer-wise inference, patch-based inference significantly decreases the usage of the SRAM memory and helps the deployment of TinyML models on MCUs. However, the redundant computation of patch-based inference enormously raises the latency of Machine Learning inference. Thus, this work proposes StreamNet that aims to eliminate such a performance bottleneck of patchbased inference. StreamNet designs 1D and 2D streaming processing to significantly remove the performance overhead of patch-based inference. Thus, StreamNet-2D achieves a geometric-mean speedup of 7.3X over patch-based inference across 10 TinyML models. Furthermore, StreamNet also skips the unnecessary computation for the padding data. Finally, StreamNet provides an auto-tuning framework to automatically search for the network that can satisfy the memory constraint of MCUs while achieving better performance over the existing patch-based inference.

References

J. Lin, W.-M. Chen, Y. Lin, C. Gan, and S. Han, 'Mcunet: Tiny deep learning on iot devices,' Advances in Neural Information Processing Systems , vol. 33, 2020.

J. Lin, W.-M. Chen, H. Cai, C. Gan, and S. Han, 'Mcunetv2: Memory-efficient patch-based inference for tiny deep learning,' in Annual Conference on Neural Information Processing Systems (NeuralPS) , 2021.

R. David, J. Duke, A. Jain, V. Janapa Reddi, N. Jeffries, J. Li, N. Kreeger, I. Nappier, M. Natraj, T. Wang, et al. , 'Tensorflow lite micro: Embedded machine learning for tinyml systems,' Proceedings of Machine Learning and Systems (MLSys) , vol. 3, pp. 800-811, 2021.

T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, M. Cowan, H. Shen, L. Wang, Y. Hu, L. Ceze, et al. , 'Tvm: An automated end-to-end optimizing compiler for deep learning,' arXiv preprint arXiv:1802.04799 , 2018.

Lin, Ji and Chen, Wei-Ming and Lin, Yujun and Gan, Chuang and Han, Song, 'MCUNet Model Zoo.' [Online]. Available: https://github.com/mit-han-lab/mcunet/blob/master/ mcunet/model_zoo.py , 2023. (accessed May. 16, 2023).

L. Lai, N. Suda, and V. Chandra, 'Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus,' arXiv preprint arXiv:1801.06601 , 2018.

ARM Ltd., 'Arm Mbed CLI.' [Online]. Available: https://github.com/ARMmbed/ mbed-cli , 2023. (accessed May. 16, 2023).

STI Ltd., 'STM32F767ZI Datasheet.' [Online]. Available: https://www.st.com/en/ microcontrollers-microprocessors/stm32f767zi.html , 2023. (accessed May. 16, 2023).

Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, 'Amc: Automl for model compression and acceleration on mobile devices,' in Proceedings of the European conference on computer vision (ECCV) , pp. 784-800, 2018.

S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and W. J. Dally, 'Eie: Efficient inference engine on compressed deep neural network,' in Proceedings of the International Symposium on Computer Architecture (ISCA) , p. 243-254, 2016.

S. Han, H. Mao, and W. J. Dally, 'Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,' arXiv preprint arXiv:1510.00149 , 2015.

J. Lin, Y. Rao, J. Lu, and J. Zhou, 'Runtime neural pruning,' Advances in neural information processing systems , vol. 30, 2017.

Y. He, X. Zhang, and J. Sun, 'Channel pruning for accelerating very deep neural networks,' in Proceedings of the IEEE international conference on computer vision , pp. 1389-1397, 2017.

Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, 'Learning efficient convolutional networks through network slimming,' in Proceedings of the IEEE international conference on computer vision , pp. 2736-2744, 2017.

Z. Liu, H. Mu, X. Zhang, Z. Guo, X. Yang, K.-T. Cheng, and J. Sun, 'Metapruning: Meta learning for automatic neural network channel pruning,' in Proceedings of the IEEE/CVF international conference on computer vision , pp. 3296-3305, 2019.

H. F. Langroudi, V. Karia, T. Pandit, and D. Kudithipudi, 'Tent: Efficient quantization of neural networks on the tiny edge with tapered fixed point,' arXiv preprint arXiv:2104.02233 , 2021.

M. Rusci, A. Capotondi, and L. Benini, 'Memory-driven mixed low precision quantization for enabling deep network inference on microcontrollers,' Proceedings of Machine Learning and Systems , vol. 2, pp. 326-335, 2020.

J. Choi, Z. Wang, S. Venkataramani, P. I.-J. Chuang, V. Srinivasan, and K. Gopalakrishnan, 'Pact: Parameterized clipping activation for quantized neural networks,' arXiv preprint arXiv:1805.06085 , 2018.

K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han, 'Haq: Hardware-aware automated quantization with mixed precision,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8612-8620, 2019.

M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, 'Xnor-net: Imagenet classification using binary convolutional neural networks,' in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV , pp. 525542, Springer, 2016.

S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, 'Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,' arXiv preprint arXiv:1606.06160 , 2016.

C. Zhu, S. Han, H. Mao, and W. J. Dally, 'Trained ternary quantization,' arXiv preprint arXiv:1612.01064 , 2016.

A. Capotondi, M. Rusci, M. Fariselli, and L. Benini, 'Cmix-nn: Mixed low-precision cnn library for memory-constrained edge devices,' IEEE Transactions on Circuits and Systems II: Express Briefs , vol. 67, no. 5, pp. 871-875, 2020.

N. Rotem, J. Fix, S. Abdulrasool, G. Catron, S. Deng, R. Dzhabarov, N. Gibson, J. Hegeman, M. Lele, R. Levenstein, et al. , 'Glow: Graph lowering compiler techniques for neural networks,' arXiv preprint arXiv:1805.00907 , 2018.

H.-I. C. Liu, M. Brehler, M. Ravishankar, N. Vasilache, B. Vanik, and S. Laurenzo, 'Tinyiree: An ml execution environment for embedded systems from compilation to deployment,' arXiv preprint arXiv:2205.14479 , 2022.

E. Liberis and N. D. Lane, 'Neural networks on microcontrollers: saving memory at inference via operator reordering,' arXiv preprint arXiv:1910.05110 , 2019.

B. H. Ahn, J. Lee, J. M. Lin, H.-P. Cheng, J. Hou, and H. Esmaeilzadeh, 'Ordering chaos: Memory-aware scheduling of irregularly wired neural networks for edge devices,' Proceedings of Machine Learning and Systems , vol. 2, pp. 44-57, 2020.

Y. Pisarchyk and J. Lee, 'Efficient memory management for deep neural net inference,' arXiv preprint arXiv:2001.03288 , 2020.

T. Jin and S. Hong, 'Split-cnn: Splitting window-based operations in convolutional neural networks for memory system optimization,' in Proceedings of the International Conference on Architectural Support for Programming Languages and Operation Systems (ASPLOS) , pp. 835-847, 2019.

O. Saha, A. Kusupati, H. V. Simhadri, M. Varma, and P. Jain, 'Rnnpool: Efficient non-linear pooling for ram constrained inference,' Advances in Neural Information Processing Systems , vol. 33, pp. 20473-20484, 2020.

ARM ltd, 'Miniaturizing Models for microNPUs: a Cascading Scheduler.' [Online Available:] https://www.youtube.com/watch?v=4MVsOwt5YLo , 2022. (accessed Jan. 20, 2023).

J. Zhao and A. Cohen, 'Flextended tiles: A flexible extension of overlapped tiles for polyhedral compilation,' ACM Transactions on Architecture and Code Optimization (TACO) , vol. 16, no. 4, pp. 1-25, 2019.

X. Zhou, J.-P. Giacalone, M. J. Garzarán, R. H. Kuhn, Y. Ni, and D. Padua, 'Hierarchical overlapped tiling,' in Proceedings of the Tenth International Symposium on Code Generation and Optimization , pp. 207-218, 2012.

K. Goetschalckx and M. Verhelst, 'Breaking high-resolution cnn bandwidth barriers with enhanced depth-first execution,' IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 9, no. 2, pp. 323-331, 2019.

J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, and S. Amarasinghe, 'Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines,' Acm Sigplan Notices , vol. 48, no. 6, pp. 519-530, 2013.

M. Alwani, H. Chen, M. Ferdman, and P. Milder, 'Fused-layer cnn accelerators,' in 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) , pp. 1-12, IEEE, 2016.

O. Rybakov, N. Kononenko, N. Subrahmanya, M. Visontai, and S. Laurenzo, 'Streaming keyword spotting on mobile devices,' arXiv preprint arXiv:2005.06720 , 2020.

A. Coucke, M. Chlieh, T. Gisselbrecht, D. Leroy, M. Poumeyrol, and T. Lavril, 'Efficient keyword spotting using dilated convolutions and gating,' in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6351-6355, IEEE, 2019.

Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Y. Wu, R. Pang, et al. , 'Streaming end-to-end speech recognition for mobile devices,' in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6381-6385, IEEE, 2019.

X. Chen, Y. Wu, Z. Wang, S. Liu, and J. Li, 'Developing real-time streaming transformer transducer for speech recognition on large-scale dataset,' in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5904-5908, IEEE, 2021.

Y. Wang, H. Lv, D. Povey, L. Xie, and S. Khudanpur, 'Wake word detection with streaming transformers,' in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5864-5868, IEEE, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents methods for speeding up patch-based inference on microcontrollers. The method creates a buffer to selectively save intermediate values that are traditionally discarded in patch-based computation. This allows StreamNet to balance latency and memory consumption. There are 1d and 2d variants of this optimization.\n\nThe paper additionally proposes a method for skipping the computation of padding data and a framework for auto-tuning the hyperparameters of StreamNet. The framework searches for patch hyperparameters that meet certain memory constraints while minimizing latency.\n\nSTRENGTHS:\n- Pure runtime optimization targeted at the most important MCU metrics (latency and memory)\n- State-of-the-art performance vs. a strong baseline (MCUNetv2) \n- Enables a new dimension for latency-memory tradeoffs\n- Includes an algorithm for automatically searching the newly created search space.\n\nWEAKNESSES:\n- Given the tradeoff between latency and memory consumption that StreamNet unlocks, it would be easier to interpret results as points on a latency-memory Pareto curve rather than cross-referencing tables and charts. The table in the appendix that gave the speedup at nearly equal memory was the most informative in understanding the benefits, but it was buried. \n\n- Only compares against MCUNetv2 and not the other patch-based optimization mentioned in the related work.\n\n- Relies heavily on existing work (MCUNetv2 and TinyEngine) \n\n- The contributions in the intro should be reworded to be consistent\n\nQUESTIONS:\n- Is this method purely a runtime/compiler optimization, or are there any model architecture implications (e.g. on accuracy)?\n\n- How might one redesign MCU class models to maximize the benefits of StreamNet? If StreamNet more easily optimizes certain layer configurations, how does that impact exiting Pareto curves?\n\n- Will StreamNet be open-sourced?\n\nLIMITATIONS:\nHow long does it take to run the auto-tuning framework? How does it compare to black-box search methods?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces StreamNet, a novel approach designed to eliminate the performance bottleneck associated with patch-based inference, which incurs additional computational overheads due to overlapping patches. StreamNet comprises two techniques, StreamNet-1D and StreamNet-2D, each offering a different trade-off between memory overhead for buffering intermediate results and computational overhead reduction. The authors also propose an auto-tuning framework to derive an inference schedule, which is a composition of the two techniques, given the memory constraints of a Microcontroller Unit (MCU). Evaluation results indicate that the proposed approach significantly improves upon prior work in terms of memory usage and latency.\n\nSTRENGTHS:\n- Clear and comprehensive illustration of the method: The paper provides good illustrations of the proposed methods, making them easy to understand. The examples used to explain the methods are also helpful in allowing readers to understand the advantages of the proposed methods and the trade-off between design elements.\n- Intuitive and straightforward methods: The proposed StreamNet-1D and StreamNet-2D techniques are intuitive and straightforward, offering a clear path to reducing computational overheads by buffering tensors shared between patches.\n- Practicality and on-device evaluation: The paper demonstrates the practicality of the proposed methods through on-device evaluation, which is a strength. However, more comprehensive testing is needed.\n\nWEAKNESSES:\n- Limited experimental settings: The experimental setting does not seem to cover the common user scenario of employing patch-based inference. This limits the generalizability of the results and makes it difficult to assess the full potential of the proposed methods. Future work should include more diverse testing scenarios to fully evaluate the effectiveness and applicability of StreamNet. In my opinion, two control variables, input resolution and number of patches, are missing in the experiments. \n    * Patch-based inference is favorable in settings with large input resolution where the overhead of re-computation can be amortized by the large patch size. For instance, the re-computation overhead reported in MCUNetV2 is only 10% for MobileNetV2, but such overhead appears significantly higher in this paper. What causes the discrepancy requires further discussion.\n    * The number of patches used also significantly influences the re-computation overheads and memory usage. It would be intriguing to observe the trade-off between latency and memory usage achieved by the proposed method in comparison to the baselines.\n\nQUESTIONS:\n1. Could you provide more insight into how the current experimental setup mirrors different user scenarios, particularly those that commonly employ patch-based inference?\n2. Could you provide the original latency and memory usage of the models without patch-based inference as reference data points?\n3. How would varying the input resolution and the number of patches impacts the performance of StreamNet?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
The patch-based inference is widely employed for TinyML models on resource-constrained microcontroller units (MCUs), which significantly reduces memory requirements compared to layer-based inference. However, path-based inference can lead to a substantial increase in Multiply-Accumulates (MACs), as it introduces a great deal of redundant computation among adjacent patches. This paper introduces StreamNet, a solution that curtails repeated computation by memorization. Furthermore, StreamNet can auto-tune the area to be skipped during path-based inference, enhancing overall efficiency.

STRENGTHS:
1. The paper provides a comprehensive explanation of the benefits and drawbacks of patch-based inference.
2. The proposed method is clearly articulated and easy to understand.
3. StreamNet not only reduces MACs by leveraging the memory of the overlapped computation area, but it also introduces an auto-tuning framework to streamline these areas.
4. StreamNet outperforms the state-of-the-art methods at a relatively minor expense of additional memory space.
5. The auto-tuning framework possesses the ability to further accelerate inference automatically if more memory space is made available.

WEAKNESSES:
1. It would be better to list the potential overlapped area percentage for several commonly used tinyML models.

QUESTIONS:
I enjoyed reading the paper, and the idea is well present, and the gains are intuitively making sense. My only question is that what is the reuse distance of patches? Is there any case that the stream buffer is not sufficient to capture all the reuses?

LIMITATIONS:
not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

comments: SUMMARY:
The processing of patch-based inference for MCUs induce a large number of redundant MACs against the layer-wise processing because of the overlapped processing. In order to address this problem,  this work designs StreamNet that employs the stream buffer to eliminate the redundant computation of patch-based inference. StreamNet uses 1D and 2D streaming processing and an auto-tuning framework to significantly improve the performance of patch-based inference with minimal requirements on the MCU’s SRAM memory space.

STRENGTHS:
1) The proposed streamnet removes the computing redundancy in prior patch-based DNN processing framework and improves DNN inference performance significantly. 

2) The paper is well organized and easy to follow. The experiments are sufficient and to the point.

WEAKNESSES:
StreamNet essentially introduces additional buffers to explore the redundant computing results induced by patched DNN processing without compromising the memory requirements. The major contribution will be the DNN computing system or implementation optimization and the novelty is relatively limited.

QUESTIONS:
1) According to the experiments, we notice that the performance speedup is sensitive to the different neural network architectures say kernel sizes and feature map sizes. While the benchmarks are mostly obtained from NAS and it is a bit difficult to evaluate the representation of these models. Could you provide more details about the models such as the number of layers, model sizes, and accuracy?

2) Transformer models are increasingly utilized, will the proposed framework be applicable to transformer models?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06