Memory-efficient Patch-based Inference for Tiny Deep Learning
Authors: Ji Lin
Wei-Ming Chen
Han Cai
Chuang Gan
Song Han
Venue: NeurIPS 2021 Poster (2021)
Keywords: tiny deep learning
tinyML

Abstract: Tiny deep learning on microcontroller units (MCUs) is challenging due to the limited memory size. We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs:  the first several blocks have an order of magnitude larger memory usage than the rest of the network. To alleviate this issue, we propose a generic patch-by-patch inference scheduling, which operates only on a small spatial region of the feature map and significantly cuts down the peak memory. However, naive implementation brings overlapping patches and computation overhead. We further propose receptive field redistribution to shift the receptive field and FLOPs to the later stage and reduce the computation overhead. Manually redistributing the receptive field is difficult. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2. Patch-based inference effectively reduces the peak memory usage of existing networks by4-8×.  Co-designed with neural networks, MCUNetV2 sets a record ImageNetaccuracy on MCU (71.8%) and achieves >90% accuracy on the visual wake words dataset under only 32kB SRAM. MCUNetV2 also unblocks object detection on tiny devices, achieving 16.9% higher mAP on Pascal VOC compared to the state-of-the-art result. Our study largely addressed the memory bottleneck in tinyML and paved the way for various vision applications beyond image classification.

MCUNetV2: Memory-Effiiient Patch-based Inference for Tiny Deep Learning
Ji Lin' Wei-Ming Chen'
Han Cai Chuang Gan
Song Han
MIT
"MIT-IBM Watson Al Lab https://mcune .itt edu
Abstract
Tiny deep learning on microcontroller units (MCUs) is challenging due the limited memory size. We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs: the first several blocks have an order of magnitude larger memory usage than the rest of the network. To alleviate this issue, we propose generie patch-by-pated inference scheduling, which operates only small spatial region of the feature map and significantly cuts down the peak memory. However, naive implementation brings overlapping patches and computation overhead. We further propose receptive field redistribution to shift the receptive field and FL.OPs the later stage and reduce the computation overhead. Manually redistributing the receptive field is difficult. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNeVV2. Patch-based inference effectively reduces the peak memory usage existing networks by 4-8x. Co-designed with neural networks, MCUNetV2 sets record ImageNet accuracy on MCU (71.8%). and achieves >90% accuracy on the visual wake words dataset under only 32kB SRAM. MCUNetV2 also unblocks object detection on tiny devices, achieving 16.9% higher mAP on Pascal VOC compared to the state- of-the-art result. Our study largely addressed the memory bottleneck in tinyML. and paved the way for various vision applications beyond image classification.
Introduction
IoT devices based on tiny hardware like microcontroller units (MCUs) are ubiquitous nowadays. Deploying deep learning models on such tiny hardware will enable us to democratize artificial intelligence. However, tiny deep learning is fundamentally different from mobile deep learning due to the tight memory budget [27]: common MCU usually has an SRAM smaller than 512kB, which is too small for deploying most off-the-shelf deep learning networks. Even for more powerful hardware like Raspberry Pi 4, fitting inference into the 1.2 cache (1MB) can significantly improve energy efficiency. These pose new challenges to efficient AI inference with a small peak memory usage.
Existing work employs pruning [16, 18, 19, 32, 31], quantization [16, 54, 53, 37, 46, 9, 39]. and neural architecture search [6, 44, 47, 5] for efficient deep learning. However, these methods focus on reducing the number of parameters and FLOPPs, but not the memory bottleneck. The tight memory budget limits the feature map/activation size, restricting us to use a small model capacity or a small input image size. Actually, the input resolutions used in existing tinyML work are usually small (< 2242) [27], which might be acceptable for image classification (e.g., ImageNet [11], VWW [10]), but not for dense prediction tasks like objection detection: as in Figure 2, the performance of object detection degrades much faster with input resolution than image classification. Such restriction hinders the application of tiny deep learning many real-life tasks (e.g., person detection).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

layer inference
peak mem: 1372kB peak mem: 172kB
per-patch inference
per-layer inference
1400
High mem.
Low mem.
Per layer memorry
Memory Usage
1120
original
smaller
Per-patch memery
840
peak mem.
per-patch
560
peak mem.
280
256kB constraint
2 3
5
10
11
12
13
14
16
16
17
Block Index
Figure 1. MohileNetV2 [41] has a very imbalanced memory usage distribution. The peak memory is determined by the first blocks with high peak memory. while the later blocks all share a small memory usage. By using per per-patch inference (4 x 4 patches), we are able to significantly reduce the memory usage of the first 5 blocks, and reduce the overall peak memory by 8x, fitting MCUs with 256kB memory budget. Notice that the model architecture and accuracy are not changed for the two settings. The memory usage measured in int8.
We perform an in-depth analysis on memory usage of each layer in efficient network designs and find that they have a very imbalanced
75
(%)
activation memory distribution. Take MobileNetV2 [41] as example,
70
as shown in Figure 1, only the first 5 blocks have a high peak memory (>450kB), becoming the memory bottleneck of the entire network. The remaining 13 blocks have low memory usage, which can casily fit a 256kB MCU. The peak memory of the initial memory-intensive stage is 8x higher than the rest of the network. Such a memory pattern leaves a huge room for optimization: if we can find way to "bypass" the Figure 2. Detection is more memory-intensive stage, we can reduce the overall peak memory by 8x. sensiitive to smaller resolutions.
degraiis
larger
65
A
VOC mAP ImgNet Top-1
60
160
224
228
352
Inagge Resolution
In this paper, we propose MCUNetV2 address the challenge. We first propose a patch-by- battchh execution order for the initial memory-intensive stage of CNNs (Figure 3). Unlike conventional layer-by-layer execution, it operates on a small spatial region of the feature map a time, instead of the whole activation. Since we only need to store the feature of a small patch, we can significantly cut down the peak memory of the initial stage (blue to yellow in Figure 3), allowing us to fit a larger input resolution. However, the reduced peak memory comes at the price of computation overhead: in order to compute the non-overlapping output patches, the input image patches need to be overlapped (Figure 3(b),, leading to repeated computation. The overhead is positively related to the receptive field of the initial stage: the larger the receptive field, the larger the input patches, which leads to more overlapping. We further propose receptive field redistribution to shift the receptive field and workload to the later stage of the network. This reduces the patch size as well as the computation overhead caused by overlapping, without hurting the performance of the network. Finally, patch-based inference brings larger design space for the neural network, giving us more freedom trading-off input resolution, model size, etc. We also need to minimize the computation overhead under patch-based execution. To explore such a large and entangled space, we propose to jointly design the optimal deep model and its inference schedule with neural architecture search given a specific dataset and hardware.
Patch-basee inference effectively reduces the peak memory usage of existing networks by 4-8x (Figure 5). The results are further improved when co-designing neural architecture with inference scheduling. On ImageNet [1]], we achieve record accuracy of 71.8% on MCU (Table 2); visual wake words dataset [10], we are able to achieve >90% accuracy under only 32kB SRAM, which is 4.0x smaller compared to MCUNetVI [27]. greatly lowering the boundary of tiny deep learning (Figure 7). MCUNetV2 further unlocks the possibility to perform dense prediction tasks on MCUs (e.g., object detection), which was not practical due to the limited input resolution. We are able to achieve 64.6% mAP under 256kB SRAM constraints and 68.3% under 512kB, which is 16.9% higher compared to the existing state-of-the-ar solution, making object detection applicable on a tiny ARM Cortex- M4 MCU. Our contributions can be summarized as follows:
We systematically analyze the memory usage pattern of efficient CNN designs and find that they suffer from a imbalanced memory distribution, leaving a huge room for optimization.
We propose a patch-based inference scheduling to significantly reduce the peak memory required for running CNN models, together with receptive field redistribution to minimize the computation overhead.
2

---

In memory
Not in memory
Currenly executing
To be executed
Overlapped
cony
cOnV
(a) Per-layer computation (executing fisst conv)
(b) Per-patch computation (executing first patch)
Figure 3. Per-patch inference can significantly reduce the peak memory required to execute sequence of convolutional layers. We study two convolutional layers (stride 1 and 2). Under per-layer computation (a), the first convolution has a large input/ouiput activation size, dominating the peak memory requirement. With per-patch computation (b), we allocate the buffer to host the final ouiput activation, and compute the results patch-by-pateh. We only need to store the activation from one patch but not the entire feature map, reducing the peak memory (the first input is the image, which can be partially decoded from a compressed format like JPEG).
With the joint design of network architecture and inference scheduling, we achieve a record performance for tiny image classification and objection detection on MCUs. Our work largely addressed the memory bottleneck for tinyML, paving the way for various vision applications.
2 Understanding the Memory Bottleneck of Tiny Deep Learning
We systematically analyze the memory bottleneck of CNN models.
Imbalanced memory distribution. As an example, we provide the per-block peak memory usage of MobileNetV2 [41] in Figure The profiling done in int 8 (details in Section 4). We can observe clear pattern of imbalanced memory usage distribution. The first 5 blocks have large peak memory, exceeding the memory constraints of MCUs, while the remaining 13 blocks easily fit 256kB memory constraints. The third block has 8x larger memory usage than the rest of the network, becoming the memory bottleneck. We also inspect other efficient network designs and find the phenomenon quite common across different CNN backbones, even for models specialized for memory-limited microcontrollers [27]. The detailed statistics are provided in the supplementary.
We find that this situation applies to most single-branch or residual CNN designs due to the hier- archical structure": after each stage, the image resolution down-sample by half, leading to 4x fewer pixels, while the channel number increases only by 2x [42, 17, 22] or by an even smaller ratio [41, 21, 45], resulting a decreasing activation size. Therefore, the memory bottleneck tends to appear at the early stage of the network, after which the peak memory usage is much smaller.
Challenges and opportunities. The imbalanced memory distribution significantly limits the model capacity and input resolution executable on MCUs. In order to accommodate the initial memory- intensive stage, the whole network needs to be scaled down even though the majority of the network already has a small memory usage. It also makes resolution- -enssiivve tasks (e.g.. object detection) difficult, as a high-resolution input will lead large initial peak memory. Consider the first convolu- tional layer in MobileNetV2 [41] with input channels 3, output channels 32, and stride 2, running it on an image of resolution 224 x 224 requires a memory of 224² + 32 x 1122 539kB even when quantized in int which cannot be fitted into microcontrollers On the other hand, if we can find a way "bypass" the initial memory-intensive stage, we can greatly reduce the peak memory of the whole network, leaving us large room for optimization.
3 MCUNetV2: Memory-Efticient Patch-based Inference
3.1 Breaking the Memory Bottleneck with Patch-based Inference
We propose to break the memory bottleneck of the initial layers with patch-based inference (Figure 3). Existing deep learning inference frameworks (e.g., TensorFlow Lite Micro [1], TinyEngine [27], MicroTVM [8], etc.) use a layer-by-layer exccution. For each convolutional layer, the inference library first allocates the input and output activation buffer in SRAM, and releases the input bufferr after the whole layer computation is finished. Such implementation makes inference optimization easy (e.g., im2col, tiling, etc.), but the SRAM has to hold the entire input and output activation for
"some CNN designs have highly complicated branching structure (e.g., NASNet [56]). but they are generally less efficient for inference [34, 44, 6]: thus not widely used for edge computing.
3

---

MBV2 large perk memory/per-eae per-pateh
small pesk memoory/pee-lgeee
MDD6 3x3
MB2 3x3
MB8 3x3
MB5 3x.3
M156 3x.
Pooking FC
MD
MB6
MBs
MBS
MPS
MBS
MIDS
MB66
Reducce
Increase
MbV2-RD
Recaptiva Field
Receptivee Fleld
MB6 3x3
MBB 3x3
MBB 3x3
MBS 3x3
MBG 3x3
3x3
MIL6 3xd
MER6 3x3
MB6 1xT
M166 3x3
MB6 3x
MB8 3x3
Payling FC
MER
MEDE
MBG
MFH
MID6
Figure 4. The redistributed MobileNetV2 (MbV2-RD) has reduced receptive field for the per-patch inference stage and increased receptive field for the per-layer stage. The two networks have the same level of performance, but MbV2-RD a smaller overhead under patch-based inference. The mobile inverted block is denoted as MB (expansion ratio) {kernel sizc). The dashed border means stride=2.
Table 1. Per-patch inference reduces the peak memory by 8x for MobileNetV2 [41] (1372KB to 172kB), but it increases the overall computation by 10% due to patch overlapping. We futher propose receptive field redistribution (M)D2-RD) which reduces the overall overhead to only 3% without hurting performance.
Patch Size
Comp. overhead
MAC5 panches)
Peak SRAM
ImgNet Top-1
VOC
Model
mAP
patch-stage overall
patch-stage overall
per-layer
per-patch
MbV2 [41]
75
+42%
+10%
130M
330M
1372kB
172KB (8x1)
72.2%
75.4% 75.7%
MbV2-RD
63
+18%
+3%
73M
301M
1372kB
172kB (8x.))
72.1% 1
each layer, which is prohibitively large for the initial stage of the CNN. Our patch- based inference runs the initial remooy-intensivv stage in a patch-by-per b--phhhy-hhhh manner. For each time, we only run the model on a small spatial region (>10x smaller than the whole area), which effectively cuts down the peak memory usage. After this stage is finished, the rest of the network with a small peak memory is executed in normal myyer-b--lue layer manner (upper notations in Figure 1).
We show an example of two convolutional layers (with stride and 2) in Figure 3. For conventional per-layer computation, the first convolutional layer has large input and output activation size, leading high peak memory. With spatial partial computation, we allocate the buffer for the final output and compute its values patch instead of the whole feature map. Note that the first activation the input image, which can be partially decoded from a compressed format like JPEG and does not require full storage.
In this manner, we only need to store the activation from one
Computation overhead. The significant memory saving comes at the cost of computation overhead. To maintain the same output results as per-layer inference, the non-overlapping output patches correspond to overlapping patches in the input image (the shadow area in Figure 3(b)). This is because convolutional filters with kernel size >1 contribute to increasing receptive fields. The bordering pixel on the output patches is dependent repeated computation can increase the overall network computation by 10- 10-77% even under optimal hyper-parameter choice (Figure 5), which is undesirable for low-power edge devices.
the inputs from neighboring patches. Such
3.2 Reducing Computation Overhead by Redistributing the Receptive Field
The computation overhead is related to the receptive field of the patch- initial stage. Consider the output of the patch-based stage, the larger receptive field it has the input image, the larger resolution for each patch, leading to a larger overlapping area and repeated computation (see Section 4,4 for quantitative analysis). For MobileNetV2, if we only consider down-sampling, each input patch has side length of 224/4- = 56. But when considering the increased receptive field, each input patch has to use shape of 75 x 75, leading to a large overlapping area.
We propose to redistribute the receptive field (RF) of the CNN to reduce computation overhead. The basic idea is: (I) reduce the receptive field of the patch- based initial stage; (2) increase the receptive field of the later stage. Reducing RF for the initial stage helps to reduce the size of each input patch and repeated computation. However, some tasks may have degraded performance if the overall RF smaller (e.g., detecting large objects). Therefore, we further increase the RF of the later stage to compensate for the performance loss.
We take MobileNerV2 as a study case and modify its architecture. The comparison is shown in Figure 4. We used smaller kernels and fewer blocks in the per-patch inference stage, and increased

---

the number of blocks in the later per-layer inference stage. The process needs manual tuning and varies case-by-case. We will later discuss how we automate the process with NAS. We compare the performance of the two architectures in Table 1. Per-patch inference reduces the peak SRAM AM by 8x for all cases, but the original MobileNetV2 design has 42% computation overhead for the patch-based stage and 10% for the overall network. After redistributing the receptive field ("MbV2-RD"), we can reduce the input patch size from 75 to 63, while maintaining the same level of performance in image classification and object detection. After redistribution, the computation overhead is only 3%, which is negligible considering the benefit in memory reduction.
3.3 Joint Neural Architecture and Inference Scheduling Search
Redistributing the receptive field allows us to enjoy the benefit of memory reduction at minimal computation/latency overhead, but the strategy varies case-by-case for different backbones. The reduced peak memory also allows larger freedom when designing the backbone architecture (e.g.. using larger input resolution). To explore such a large design space, we propose to jointly optimize the neural architecture and the inference scheduling in an automated manner. Given a certain dataset and hardware constraints AR limit, Flash limit, latency limit, etc.), our goal is achieve the highest accuracy while satisfying all the constraints. We have the following knobs to optimize:
Backbone optimization. We follow [27] to use a MnasNet-alikee search space [44, 6] for NAS, so that we can have a fair comparison. The space includes different kernel sizes for each inverted residual block k (3/5/7), different expansion ratios et] (3/4/6), and different number of blocks for each stage d[[ (2/3/4). More recent search space designs like MobileNetV3 [21] have better accuracy-computation trade-off, but are hard to quantize due to Swish activation function [36], making deployment on MCU difficult. As shown [27]. the search space configuration (i.e., the global width multiplier w and imput resolution r) is crucial to the final NAS performance. We argue that the best search space configuration is not only hurdware-awan but also task-aware: for example, some tasks may prefer higher resolution over larger model size, and vice versa. Therefore, we also put r and w into the search space. We further extend w to support per-block width scaling WIJ- Including wy (0.5/0.75/1..0 10) and (96-256) expands the search space scalability, allowing us to fit different MCU models and tight resource budgets (ablation study provided in the supplementary).
Inference scheduling optimization. Given a model and hardware constraints, we will find the best inference scheduling. Our inference engine is developed based on TinyEngine [27] further patch-based inference. Apart from the optimization knobs in TinyEngine, we also need to determine the patches number p and the number of blocks n to perform patch-based based inference, so that the inference satisfies the SRAM constraints, According to Section 4.4, a smaller and n lead to a smaller computation overhead and faster inference. But varies case-by-case, so we jointly optimize it with the architecture.
Joint search. We need to co-design the backbone optimization and inference scheduling. For example, given the same constraints, we can choose to use smaller model that fits per-layer execution (p 1, no computation overhead), or a larger model and per- per-patch inference (p >> 1, with small computation overhead). Therefore, we put both sides in the same loop and use evolutionary search to find the best set of (k1 J. [[], d[j 1 "", n)) satisfying constraints. Specifically, we randomly sample neural networks from the super network search space; for each sampled network, we enumerate all the p and choices (optimized together with other knobs in TinyEngine) and find the satisfying combinations. We then report the best (p. n) pair with minimal computation/latencyy and use the statisties supervise architecture search. We provide the details and pseudo code in supplementary.
4 Experiments
Memory profiling. The memory usage is dependent on the inference framework implementa- tion [27]. To ease the comparison, we study two profiling settings:
(1) We first study analytic profiling, which is only related to the model architecture but not the inference framework. Following [10, 40), the memory required for layer the sum of input and output activation (since weights can be partially fetched from Flash); for networks with bulti-branches (e.g., residual connection), we consider the sum of memory required for branches at the same time (if the same input is shared by multiple branches, it will only be counted once).
5

---

Per-layer
Per-parch (4x4)
1400
480
1120
360
1056
redistribute
1396
451
840
560
smaller
B.Dx Snaller
smaller
701
smaleer
omaaoor
240
801
SUI
280
211
0
MbV2
MbV2-RD OFA-CPU MaasNet (a) Peak Memory (kB)
FBNet-A
M6V2
MnasNet FBNet-A (b) Computation MACs)
MbV2-RD
Figure 5. Analytical profiling: patch-base based inference significantly reduces the inference peak memory by 3.7-8.0x at small computation overhead of 8- 8-17%. The memory reduction and computation overhead related to the network design. For MobileNetV2, redistributing receptive field. All networks take an input resolution of 221 4 patches.
reduce the computation overhead from 10% to 3% by
Por-layer (w/ TingEng
Por-patch (2-2)
Par-paich (3x3)
+155
radistribuut
256
600
741
+4
192
SEP S64
D4
400
128
200
MbV2 5,
FBNet-A Mcasured Peak SRAM (kB)
MbV2-RD
MCUNet w..0,,
MbV2
FBNet-A (b) Measured Lateney (ms)
MbV2-RD
MCUNet
W0.5.
wrt.5,
Figure On-device measurement: patch-based inference reduce the measured peak SRAM usage by 4-6x when running on MCUs. The latency overhead could be large for some networks, but we can reduce it to 4% with proper architecture design (MbV2-RD). We scale down width and resolution to fit MCU memory.
(2) We also study or-device profiling to report the measured SRAM and Flash usage when executing deep model on MCU. The number is usually larger than the analytic results since we need to account for temporary buffers storing partial weights, Im2Col buffer, etc.
Datasets. We analyze the advantage of our method on image classification datasets: ImageNet [1I]
as the standard benchmark, and Visual Wake Words [10] to reflect TinyML applications. We further validate our method on object detection datasets: Pascal VOC [13] and WIDER FACE [48] to show advantage: be able to fit larger resolution the MCU.
Training& mee.loyment. We follow [27] for super network training and evolutionary search, detailed in the supplementary. Models quantized to int for deployment. We extend TinyEngine [27] to support patch-based inference, and benchmark the models on MCU models with different hardware resources: STM32F412 (Cortex-M4, 256kB SRAM/IMB Flash), STM32F746 (Cortex-M7, 320kB SRAM/IMB Flash), STM32H743 (Cortex-M7, 512kB SRAM/2MB Flash).
4.1 Reducing Peak Memory of Existing Networks
We first analyze how patch-based inference can significantly reduce the peak memory for model inference, both in analytic profiling and on-device profiling.
Analytic profiling. We study several widely used deep network backbones designed for edge inference in Figure 5: MobileNetV2 [41] (MbV2), redistributed MobileNerV2 (MbV2-RD), Once- For-All CPU (OFA-CPU) [5], MnasNet [44], and FBNet-A [47]. All the networks use an input resolution of 224 224; for patch-based inference, used x patches. The memory profiled in int8. Per-patch inference significantly reduces the peak memory by 3.7-8.0x, while only incurring 8-17% of computation overhead. For MobileNetV2, we can reduce the computation overhead from 10% to 3% by redistributing the receptive field without hurting accuracy (Table 1). The memory saving and computation reduction are related to the network architecture. Some models like MnasNet have a larger overhead since it uses large kernel sizes in the initial stage, which increases receptive fields. shows the necessity to co-design the network architecture with the inference engine.
On-device measurement. We further profile existing networks rumning on STM32F746 MCU, We measure the SRAM usage of the network with per-layer and per-patch (2 or 3 x 3 patches) inference. Due to the memory limit of MCU (320kB SRAM, 1MB Flash), we have to scale down the width multiplier w and input resolution r. Figure 6, per-patch based inference reduces the measured peak SRAM by 4-6x. Some models may have a large latency overhead, since the initial

---

Table 2. MCUNetV2 significantly improves the ImageNet accuracy microcontrollers, outperforming the state-of-the-arts by 4.6% under 256kB SRAM and 3.3% under 512kB. Lower or mixed precisions (marked gray) are orthogonal techniques, which we leave for future work. Out-of-memory (OOM) results are struck out.
Model / Library
Quant.
MACs
SRAM
Flash
Top-1
Top-5
STM32F412 (256kB SRAM, 1MB Flash)
MbV2 0.35 x (r= 144) [41] Tiny TinyEngine [27]
int8
24M
308kB
862kB
49.0%
73.8%
Proxyless 0.3x x ()=176) [6] /Tinynngine [27]
int8
38M
292kB
892kB
56.2%
79.7%
MbV1 0.5x (r=192) [22] Rusci al. [39]
mixed
110M
<256kB
<IMB
60.2%
MCUNet (TmyyNAS/ / TinyEngine) [27]
int8
68M
238kB
1007KB
60.3%
MCUNet (TinyNAS / TinyEngine) [27]
int4
134M
233kB
1008kB
62.0%
MCUNetV2-M4
int8
119M
196kB
1010kB
64.9%
86.2 %
STM32H743 (512kB SRAM, 2MB Flash)
MbV1 0,75x (x=224) [22) Rusci et al. [39]
mixed
317M
<512kB
<2MB
68 0%
MCUNet (TinyNAS / TinyEngine) [27]
int8
126M
452kB
2014kB
68.5%
MCUNet (TinyNAS/ TinyEngine) [27]
int4
474M
498kB
2000kB
70.7%
MCUNetV2-117
int8
256M
465kB
2032kB
71.8%
90.79 %
MCUNetV2
MCUNet
MDV2+TF-Litc
Proxyless TF-Licc
VWW Accuracy (%)
94
62kB
1188B
94
256kB
92
44.0%
constraint
92
T38 raster
90
1.0rsmaller
on MCU
90
30kB
1+..6%
88
88
86
86
SRAM 320kB
Flash
IMB
Flash 1MB
84
84
20
88
156
224
292
360
150
320
490
660
830
1000
Measured Peak SRAM (kB)
Measured Latency (ms)
(a) Trade-off: accuracy v.s. peak SRAM
(b) Trade-off: accuracy uss. measured latency
Figure 7. Left: MCUNeVV2 has better visual wake word (VWW) accuracy vs. peak SRAM trade-off. Compared
MCUNet [27]. MCUNeIV2 achieves better accuracy at 4.0x smaller peak memory. achieves >90% accuracy under <32kB memory, facilitating deployment on extremely small hardware. Right: patch-based method expands the search space that can fit the MCU, allowing better accuracy vs. latency tradc-off.
stage has worse hardware utilization. But with a proper architecture design (MbV2-RD), we can reduce the latency overhead 4%, which is negligible compared the memory reduction benefit.
4.2 MCUNetV2 for Tiny Image Classification
With joint optimization of neural architecture and inference scheduling, MCUNetV2 significantly pushes the state-of-the-art results for MCU-based tiny image classification.
Pushing the ImageNet record on MCUs. We compared MCUNetV2 with existing state-of-the-ar solutions on ImageNet classification under two hardware settings: 256kB SRAM/IMD Flash and 512kB SRAM/2MB Flash. The former represents widely used Cortex-M4 microcontroller: the latter corresponds to a higher-end Cortex-M7. The goal is to achieve the highest ImageNet accuracy on coourre-constrained MCUs (Table 2). MC significantly improves the ImageNet accuracy of tiny deep learning on microcontrollerss. Under 256kB SRAM/IMB Flash, MCUNetV2 outperforms the state-of-the-as method [27] by 4.6% at 18% lower peak SRAM. Under 512BB SRAM/2MB Flash, MCUNetV2 V2 achieves a new record ImageNet accuracy of 71.8% on commercial microcontrollers, which is 3.3% compared to the best solution under the same quantization policy. Lower-bit bit (int 4) mixed-precision or mixed- precision quantization can further improve the accuracy (marked in gray in the table). We believe that we can further improve the accuracy MCUNetV2 with a better quantization policy, which we leave to future work.
Visual Wake Words under 32kB SRAM. Visual wake word (VWW) reflects the low-energy application of ML. MCUNetV2 allows us to run a VWW model with a modest memory require- ment. As in Figure 7, MCUNetV2 outperforms state-of-thee method [27] for both accuracy vs.
7

---

Table 3. MCUNetV2 significantly improves Pascal VOC [13] object detection MCU by allowing higher
input resolution. Under STM32H743 MCU constraints, MCUNeIV2-H7 improves the mAP by 16.9% compared to [27], achieving a record performance on MCU. It can also scale down to cheaper MCU STM32F412 with only 256kB SRAM while still improving mAP by 13.2% at 1.9x smaller peak SRAM and a similar computation.
MCU Model
Constraint
Model
#Param
MACs
peak SRAM
VOC mAP
Gain
MbV2+ #bV2+CMSIS CMSIS [27]
0.87M
34M
519kB 466kB
31.6%
SRAM
H743 (~-S7)
MCUNet (27]
1.20M
168M
51.4%
0%
<512kB
MCUNetV2-H7
0.67M
343M
438kB
68.3%
+ +16.9%
F412 (~-54
<256kB
MCUNe(V2-M4
0.47M
172M
247kB
64.6%
+ +13.2%
Table 4. MCUNetV2 outperforms existing methods for memory-efficiet face detection on WIDER FACE [48] dataset. Compared to RNNPool-Facce [40], MCUNetV2-L can achieve similar mAP at 3.4x smaller peak SRAM and 1.6x smaller computation. The model statistics are profiled on 640 x 480 RGB input images
following [40].
mAP t
mAP ($3 faces)
Method
MACS + Peak Memory | (fp32)
Easy
Medium
Hard
Easy
Medium
Hard
EXTD [49]
8.49G
18.8MB (9.9x)
0.90
0.88
0.82
0.93
0.93
0.91
LFFD [20]
9.25G
18.8MB (9.9x)
0.91
0.88
0.77
0.83
0.83
0.82
RNNPool-Face Facel--ae--CC-ee [40]
1.80G
6.44MB (3.4x)
0.92
0.89
0.70
0.95
0.94
0.92
MCUNetV2-I.
1.10G
1.89MB
(1.0x)
0.92
0.90
0.70
0.94
0.93
0.92
EagleEye [52]
0.08G
1.17MB
(188)
0.74
0.70
0.44
0.79
0.78
0.75
RNNPool- [40]
0.10G
1.17MB (1.8x)
0.77
0.75
0.53
0.81
0.79
0.77
MCUNetV2-S
0.11G
672kB (1.0x)
0.85
0.81
0.55
0.90
0.89
0.87
peak memory and accuracy latency trade-off. We perform neural architecture search under both per-layer and per-patch inference settings using the same search space and super network for abla- tion. Compared to per-layer inference, MCUNetV2 can achieve better accuracy using 4.0x smaller memory. Actually, it can achieve >90% accuracy under 32kB SRAM requirement, allowing us to deploy the model on low-end MCUs like STM32F410 costing only $1.6. For the latency-constrained setting, we jointly optimized the model architecture and inference scheduling, where a smaller patch number is used when possible. Per-patch inference also expands the search space, giving us more freedom to find models with better accuracy vs. latency trade-off.
4.3 MCUNetV2 for Tiny Object Detection
Object detection is sensitive to a smaller input resolution (Figure 2). Current state-of-the-art [27] cannot achieve decent detection performance on MCUs due the resolution bottleneck. MCUNetV2 breaks the memory bottleneck for detectors and improves the mAp by double digits.
MCU-based detection Pascal VOC. We show the object detection results on Pascal VOC trained with YOLOv3 [38] on Table 3. We provide MCUNetV2 results for M4 MCU with 256kB SRAM and H7 MCU with 512kB SRAM. On H7 MCU, MCUNetV2-H7 improves the mAP by 16.7% compared to the state-of-the-an method MCUNet [27]. It can also scale down to fit a cheaper commodity Cortex-Ma. M4 MCU with only 256kB SRAM, while still improving the mAp by 13.2% at 1.9x smaller peak SRAM. Note that MCUNetV2-M4 shares a similar computation with MCUNet (172M 168M) but much better mAP. This is because the expanded search space from patch-based inference allows us to choose a better configuration of larger input resolution and smaller models.
Memory-efficient face detection. We benchmarked MCUNetV2 for memory-efficiee face detec- tion on WIDER FACE [48] dataset in Table We report the analytic memory usage of the detector backbone in f2 following [40]. We train our methods with S3FD face detector [50] following [40] for fair comparison. We also report mAP on samples with < faces, which a more realistic setting for tiny devices. MCUNetV2 outperforms existing solutions under different scales. MCUNetV2-L. achieves comparable performance 3.4x smaller peak memory compared to RNNPool- [27] and 9.9x smaller peak memory compared to LFFD [20]. The computation is also 1.6x and 8.4x
8

---

peak memory
patch size
MbV2: 72.2%
MBV2-RD: 72.1%
4F1400 14000
with p=4
with 5
140
computtiion overhead
75%
with 1=
1200
116
peak memory
840
8x smaller
92
pitch
50%
smalle
560
25%
280
44
20
0%
2
14
28
14
28
# patch-based block n
patches p (a) Analysis on peak memory & patch size
patches p
(b) Analysis on computation overhead
Figure Ablation study patch-based inference. Left: the peak memory generally goes down with more blocks being executed patch-by by-patch (n) and larger patch number (p). The optimal index MobileNetV2 n' = 5. where the feature map is down-sampled by 8x. Right: splitting the input images into more patches (larger p) leads to larger computation overhead. Receptive field redistribution reduces the overhead (MbV2-RD)
Table 5. Comparing MCUNetV2 with other memory-saving methods. Non-overlapping patches suffer from a de- graded detection performance; RNNPool [40] leads to worse performance and slower training time, MCUNetV2 with redistributed model maintains the accuracy at the same training cost. Degraded items marked in red.
Model
Inference
Invariant
Peak Memeg32
Train time
ImgNet Top-1
VOC mAP
MbV2 [41]
Per-layer
2.29MB
1.0x
72.2%
75.4%
Non-overlap
Per-patch
X
0.19MB
1.0x
71.8%
73.9%
MbV2-RMM -NVoollloool [40]
Streaming
X
0.24MB
3,2x
70.1%
71.0%
MbV2-RD (ours)
Per-patch
0.19MB
1.0x
72.1%
75.7%
smaller. MCUNetV2-S consistently outperforms RNNPool- [40] and EagleEye [52] at 1.8x smaller peak memory.
4.4 Analysis
Hyper-parameters for patch- based inference. We study the based inference: the number of blocks to be executed under patch-based inference nı; the number of patches to split the input image p (splitting the image into p x p overlapping patches). We analyze MobileNet [41] in Figure 8(a), with larger n, the patch size increases due the growing receptive field. The peak memory first goes down since the output feature map is smaller then goes up due to larger receptive field overhead. which helps to reduce the peak memory. However, it also leads to more computation overhead due to more spatial overlapping (Figure 8(b)). Receptive field redistribution can reduce the overhead significantly (MbV2-RD). The optimal design is n²" = 5, = 4 to reach the minimum peak memory with the smallest overhead. The choice of p and varies for different networks. Therefore, we use an automated method to jointly optimize with neural architecture (Section 3.3).
used for patch-
5 is optimal. For larger p (given n=5), each patch is smaller,
Comparison to other solutions. We also compare MCUNetV2 with other methods that reduce inference peak memory. The comparison on MobileNetV2 [41] is shown in Table 5. A straightforward way is to split the input image into non-overlapping patches ("Non-overlap") for the first several blocks as done in [12]. Such a practice does not incur extra computation, but it breaks the feature propagation between patches and the translational invariance of CNNs. It achieves lower image classification accuracy and significantly degraded object detection mAP (on Pascal VOC) due to the lack of cross cross-patch communication (a similar phenomenon is observed in [30]). For MobileNetV2 with RNNPool [40], it can reduce the peak memory but leads to inferior ImageNet accuracy and object detection mAP. Its training time is also 3.2x longer" due the complicated data path and the RNN module. On the other hand, MCUNetV2 acts exactly the same as a normal network during training (per-layer forward/backward)) while also matching the image classification and objection detection performance. MCUNetV2 can further improve the results with joint neural architecture and inference scheduling search (Section 4.2).
Dissecting MCUNetV2 architecture. We visualize one of the MCUNetV2 model architecture on the VWW [10] dataset in Figure 9. We can find the following patterns:
The kernel size in the per-patch inference stage is small (1 and 3 3) reduce the receptive field and spatial overlapping, thus reducing computation overhead.
measuree with official PyTorch code (MIT License) using batch size of 64 on NVIDIA Titan RTX.

---

per-patch
per-layer
MR4 3x3
5x5
ME1 7x77
MBH 7x7
MBH 7x7
MB6 3x.3
MB6 5u5
MB3 3x3
Miss Sx.5
cis lacad
10x1660
80x80
40x40
10x10
5x5
Figure 9. An MCUNetV2 architecture VWW. The color represents the kernel size; the height of each block represents the expansion ratio. The name MB (expansion ratio) :eenell size ) (kernel s sizc). Blocks with dashed borders have stride=2. ()x() in the bottom denotes the feature map resolution.
The expansion ratio of the middle stage (early in per-layer stage) is small to further reduce the peak memory; while the expansion ratio for the later stage large increase performance. Large expansion ratios and large kernel sizes usually do not appear together to reduce the computational cost and latency: if the expansion ratio is large (like 6), the kernel size is small (3 x 3 or x 5) if the kernel size is large (7 7), the expansion ratio is small (3 4), datasets like VWW compared to MCUNet [27], since per-layer inference cannot fit a large input resolution.
The input resolution is larger on eesolution-sensieee
Notice that all the patterns are automatically discovered by the joint neural architecture and inference scheduling search algorithm, without human expertise.
Related Work
Tiny deep learning on microcontrollers. Deploying deep learning models on memory-constrained microcontrollers requires an efficient inference framework and model architecture. Existing de- ployment frameworks include TensorFlow Lite Micro [1]. CMSIS-NN [23], TinyEngine [27], Mi- croTVM [8]. CMix-NN [7], etc. However, all of the above frameworks support only per-layer inference, which limits the model capacity executable under a small memory and makes higher resolution input impossible.
Efficient neural network. For efficient deep learning, people apply pruning [16, 18, 28, 19, 32, 31] and quantization [16, 54, 53, 37, 46, 39, 24] to compress an off-the-shelf deep network, or directly design an efficient network architecture [22, 41, 21, 34, 51]. Neural architecture search (NAS) [55, 56, 29, 6, 44, 47] can design efficient models in an automated way. It has been used to improve tinyML on MCUs [27, 4, 25, 14, 33J. However, most of the NAS methods use the conventional hierarchy CNN backbone design, which leads to an imbalanced memory distribution under per-layer inference (Section 2), restricting the input resolution. Therefore, they are not able to achieve good performance on tasks like object detection without our patch- basch inference scheduling.
Computation schedulingrr-ooddernngg. The memory requirement to run a deep neural network related to the implementation. It possible to reduce the required memory by optimizing the convolution loop-nest [43], reordering the operator executions [26, 2], or temporarily swapping data off SRAM [35]. Computing partial spatial regions across multiple layers can reduce the peak memory [15, 3, 40]. However, system-onf, moly optimization leads to either large repeated computation or highly complicated dataflow. Our work explores joint system and model optimization to reduce the peak memory at a negligible computation overhead while still allowing conventional convolution optimization techniques like im2col, tiling, etc.
6 Conclusion
In this paper, propose patch-based based inference to reduce the memory usage for tiny deep learning by up to 8x, which greatly expands the design space and unlocks powerful vision applications on IoT devices. We jointly optimize the neural architecture and inference scheduling to develop MCUNetV2. MCUNetV2 significantly improves the object detection performance on microcontrollers by 16.9% and achieves record ImageNet accuracy (71 (71.8%). For the VWW dataset, MCUNetV2 can achieve >90% accuracy under only 32kB SRAM, 4x smaller than existing work. Our study largely addresses the memory bottleneck in ML and paves the way for vision applications beyond classification.
10

---

Acknowledgments
We thank MIT-IBM Watson AI Lab, Samsung, Woodside Energy, and NSF CAREER Award #1943349 for supporting this research.
References
[1] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michacl Isard, al. Tensorflow: A system for large-scale machine learning. In OSDI, 2016.
[2] Byung Hoon Ahn, Jinwon Lee, Jamie Menjay Lin, Hsin-Pai Cheng, Jilei Hou, and Hadi Esmaeilzadeh. Ordering chaos: Memory-aware scheduling of irregularly wired neural networks for edge devices. arXiv preprint arXivx2003,00000 023669 2020.
[3] Manoj Alwani, Han Chen, Michael Ferdman, and Peter Milder. Fused-layer can accelerators. In 2016 49th Annual IEEE/ACM International Symposiun on Microarchitecture (MICRO), pages 1-12. IEEE, 2016.
[4] Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul Whatmough. Micronets: Neural network architectures for deploying tinyml applications commodity microcontrollers. Proceedings of Machine Learning and Systems, 2021.
[5] Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once for All: Train One Network and Specialize it for Efficient Deployment. In ICLR, 2020.
[6] Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct Neural Architecture Scarch on Target Task and Hardware. In ICLR, 2019.
[7] Alessandro Capotondi, Manuele Rusci, Marco Fariselli. and Luca Benini. Cmix-nn: Mixed low-precision can library for constrainee edge devices. IEEE Transactions on Circuits and Systems II: Express Briefs, 67(5):81--875; 2020.
[8] Tianqi Chen, Thierry Morcau, Ziheng Jiang, Lianmin Zheng, Eddic Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, al. (TVM}: An automated end-to-end optimizing compiler for deep learning. In OSDI, 2018
[9] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arxiv: 805, 06085, 2018.
[10] Aakanksha Chowdhery, Pete Warden, Jonathon Shlens, Andrew Howard, and Rocky Rhodes. Visual wake words dataset. arXiv preprint arXiv: 05721, 2019.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR, 2009.
[12] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010. 1929, 2020.
[13] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[14] Igor Fedorov, Ryan P Adams, Matthew Mattina, and Paul Whatmough. Sparse: Sparse architecture search for cnns constrae-onsttrainee microcontrollers. In NewrIPS, 2019.
[15] Koen Goetschalekx and Marian Verhelst. Breaking high resolutsol eun bandwidth barriers with euhanced depth-first execution. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(2):323- 2019.
[16] Song Han, Huizi Mao, and William Dally. Deep Compression: Compressing Deep Neural Networks with Pruning. Trained Quantization and Huffman Coding. In ICLR, 2016.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[18] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoMI. for Model Compression and Acceleration on Mobile Devices. In ECCV, 2018.
[19] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
[20] Yonghao He, Dezhong Xu, Lifang Wu, Meng Jian, Shiming Xiang, and Chunhong Pan. Lifd: A light and fast face detector for edge devices. arXiv preprint 191633904.1063,, 2019.
11

---

[21] Andrew Howard, Mark Sandler, Grace Chu, Liang- Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for MobileNetV3. In ICCV, 2019.
[22] Andrew G. Howard, Menglong Zhu, Bo Chen, Dimitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, arXiv, 2017.
[23] Liangzhen Lai, Naveen Suda, and Vikas Chandra. Cmsis-nn: Efficient neural network kernels for arm cortex-m m cpus. arXiv preprint arXiv: 06600,, 2018.
[24] Hamed Langroudi. Vedant Karia, Tej Pandit, and Dhireesha Kudithipudi. Tent: Efficient quantization of neural networks on the tiny edge with tapered fixed point. arXiv preprint arXiv:2 2021.
[25] Edgar Liberis, Eukasz Dudziak, and Nicholas D Lanc. μnas: Constrained neural architecture search for microcontrollers. arXiv preprint arXiv:2010. 14246, 2020.
[26] Edgar Liberis and Nicholas D Lane. Neural networks on microcontrollers: saving memory at inference via operator reordering. arXiv preprint vrXiv:1910.055 10, 2019.
[27] Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, and Song Han. Mcunet: Tiny deep learning on iot devices. In NeurlPS, 2020.
[28] Ji Lin, Yongming Rao, Jiwen La, and Jie Zhou. Runtime neural pruning. In NeurlPS, 2017.
[29] Haoxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In ICLR, 2019.
[30] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 2021.
[31] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning. /CCV, 2019.
[32] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. ICCV, 2017.
[33] Bo Lyu, Hang Yuan, Longfei Lu, and Yunye Zhang. Resource-constrained neural architecture search on edge devices. IEEE Transactions on Network Science and Engineering, 2021.
[34] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. ECCV, 2018.
[35] Hongyu Miao and Felix Xiaozhu Lin. Enabling large neural networks tiny microcontrolleers with swapping. arXiv preprint arXiv:2101.0877 2021.
[36] Prajit Ramachandran, Barret Zoph, and Quoc Le. Searching for activation functions. arXiv preprint arXiv:1710005441 2017.
[37] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifica- tion using binary convolutional neural networks. ECCV, 2016.
[38] Joseph Redmon and Ali Farhadi. YOLOv3: An Incremental Improvement. arXiv, 2018
[39] Manuele Rusci, Alessandro Capotondi, and Luca Benini. Memory-driven mixed low precision quantization for enabling deep network inference on microcontrollers. In MLSys, 2020.
[40] Oindrila Saha, Aditya Kusupati, Harsha Vardhan Simhadri, Manik Varma, and Prateek Jain, Rnnpool: Efficient ninelr pooling for ram constrained inference. arXiv preprint 191)) 2020.
[41] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In CVPR, 2018.
[42] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 143i::1409.110 1556, 2014.
[43] Arthur Stoutchinin, Francesco Conti, and Luca Benini. Optimally scheduling enn convolutions for efficient memory access. arXiv preprint arXiv: /902.01492 2019.
[44] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Lc. MnasNet: Platform- Aware Neural Architecture Search for Mobile. In CVPR, 2019.
[45] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pages 6105-6114, PMLR, 2019.
[46] Kuan Wang. Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware- Aware Automated Quantization with Mixed Precision. In CVPR, 2019.
12

---

[47] Bichen Wu. Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian. Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. CVPR, 2019.
[48] Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Wider face: A face detection benchmark. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[49]
YoungJoon Yoo, Dongyoon Han, and Sangdoo Yun. Extd: Extremely tiny face detector via iterative filter reuse. arXiv preprint arXiv: / 196799657 2019.
[50] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Li. S3td: Single shot scale-invariant face detector. In Proceedings of the IEEE international conference on computer vision, pages 192-201, 2017.
[51] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An Extremely Efficient Convolu- tional Neural Network Mobile Devices. In CVPR, 2018.
[52] Xu Zhao, Xiaoqing Liang, Chaoyang Zhao, Ming Tang. and Jinqiao Wang. Real-time multi-scale face detector embedded devices. Sensors, 19(9)21158 2019.
[53] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 16666666// 60, 2016.
[54] Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv preprint arXiv:1612.0000 2016.
[55] Barret Zoph and Quoe Le. Neural Architecture Search with Reinforcement Learning. ICLR, 2017.
[56] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning Transferable Architectures for Scalable Image Recognition. In CVPR, 2018.
13


Reviews:
[{"review_id": "xvnF7pzhVSZ", "reviewer": "Reviewer_rwM3", "summary": "Working memory utilization for most CNN based models typically is more than what can be afforded on Tiny devices. Thus, it is often quite difficult to extract reasonable performance for visual tasks on tiny devices.\n\nAuthors attack this problem by a) introducing a patch based inference method that improves peak memory utilization at the cost of compute and b) introducing a redistribution process that reduces the overall compute while maintaining/improving performance. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Working memory utilization for most CNN based models typically is more than what can be afforded on Tiny devices. Thus, it is often quite difficult to extract reasonable performance for visual tasks on tiny devices. The high memory utilization is a result of storing activation between layers in CNN inference. In a typical layer by layer inference implementation, one has to store the intermittent activation in memory while the next set of activation are computed. This means that for large layers or layers with a lot of filters, there is a high working memory requirement. \n\nTo attack this problem, authors propose a patch based inference method -- essentially deviating from the layer-by-layer evaluation to a more 'patch-by-patch' evaluation method. This approach critically depends on the fact that later layers in CNN architectures are typically smaller in their activation sizes, and one can compute each entry of these smaller activations while by streaming in its respective receptive field --- we only need to store the smaller activation of this later layer in memory. \n\nOf course such a patch based inference often comes with the cost of extra computation. To counteract this, the authors propose modifying the network architecture to be computation heavy on the later layers. Authors propose a NAS approach to guide this modification (and show certain advantages of this approach like a larger design space). Authors evaluate their method experimentally and validate their claims on tiny-ml benchmarks. The experimental results on existing datasets look very promising, with good/better performance metrics than existing models at marginally more computational costs/latency while hitting SRAM requirements. Similar results are also reported for Tiny image classification with PatchNN offering a better SRAM-accuracy and latency-accuracy tradeoffs. \n\nOverall  I feel that the work attacks an important problem and presents a very viable, conceptually simple solution with good empirical performance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "TE7-iZj_Ei6", "reviewer": "Reviewer_VGan", "summary": "This paper addresses the problem of large activations that quickly arises in devices with little memory, such as MCUs, when deploying larger networks. The main contribution proposes a relatively simple but effective approach to reducing the memory footprint of activations (by a large 4x-8x factor) while introducing some unavoidable extra compute (resulting in up to 15% MACs, which translate into  up to ~1.27x higher latency). The impact in terms of compute overhead of the proposed per-patch processing is alleviated by redistributing the receptive filed. Which means: reducing the filter WxH in early layers and use striding>1; then stack more blocks in deeper parts of the network where the memory peak of activations is not so high anymore. A NAS-based mechanism is introduced to jointly discover architectures and receptive filed redistribution strategies.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nLarge activations is arguably the main problem that’s preventing the deployment of more sophisticated ML-based applications to very constrained devices such as MCUs. This challenge is even worse in other domains such as graphs. This paper is very well presented and touches upon a known problem that has mainly been addressed from a neural architectural point of view but often only indirectly. The results are very compelling, specially in the case of object detection which, as the authors state, it is more sensible to reduced input or feature map resolutions. The list of references is complete, including recent works.\n\nTo revise/improve/expand:\n- From Figure 3 and the first intuitive explanation given in the introduction and later in 3.1 in a bit more detail, I’m still not 100% sure how this is really implemented in practice. I would advice adding an \\algorithm{} to describe how this is done.\n- Could the authors elaborate on why/how PatchNN is 1.8x faster than MCUNet for the same ~92% on VWW (Figure6.b) , specially given the unavoidable extra compute needed for per-batch executions. Could the authors share details on the PatchNN architecture?\n- Could the authors elaborate on what sorts of optimisations (e.g. im2row/col, SIMD etc) will be hard to make use of when relying on the proposed patch-based inference ?\n- Minor: Line 51 says “Figure 3” but I believe the authors referred instead to “Figure 1”.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of the patch-based inference is well covered and addressed with the receptive field re-distribution analysis (first with MobileNetV2) and, later introduced in the NAS stage. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "G80NqqJcx_b", "reviewer": "Reviewer_8aP4", "summary": "The main problem the paper tries to tackle is reducing the peak memory footprint of the neural networks by deviating from the usual layer-by-layer execution. The main intuition seems to be very similar to tiling combined with layer fusion-like ideas. However, the main spark of the paper seems to be leveraging this dimension in the context of Neural Architecture Search, which seems to be novel.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe the paper tackles a very important topic or reducing peak memory footprint of neural network execution. Especially, considering that many modern frameworks rely only on layer-by-layer computation based on the fact that it is mainly designed for less memory-constrained scenarios, what the paper proposes as patch-based is a good direction for MCUs.\n\nI believe the main insight in the paper that should be highlighted is leveraging this as a new dimension for NAS. Overall, it seems that the patch-based nature allows smaller footprint and NAS acts as a good lubricant to leverage this aspect more naturally.\n\nSingle request that I have is the visualization of the output network architecture. I believe this would help the readers gain better insight about the work.\n\nAs a side note, I think the paper is missing a citation regarding reordering the operator execution to enable execution on MCUs have been extensively studies in [1.\n\n[1] Ahn, B. H., Lee, J., Lin, J. M., Cheng, H. P., Hou, J., & Esmaeilzadeh, H. (2020). Ordering chaos: Memory-aware scheduling of irregularly wired neural networks for edge devices. arXiv preprint arXiv:2003.02369.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1-2"}, "extra_text": {}}, {"review_id": "7N-qsOAxj6x", "reviewer": "Reviewer_bZRz", "summary": "Instead of the conventional layer-by-layer execution, this paper proposes a patch-by-patch scheduler to reduce the peak memory required by a CNN for inference on MCU.\nThe author also proposes a network redistribution method, which finetunes the network architecture (CONV kernel size), to mitigate the computation overhead caused by their patch-by-patch execution. The results show that, with a certain computation overhead, the proposed methods can effectively reduce the inference peak memory while not losing accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The MCU devices has drawn great attention recently, and this paper targets memory-efficient inference on MCU devices. The proposed method can effectively reduce the peak memory-bound during the NN inference. Since the peak memory is reduced, it allows the MCU device to accommodate larger NN or using larger input size to improve the accuracy. This provides the proposed method a clear advantage when compared to the baseline works. This paper is well-written and easy to follow.\n\nHowever, the search method used in this paper also follows the previous works with some additional search space.\n\nThe author shows the classification results on large dataset (e.g., ImageNet), which is great. But for the object detection results, the author shows the results on VOC dataset, which is considered an easy dataset. The author proposes the network redistribution that will change the receptive field of the well-designed network structure. It is unknown whether it may degrade the performance on larger datasets e.g., MS-COCO. Because the object detection tasks may be more sensitive to the receptive field changes.\nIt seems that the patch-by-patch and network redistribution techniques quite rely on the proposed joint NAS to reduce the computation overhead and maintain accuracy. So, it would be good to know that whether the proposed joint NAS is still valid and affordable on larger datasets for object detection tasks such as MS-COCO.\n\nIn the introduction section, the author claims that the pruning, quantization, and NAS methods ``focus on reducing the number of parameters and FLOPs, which cannot address the memory bottleneck’’. I think this is not correct since those model compression techniques can reduce the memory cost for both model weights and activations in an order of magnitude manner.\n\nFor branches network structure, the author mentioned that the memory requirement is calculated by summing all the memory required by each branch together. But it may overestimate the peak memory, because same branches may be able to share the input or output activations.\n\nMinor issues:\nA typo in line 65 “On ImageNet”\nSince the inference process only uses 1 picture at a time, it would be good to know the percentage of memory cost by the weights.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors tackle an important and salient problem: inference on microcontrollers for DL, where memory is very constrained. The proposed scheme, patch by patch inference, works well, the reviewers were happy with the paper and the method itself, therefore I recommend acceptance.
Decision: Accept (Poster)