MCUFormer: Deploying Vision Tranformers on Microcontrollers with Limited Memory
Authors: Yinan Liang
Ziwei Wang
Xiuwei Xu
Yansong Tang
Jie Zhou
Jiwen Lu
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision transformer
microcontroller
network architecture search
TL;DR: MCU, transformer, deploy

Abstract: Due to the high price and heavy energy consumption of GPUs, deploying deep models on IoT devices such as microcontrollers makes significant contributions for ecological AI. Conventional methods successfully enable convolutional neural network inference of high resolution images on microcontrollers, while the framework for vision transformers that achieve the state-of-the-art performance in many vision applications still remains unexplored. In this paper, we propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory, where we jointly design transformer architecture and construct the inference operator library to fit the memory resource constraint. More specifically, we generalize the one-shot network architecture search (NAS) to discover the optimal architecture with highest task performance given the memory budget from the microcontrollers, where we enlarge the existing search space of vision transformers by considering the low-rank decomposition dimensions and patch resolution for memory reduction. For the construction of the inference operator library of vision transformers, we schedule the memory buffer during inference through operator integration, patch embedding decomposition, and token overwriting, allowing the memory buffer to be fully utilized to adapt to the forward pass of the vision transformer. Experimental results demonstrate that our MCUFormer achieves 73.62\% top-1 accuracy on ImageNet for image classification with 320KB memory on STM32F746 microcontroller. Code is available at https://github.com/liangyn22/MCUFormer.

MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory
Yinan Liang' Ziwei Wang²
Jiwen Lu (' (Shenzhen International Graduate School," DDeparmeent of Automation), Tsinghua University
Xiuwei Xu²
Yansong Tang Jie Zhou?
Abstract
Due to the high price and heavy energy consumption of GPUs, deploying deep models on IoT devices such as microcontrollers makes significant contributions for ecological AI. Conventional methods successfully enable convolutional neu- ral network inference of high resolution images on microcontrollers, while the framework for vision transformers that achieve the state-of-the-art performance in many vision applications still remains unexplored. In this paper, we propose a hardware-algorithn algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory, where we jointly design transformer architecture and construct the inference operator library to fit the memory resource constraint. More specifically, we generalize the one-shot network architecture search (NAS) to discover the optimal architecture with highest task performance given the memory budget from the microcontrollers, where we enlarge the existing search space of vision transformers by considering the low-rank decomposition dimensions and patch resolution for memory reduction. For the construction of the inference operator library of vision transformers, we schedule the memory buffer during inference through operator integration, patch embedding decomposition, and token overwriting, allowing the memory buffer to be fully utilized to adapt to the forward pass of the vision transformer. Experimental results demonstrate that MCUFormer achieves 73.62% top-1 accuracy ImageNet for image classification with 320KB memory on STM32F746 microcontroller. Code is available at http://gghhb..oom//ingggmMCuuuu
Introduction
Deep neural network deployment for realistic applications usually requires hlgh-performance com- puting devices such as GPUs and TPUs 16 12 Due to the high price and energy consumption of these devices, the unacceptable deployment expenses strictly restrict the utilization of deep models in widely variety of tasks 17, 39), especially in the scenarios without sufficient battery support. Deploying deep neural networks cheap IoT devices with low energy consumption becomes
practical solution in many industrial applications such as maritime detection [34] 24], agricultural
picking 31 43 and robot navigation 22 [29].
widely studied to deploy deep convolutional networks with high storage and computational complexity to microcontroller units (MCU) with extremely limited memory (SRAM) and storage (flash). By searching the optimal architectures and arranging the memory buffer according to the overall network topology, deep convolutional neural networks with large numbers of parameters and high FL.PPs are successfully deployed on MCUs with comparable accuracy inferencing on high- -eggppeeffomccce computing devices. Vision transformers [22 33] have surpassed convolutional neural networks a wide variety tasks, which can further push the limit of intelligent visual perception in resource-constrainee devices.
Recently, the hardware-alpe allgorith co-design framework 20
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, the memory footprint of vision transformers significantly exceeds the budget of MCUs due the matrix multiplication with large sizes, and the lack of operator library for transformer inference on MCUs also obscure the practical deployment.
In this paper, we present hardware-algorithe co-optimizations framework to deploy vision trans- formers on MCUs with extremely limited memory budget. We jointly design transformer architectures that achieve the highest task performance within the memory resource constraint and construct an operator library for transformer inference with efficient memory scheduling, More specifically, we leverage the one-shot network architecture search method discover the optimal architecture, where we enlarge the search space by considering low-rank decomposition ratios and token numbers that significantly influence the peak memory during the forward pass. To efficiently acquire effective supernets in the huge search space, we learn the supernet in the fixed search space consisting of given low-rank decomposition ratios and token numbers, and evolve the search space of supernet optimization by predicting the correlation between the task performance and the search space. To construct the operator library within the memory budget, we approximate full-precision operators such as softmax and GeLU layers with integer arithmetics. Moreover, we decompose the patch embedding layers with large convolution kernels to multiple patchification steps with small receptive fields, and overwrite the input token features in the matrix multiplication during the inference for further memory reduction. Extensive experimental results show our method can achieve 73.62% top-1 accuracy the large-scale image classification dataset ImageNet [T]] with the 320KB SRAM STM32F746 microcontrollers, and significantly outperforms the CNN based state-of-the of-tt-----hh-rrr art networks on MCUs by 5.4%. Our contributions can be summarized as follows:
To the best of our knowledge, we propose the first hardware-algorithn co-optimizations framework that successfully deploys vision transformers on MCUs with the competitive performance in challenging computer vision tasks
We present the search space evolution framework for effective architecture search of vision trans- formers within the memory limits, and construct a operator library with efficient memory scheduling to enable practical deployment.
We conduct extensive empirical studies to show the feasibility of deploying vision transformers on MCUs with extremely limited memory.
2 Related Work
Efficient vision transformers: Vision transformers mine the global dependencies via self-attention that outperform convolutional neural networks on a wide variety of tasks such as image classification
22 37
object detection [5] [4 and instance segmentation 355 36). To reduce the computational
and storage complexity, efficient architecture design, sparse attention and parameter quantization are presented for efficiency enhancement. For efficient architecture design, Chen er al. [6] shared the self-attention in consecutive layers to avoid redundant calculation in dependency discovering, and Fan et al. [14] fused multiscale intermediate features to reduce the resolution of top layers. For sparse attention, Rao et al. (26) dynamically pruned uninformative tokens during inference according to the computation budget and sample features. For parameter quantization, Liu et al. 23 extended the post-training quantization framework by mimicking the self-attention rank in the full full-precision model. However, existing methods usually focus on reducing the computational complexity measured by FLOPs and the storage cost evaluated by the number of parameters, and ignore the memory limit in inference that is the main resource bottleneck of MCUs.
Network architecture search aims to discover the topology including operators and connections to achieve the optimal trade-off between the accuracy and complexity. which can be divided into two categories according to the search algorithms. -on-diiferentiaale methods 33] 7 44 utilize reinforcement learning evolutionary algorithms update the selection regarding the defined reward or fitness, where the candidates can be updated from scratch or from a construct large supernet that contains all possible choices represented by different parallel branches. By optimizing the importance weights of different choices, the operators and connections with the highest importance weight are selected to form the final architecture. In this paper, we focus on learning the supernet in the optimal search space defined by the low-rank decomposition ratio and token numbers, which influence the inference memory significantly.
Network architecture search:
pre pre-trained supernet. Differentiable methods [2] 4 42
2

---

Efficient Operator Design
Search Space Selection
Search Space (xg
y wx Pa
New Search
Evolutin
x
Original Search
Search Space 3 (rs. fy)
Space
Evolving
IUK
RM
Low-rank Weight Decomposition
Law-rank Ratio/Token Number
Search Space (r', Ez)
Tendency Regression
Patch Embedding Layer
Choices
Enibed
Search Space (r (g)
Omm
MLP
Rato
XG Tokenization
TokeyTokens
Search Space Evolution
Hypemetwork Training
Figure I: The pipeline of the network architecture search with memory constraint. We search the low- rank decomposition ratio and the token numbers for search space evolution, where the evolutionary process is decided according to regressed tendency between the defined score and the search space factors. With the optimal search space, the architecture with the highest performance within the memory limit is acquired from the trained supernet.
Deep learning on microcontrollers: Due to the low price and energy consumption, employing microcontrollers for deep learning inference and training has been comprehensively researched in recent years. Existing frameworks contain TensorFlow Lite Micro [. CMSIS-NN [18], CMix-NN 4), MicroTTVM [10) and TinyEngine [20]. Nevertheless, these compiling libraries designed for convolutional neural networks and lack operators of vision transformers such as LayerNorm and GeLU. Meanwhile, the layer-wise optimization of memory footprint cannot fully utilize the SRAM budget of the microcontrollers and fails to deploy vision transformers with high task performance. For the memory-efficient network architecture design, Lin al. reduced the image resolution and the width multiplier for peak memory reduction. Sun et al. leveraged the mixed -exeed-rrecision quantization that assigned the optimal bitwidth for each layer achieve the optimal accuracy-memort trade-off, and Saha et al. [27 decreased the resolution of the activation after the proposed RNNPool layer. However, the memory reduction techniques are designed for convolutional neural networks, which are not feasible for vision transformers with different operators.
3 Approach
In this section, we detail the hardware-alee aalloare-algorihhm co-optimizations framework to deploy vision transformers on microcontrollers. We first introduce the network architecture search of vision transformers with limited memory constraint, and demonstrate the operator library construction for vision transformer inference on microcontrollers.
3.1 Network Architecture Search with Memory Constraint
Conventional architecture search methods usually consider the computational complexity (FLOPs) and the storage cost (the number of parameters), while the peak memory obscures the deployment of vision transformers on microcontrollers due to the extremely limited SRAM. We extend the one-shot NAS to search the optimal architecture, where we first learn some of supernets randomly which containing all possible topology choices and employ evolutionary algorithms to seek for the optimal supernet given the resource budget. Figure I demonstrates the pipeline of the network architecture search with memory constraint. To fully leverage the achievements of vision transformer NAS methods, we build the search space of sub-networks on the basis of AutoFormer [8] which searches embedding dimension, QKV dimension, MLP ratio, head number and depth number with trained hypernetworks. The search space of supernet is defined by the the low-rank decomposition ratio and token numbers that can significantly influence memory footprint. The token number impacts the size of intermediate features during inference and can be modified by varying the receptive

---

field in the patch embedding layers. Different from the multipy--accumlla (MAC) operations of convolutional neural networks. the matrix multiplication in large sizes causes heavy memory burden. To address this, we generalize low-rank decomposition for the multi-layer perceptron (MIP) layers in the following form:
MLP(=)) = - V, b.
where z E R means the input features with n tokens and d elements of each token. U, Rdxr and V, E TRdxr are two low-rank matrices approximating the original weights in the MLP layer, and b means the bias term. Since the dimension of low-rank matrices affects the trade-off between memory footprint and task performance, we are required to find the optimal low-rank decomposition ratio in our search space selection.
Directly integrating the token numbers and the low-rank decomposition ratio into the original search space in AutoFormer causes convergence difficulties of supernets because of the large discrepancy among optimal architectures of different low-rank decomposition ratios. Therefore, we first optimize the search space of hypernetworks that is composed of the token numbers and the low-rank decompo- sition ratio, and then performs architecture search in the updated search space of hypernetworks. The above two steps are iteratively implemented until convergence or achieving the maximum search cost. Enumerating all search space for architecture search is not feasible due to the extremely high training cost for single supernet in the given search space. Inspired by the observation in [9] that there is strong correlation between the task errors and the continuous choices, we evolve the search space by considering the relation among the performance, memory footprint and the search space factors including token number and the low-rank decomposition ratio. The score function S to evaluate the optimality of different hypernetwork search space defined as follows:
S(A, R) = A n/,
(2)
where A is the accuracy of the hypernetwork in the search space and rj a hyperparameter. R the ratio of sub-networks whose memory footprint is within the SRAM limit. Mcanwhile, enumerating all sub-networks the supernet to acquire the average memory is prohibitive due to the extremely large numbers of candidates in the subspace, and we randomly sample N sub-networks to approximately evaluate the value of R for the given hypernetwork. The linear fitting functions are employed to estimate the tendency between the score and the factors of the search space as S = w x So with
the intercept Sg, where x = [x2] consists of low-rank decomposition ratio and token numbers. The search space is updated with the following rules:
11+1) 24
(1)
Wi h4
A..
i= 1,2,
where x. means the ich element in at the to step and w; represents the ies element in w. The optimization step A; is set to the interval of adjacent factor values in our method, and the update threshold h; are hyperparameters for the optimization. [x]] demonstrates the largest integer smaller than x..
Training the hypernetwork in the evolving search space until convergence requires unacceptable optimization cost, and we fit the correlation between task performance and the continuous choices of search space factors in the training process. For each evolved search space, we train the supernet for I epochs from the resume points or scratch if pre-trained model does not exist. To fairly discover the tendency between the task performance and the continuous choices, we fit the correlation weight matrix w with the data points in the same total training epochs. Since the linear fitting function only holds in a small local region for given search space, we utilize the piecewise linear fitting function that only considers top-k nearest neighbors for the search space factor æ. The data point sampling strategy for tendency fitting can be represented as follows:
((x)= = {z) z e E())
where (xx) means the set of data points consisting of task accuracy with memory and continuous choices of low-rank decomposition ratio and token numbers for fitting the linear function in the evolving search space æ. The neighborhood for x with nearest neighbors is denoted as No (w), and the total training epochs for the supernet in the search space æ are represented by Exx). When the neighborhood of the given search space æ is empty, we construct the neighborhood by considering the top-k nearest neighbors whose total training epochs are E(x) t. We train hypernetorrss in

---

Large Kernel
Temporary Buffer
MAC
Token Features
Weights
Convolution
Multiple Small
Kand
Satret4
Accumulation
Temporary Features
Output
Token Features
Output Features
(a) Patch Embedding Decomposition
(b) Token Overwriting
Figure 2: (a) The comparison between the conventional patch embedding operation and our decom- position. We decompose the convolution operator with a large receptive field into multiple operators with small receptive fields and utilize temporary buffer to record the intermediate accumulations. (b) Demonstration of token overwriting. The buffer for input features is re-used by the output features, and the peak memory is reduced significantly.
these search space for epochs for accuracy update to fairly estimate the correlation matrix w for the given search space æ.
The selected supernet in the optimal search space used for subsequent training until full convergence and is employed for the network architecture search of vision transformers. In this process, the evolutionary algorithm is applied with the aim of maximizing accuracy while adhering to the memory constraint.
3.2 Operator Library Construction for Vision Transformers
Deploying the vision transformer microcontrollers for inference requires the operator library transform the model into executable functions. Existing operator libraries for deep learning are comparable with convolutional neural networks, while fail to implement operations in vision transformers such the GeLU activation function and the layer normalization. Meanwhile, the inference efficiency of the library cannot fully utilize the memory budget of microcontrollers to achieve higher task performance with larger vision transformer models. Despite of the efficient memory scheduling techniques that are general to all network architectures in [20], we present the following memory utilization enhancement techniques to enable deployment vision transformers on microcontrollers.
Patch embedding decomposition: Images are patchified by the patch embedding layer in vision transformers for inference. The imput samples are transformed into patches by convolution operations, where both the kernel size and stride are both equal the patch size. Since the kernel size in the patch embedding layer (larger than 16 x 16) is much larger than common convolutional neural networks, most memory cost of the patch embedding layer results from the filter weights instead of the input and output activation. In order to effectively address this challenge, we decompose the patch embedding layer by transforming the convolution with large receptive field to multiple convolution operations with a small receptive field. To add the results from multiple convolution operations, we leverage small memory buffer to record intermediate value of accumulation, effectively reducing the peak memory usage during image patchification. Figure 2 demonstrates the comparison between the original patch embedding operation and that with our decomposition. As decomposing the patch embedding layer into excess convolutional operators with small receptive fields can obviously increase the inference latency due to the multiple forward passes, we select the receptive field of the decomposed patch embedding layer x 4 to achieve the satisfying trade-off' between the memory footprint and the inference latency.
Operator integration: Quantizing network parameters and substituting MAC operations with integer arithmetics have been proven beneficial in reducing the memory of convolutional neural networks without causing performance degradation. However, conventional operator library with quantized operators not feasible for vision transformer inference because of the specific operators including the GeLU activation and layer normalization. Since GeLU requires the cumulative distribution
5

---

function (CDF) of Gaussian distribution, we approximate the activation function with quantization by multiplication and softmax function which can both implemented with int8 arithmetics. Similarly, the square root operators in the layer normalization is not supported by int8 arithmetics in operator library. We construct surrogate equations with fixed-point iterative methods to calculate the output of the square root operators. Considering the inference latency of layer normalization operator, we only iterate the surrogate equation for times with balance between the inference latency and the prediction precision. We detail the pseudo algorithm of integer-only square root operation in the supplementary material.
The matrix multiplication in vision transformer inference that results in high memory consumption impose element interaction only within tokens, and the interaction among tokens is enabled in self- attention computation whose memory cost is far less than the SRAM limit. Therefore, we can overwrite the intermediate features during inference of fully-connected layers because the input activation of each token will no longer be used after acquiring the output features of the corresponding token. Figure Pp illustrates the memory scheduling of matrix multiplication in existing inference operator libraries and our token overwriting techniques, where our method writes the output of matrix multiplication for each token in the same memory space as the input activation. Since the peak memory usage occurs in the matrix multiplications, we assign the maximum memory size M for vision transformer inference based on the network architecture, according to the following criteria:
Token overwriting:
M = sup ht'y uuy + max(h//w h w,
(5)
where kly, hy and h², represent the height of weights, input features, output features in the ln fully- connected layers, and w²y, w/ and w/, demonstrate the width of the above tensors. Since the memory consumed by the input activation can be overwritten by the output features, we only assign the memory buffer with the same size as the larger one between input and output tensors.
4 Experiments
In this section, we first introduce our hardware configuration, dataset and the implementation details of our hardware-algorithe co-optimizations framework, and then conduct thorough performance analysis including the task performance and the memory consumption with respect to the presented techniques in network architecture search and operator library construction. Finally, compare the performance in accuracy and efficiency with existing network architecture search methods and operator libraries for deep model deployment on microcontrollers.
4.1 Setups
Hardware configuration: We deploy the vision transformers with our hardware-algorithn co- optimizations framework in different microcontrollers with various resource constraint including STM32F427 Nortex-M4425663 memory/IMB flash), STM32F746 (Cortex-M7/320KB mem- ory/IMB flash) and STM32H743 (C75tex-M7/512KB memory/2MB flash). In the performance analysis, we evaluate our framework on STM32F746 to acquire the accuracy and memory.
We conduct the experiments on ImageNet for image classification, which contains 1.2 million training images and 50k validation images from 1000 classes. All images are scaled and biased into the range [-1,1] for normalization. For the training process, we resize the images with the shorter side as 256 and randomly crop a 240 region. For inference, we utilize the central crop with the size of 240x240.
Dataset:
Implementation details: For the network architecture search of vision transformers, our choices for the search space consisting of low-rank decomposition ratio and the token size e can be selected from E [0.4 0.05: 0.95) and c {16, 20, 24, 28, 32). The supernet design and the evolutionary search in the learned hypernetwork keep the same with those in [8] to fully leverage the potential in architecture search with high degrees of freedom. During the search space evolution, we only select 5 nearest neighbors with the same total training epochs to fit the piecewise linear function for acquiring the slope. For each time of evolving, the supernet in the selected search space trained for 30 epochs from the resumed points if existed or from scratch. If the neighborhood set for learning the tendency function empty, we select the top-5 nearest neighbors whose training epochs are 30 less than the given hypernetwork. Then we train hypernetworks in these search space to acquire the

---

Table Comparison of flash, memory and top-1 accuracy variation across different search space selection method.
Table The effects of different neighborhood sizes in tendency fitting, where we report the inference efficiency and the top-1 accuracy.
Search Space
Flash
Memory
Top-1
Neighborhood
Flash
Memory
Top-1
Random Maximal
L03MB
332kB
65.2%
2
0.79MB
207kB
65.4%
1.11MB
271kB
69.2%
3
0.84MB
231kB
68.2%
Composition
0.96MB
218KB
68.6%
5
0.89MBB
218kB
71.1% %
Ours
0.89MB
218kB
71.1%
8
0.89MB
218kB
71.1%
Search Space Evolution
Token
Patch Embedding decomposition
Operator integerization
MCUFormer
overwriting
218kB
111kB
155kB
327kB
2400kB
Figure 3: The peak memory reduction is achieved through the operator library with memory schedul- ing techniques, including patch embedding decomposition, Operator integration, and token overwrit- ing. The peak memory of the final model adheres to the SRAM constraint, which is 256kB.
accuracy for tendency fitting. The initialized data points for tendency fitting randomly sampled continuous choices. For operator library construction, we utilize int8 quantization for all tensors in the vision transformer during inference. The filter size of the decomposed patch embedding layer is set to 4 x 4 with multiple forward passes to reduce the peak memory, and we iterate the surrogate assignment from the fixed-point iterative methods for 4 times calculate the square root in the layer normalization operators. The hyperparameter assignment and approximation method is listed in the supplementary material.
4.2 Performance Analysis
Effects search space evolution: Low-rank decompo- sition ratio and token number influence the memory sig- nificantly for vision transformer deployment. In order to discover the optimal search space with satisfying trade- off between task performance and memory, we evolve the search space efficiently according the estimated tendency. We compare our search space evolution method with other selection criteria including randomly sampled search space, the maximal search space and composing the search space selection into the supernet learning. Table shows the top- I accuracy, the memory footprint and the flash of different methods, where our approach outperforms others in both the accuracy and efficiency. Randomly selecting the search space fails discover the optimal trade-off between the task performance and the efficiency of storage and memory, while choosing the search space with the highest complex- ity underperforms the evolution methods because of the low ratio of sub-networks within the memory limit. Di- rectly composing the search space selection in supernet training amplifies the search complexity by 15-16 orders magnitude, and the search deficiency prevents the acquisition of the optimal network architectures.
Patch
apochs
0.5
0+
U. Rank
Patch Sie
Patch Size
60 epochs
90 epochs
Low Rank Rabio
Low Rank Ratio
Figure 4: Visualization of search space evolution.
Effects of memory scheduling in the operator library: To enforce the vision transformers to be executable on microcontrollers, we construct operator library to include the lack functions required by transformers. Moreover, we also reschedule the memory during inference fully utilize the memory constraints of microcontrollers through patch embedding decomposition, Operator integration and token overwriting. Figure 3 illustrates the comparison peak memory for the operator library with different memory scheduling techniques. Operator integration makes the most significant contribution to memory reduction (23.8%) because the memory to represent the int8 tensor only costs 25% of that for float one. Patch embedding decomposition is necessary because the large
7

---

convolutional kernels requires large memory space for the MAC operation in convolution. Token overwriting also decreases the memory cost by reusing the buffer for each step of multiplication, since the memory for both input and output can be replaced by that only represents the larger one of them. In conclusion, integrating all above techniques into the operator library reduces the memory cost by 9% and enables deployment of large-scale vision transformers on microcontrollers with satisfying task performance.
The search space evolves according to the fitted tendency between the continuous choices and the task performance and memory, and effective tendency estimation is important to acquiring the optimal network architectures. Considering both the simplification and the fitting precision, we leverage the piecewise linear function to fit the tendency with the neighboring data points near the given search space. Large neighborhoods are more robust to outliers of task performance and continuous choices, while small neighhorhoods generate more precise fitting functions for the selected search space. We investigate the influences of the neighhorhood size on tendency fitting, and deepictss the results for data point sampling with different numbers of nearest neighbors. The assignment of moderate numbers of data points in the neighborhood results in the architecture with the highest accuracy within the memory constraints due to effective search space evolution. We also provide visualization of search space evolution example in Figure 4 where the search space that is selected more frequently during evolution is verified to be more optimal. Different colors represent the evolutionary paths of various candidates, and the arrows indicate the direction of evolution. Double arrows signify that the candidates after evolution are trained with t more times than current total training epochs.
Influence of tendency fitting:
Influence of hyperparameters in search space evolution: Crucial hyperparameters in search space evolution contains the update threshold h2 and h22 which are varied in the ablation study to investigate the influence. where both medium thresholds result in the highest performance. Large thresholds cause frequent search space change without sufficient confidence, while the small ones enforce the evolution process to stuck at the local optimum. Meanwhile, the performance is more sensitive to the update threshold for low-rank decomposition ratio, which significantly changes the optimal architecture.
the top-1 accuracy for different value assignment of thresholds,
4.3 Evaluation on Large-scale Image Classification
In this section, we compare the network architecture search techniques in MCUFormer with tfat-oof-the-ar architecture design methods including the vision transformer based NAS methods including AutoFormer [8] We also compare our op- erator library with existing ones designed for convolutional neural network inference. Since the conventional operator library such as CMSIS-NN [ and TinyEngine [20] cannot be directly utilized for vision transformer inference due to the lack of executable functions for specific design such as GeLU supplemented the missing func-
Acc
m 0.05
66
- 0.000
by 0.015
63
0.1
0.2
Figure 5: The performance variation with respect to the threshold h, and h2 in search space evolution.
and layer normalization, tions to the conventional library to ensure a fair comparison of memory scheduling in vision transformer inference.
Image classification on ImageNet is highly challenging, primarily because of its large scale and high diversity, particularly when using lightweight models deployed on microcontrollers. Table illustrates the t classification accuracy, the memory footprint and flash of different deep neural networks including CNN based and transformer based architectures and various operator libraries. The gray number in the table indicates the the required memory or flash exceeds the memory constraints or flash constraints. For the combination of architectures and operator libraries that exceeds the memory constraint of the microcontrollerss, we report the accuracy achieved during inference with GPUs. For the CNN based architectures (8 30), we employ the TinyEngine 20 for compilation. MCUFormer- and MCUFormer-T respectively represent our method that replaces the network architecture search with AutoFormer and utilizes the TinyEngine with supplemented layers of vision transformers for compilation. Except for MCUFormer-T T, we employ our proposed engine for compilation to evaluate all transformer based architectures.
Since vision transformers outperform CNNs by a sizable margin on large-scale vision tasks, deploying transformers microcontrolleers is very desirable for practical applications. Compared with the
8

---

Table 3: The flash, memory and top-1 accuracy on ImageNet for different network architecture methods and deep learning libraries for microcontrollers, where we utilize three MCUs with various SRAM limits for evaluation.
STM32F4 256KB
STM32F7 320KB
STM32H7 S12KB
Model
Mcm.
Fla.
Acc.
Mem.
Fla.
Acc.
Mem.
Fla.
Acc.
MBV2+CMSIS
308kB
0.72MB
49.0%
MCUNetVI
238kB
0.70MB
60.3%
293kB
0.70MB
61.8%
452kB
1.65MB
68.5%
MCUNetV2
233kB
0.67MB
62.0%
282kB
0.67MB
63.5%
498kB
1.56MB
70.7%
EMQ
253kB
0.73MB
66.5%
308kB
0.71MB
68.2%
507kB
1.67MB
72.8%
AutoFormer
681kB
1.24MBB
74.7%
MCUFomme--
328kB
0.86MB
70.4%
411kB
0.98MB
69.3%
842kB
2.11MB
72.7%
MCUForme-T
816kB
3.81MB
73.3%
872kB
3.90MB
75.4%
1.5MB
7.45MB
76.4%
MCUFormer
218kB
0.89MB
71.1%
319kB
0.90MB
73.6%
505kB
1.95MB
74.0%
state-of-the-art CNN based model EMQ on microcontrollers, our method achieves 5.4% higher top-1 accuracy (68.2% vs. 73.6%) with the 320KB memory limit. Both the search space evolution in architecture search and the memory scheduling in the constructed operator library contribute to deploying vision transformers on microcontrollers with high accuracy by comparing MCUFormer-T, MCUFormer-A and MCUFormer, because the redundant representation that consumes the memory of MCUs in the inference is removed. Although the state-of- the art NAS methods AutoFormer 8 effectively discover the topology of vision transformers with the optimal accuracy-complexigy trade-offs, they fail consider the extremely memory constraints of microcontrollers which not concern for GPUs. Directly supplementing the low-rank decomposition ratio and token numbers to the original search space usually causes search deficiency because of the convergence problems in the supernet training. Meanwhile, TinyEngine optimizes the memory scheduling for general deep network deployment while still ignores the specific architectures in vision transformers for further memory utilization ratio enhancement. Generally speaking, the presented search space evolution in network architecture search and memory scheduling in library construction respectively boost the top-1 accuracy by 4.3% (69.3% vs. 73.6%) and reduce the peak memory by 63% (319kB vs. 872kB) with the STM32F746 which get better result.
We also deploy our hardware-algorithe co-optimizations framework in different controllers with various resource constraint, and Table also demonstrates the top-1 accuracy for various methods. With 256KB memory and IMB flash, our vision transformer still achieves 71.1% 1% top-1 accuracy on ImageNet and outperforms the EMQ method by 6.9%. The task performance is boosted with the increase of the constraint of memory and flash, where the acquired the 74.0% top-1 accuracy on STM32H743 with 512KB memory and 2MB flash even outperforms the Deit [3] 1.8% inferenced on GPUs Hence, our vision transformer can be applied in practical applications with high requirement of accuracy.
Conclusion
In this paper, we have presented MCUFormer for hardware-algorithm co-optimizations that suc- cessfully deploys the vision transformer on microcontrollers with satisfying task performance, We generalize the network architecture search with enlarged search space considering the low-rank decomposition ratio of weight matrix and token number. The search space evolves for supernet training to discover the model topology with the highest accuracy given the resource constraints, where the evolution leverages the tendency between task performance and the continuous choices with enhanced search efficiency, Meanwhile, we construct the operator library for vision transformer inference that converts the model into executable functions, and fully utilize the memory by patch embedding decomposition, Operator integration and token overwriting. Extensive experiments on image classification, person presence identification and key word spotting shows that our MCUFormer achieves competitive performance on microcontrollers with the state-of-the-art vision transformer models. Our work currently contains the following limitation. Considering transformer deployment on MCUs in more diverse vision tasks such as DETR 5 for object detection is important in realistic applications including autonomous driving 38 25 and robot navigation [5 13]

---

Broader Impacts
Deploying transformer on a microcontroiler is particularly beneficial for applications that require quick decision-making or operate in resource-constrained environments. Unlike the traditional CNN, the peak memory of the transformer will not change with the number of channels, so it is relatively difficult to reduce the peak memory. The MCUFormer we designed reduces the peak memory from the structural design and improves the accuracy at the same time.
Acknowledgments
This work was supported in part by the National Key Research and Development Program of China under Grant 20222D0160102, and in part by the National Natural Science Foundation of China under Grant 62321005, Grant 62336004, and Grant 62125603.
References
[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving. Michael Isard, et al. [TensorFlow]: system for (Large-Scale] machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI
16), pages 265-283, 2016.
[2] Michal Adamkiewicz. Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette
Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. JEEE Robotics and
Automation Letters. 7(2):4606-4613, 2022.
[3] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint (e.::i///2..0.331 2018.
[4] Alessandro Capotondi, Manuele Rusci, Marco Fariselli, and Luca Benini, Cmix-nn: Mixed low-precision
cnn library for memory-constrained edge devices. IEFF Transactions on Circuits and Systems II: Express Briefs, 67(5))871-875, 2020.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[6] Boyu Chen, Peixia Li, Baopu Li, Chuming Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Psvit: Better vision transformer via token pooling and attention sharing. arXiv preprint arXiv:2 03428, 2021.
[7] Boyu Chen, Pcixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 12-21, 2021.
[8] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages
12270- 12280, 2021.
[9] Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, and Haibin Ling. Searching the search space of vision transformer. Advances in Neural Information Processing Systems.
34:8714-8726, 2021.
[10] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddic Yan. Meghan Cowan, Haichen Shen, Leyuan Wang. Yuwei Hu, Luis Ceze, al. Tym: An automated end-to-end optimizing compiler for deep learning. arXiv preprint arXiv: pet::000290999 2018.
[11] Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-24188 255.. leee, 2009.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19999, 2020.
[13] Heming Du, Xin Yu, and Liang Zheng. Vtnet: Visual transformer network for object goal navigation.
arXiv preprint arXiv:2 09447, 2021.
[14] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. Proceedings of the IEEE/CVF international conference on computer vision. pages 6824-6835, 2021.
[15] Tianrui Guan, Divya Kothandaraman, Rohan Chandra, Adarsh Jagan Sathyamoorthy, Kasun Weerakoon, and Dinesh Manocha. Ga-nav: Efficient terrain segmentation for robot navigation in unstructured outdoor environments. IEEE Robotics and Automation Letters, ((3))8138888) $155, 2022.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770- 778, 2016.
10

---

[17] Yuanhui Huang. Wenzhao Zheng. Yunpeng Zhang. Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 9223-9232, 2023.
[18] Liangzhen Lai, Naveen Suda, and Vikas Chandra. Cmsis-nn: Efficient neural network kernels for arm
cortex-m cpus. arXiv preprint arXiv: 1eii:/800.06600 3.0600660, 2018.
[19] Ji Lin. Wei-Ming Chen, Han Cai, Chuang Gan, and Song Han. Mcunetv2: Memory-efficient patch-based inference for tiny deep learning. arXiv preprint arXiv:2110.1553
2021.
[20] Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, ct al. Mcunet: Tiny deep leamning on iot
devices. Advances in Neural Information Processing Systems, 33:11711111722 2020.
[21] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint
arXiv:1806.0000 2018.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF
international conference computer vision, pages 10012- 2021.
[23] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang. Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092-22 28103, 2021.
[24] Schastian Mooshauer, Daniel Konig. Jens Jakel, and Michael Teutsch. A benchmark for deep learning based object detection maritime euvironments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[25] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7077-7087, 2021.
[26] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems, -43139971,,999 2021.
[27] Oindrila Saha. Aditya Kusupati, Harsha Vardhan Simhadri, Manik Varma, and Prateek Jain. Rnnpool: Efficient non-linear pooling for ram constrained inference. Advances in Neural Information Processing Systems, 33:20473- 2020.
[28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich
Chich Chen. Mobilenety2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[29] Dhruv Shah. Blažej Ositiski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language. vision. and action. In Conference on Robot Learning. pages 492-504. PMLR. 2023. [30] Zhenhong Sun. Ce Ge. Junyan Wang, Ming Lin, Hesen Chen, Hao Li, and Xiuyu Sun. Entropy-driven mixed-preed precision quantization for deep network design. Advances in Neural Information Processing Systems, 15:21508-21555 2022.
[31] Yunchao Tang. Mingyou Chen, Chenglin Wang. Lufeng Luo, Jinhui Li, Guoping Lian, and Xiangjun Zou. Recognition and localization methods for vision-based fruit picking robots: review. Frontiers in Plant Science, 11:510, 2020.
[32] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J' I'egou. Training data-efficient image transformers & distillation through attention. ArXiv, 12277712.12877, 2020.
[33] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[34] Leon Amadeus Varga, Benjamin Kiefer, Martin Messmer, and Andreas 7ell. Seadronessce: A maritime benchmark for detecting humans open water. In Proceedings of the IEFF/CVF winter conference on applications of computer vision, pages 2260-2270, 2022.
[35] Yuqing Wang. Zhaoliang Xu, Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end vidco instance segmentation with transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8741 18450, 2021.
[36] Ziyi Wang. Yongming Rao, Xumin Yu, Jie Zhou, and Jiwen Lu. Semaffinet: Semantic-aftine transformation for point cloud segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 11819-11829, 2022.
[37] Ziwei Wang, Changyuan Wang, Xiuwei Xu, Jie Zhou, and Jiwen Lu. Quantformer: Learning extremely low-prccision vision transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. [38] Yi Wei, Linging Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21729-2111 21740, 2023.
[39] Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. arXiv preprint 2r37v:2307.01844
2023.
[40] Zhenyu Wu, Xiuwei Xu, Ziwei Wang. Chong Xia. Linging Zhao, Jiwen Lu, and Haibin Yan. Anyview: Generalizable indoor 3d object detection with variable frames, arXiv preprint arXiv:2310.05344 2023. [41] Han Xiao, Ziwei Wang. Zheng Zhu, Jie Zhou, and Jiwen Lu. Shapley-nas: discovering operation contribution for neural architecture search. In Proceedings of the IEEE/CVF cemte/CCeo on computer vision and pattern recognition, pages 1 11892-11911, 2022.

---

[42] Yubui Xu, Lingxi Xie, Xiaopeng Zhang. Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memooyy-efliiien architecture search. arXiv preprint wrXiv:1907.05737,
2019.
[43] Jin Yuan, Yang Li, Xuemei Liu, Xinxue Zhao, and Linfei He. Structure design of egg auto-picking system manipulator motion planning. Transactions of the Chinese Society of Agriculturll Engineering. 3285),48555, 2016.
[44] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: /6/ 1.01588, 2016.
12


Reviews:
[{"review_id": "k8MOZeTlyP", "reviewer": "Reviewer_km2A", "summary": "This paper proposes a co-design method (MCUFormer) that enables the deployment of vision transformers on microcontrollers with limited memory. The method enlarges the search space of vision transformers by considering low-rank decomposition dimensions and patch resolution for memory reduction. The inference library of vision transformers is constructed by scheduling the memory buffer during inference through operator integerization, patch embedding decomposition, and token overwriting.", "questions": "See weakness.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The compiling library for Vision Transformers marks a great contribution to the field. This may provide valuable insights and tools for future research within this domain.\n2. The results showcased in Table 3 are remarkable.", "weaknesses": "While the Compiling Library Construction does provide an element of novelty, the search algorithm essentially leverages the AutoFormer. A deeper discussion and analysis would be beneficial, particularly low-rank decomposition dimensions and patch resolution for memory reduction.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for careful reading and valuable comments. We will check the paper carefully, and modify the presentation of the ambiguous parts in the final version. We provide answers to the questions as follows: \n\nQ: The role of low-rank decomposition dimensions and patch resolution for memory reduction. \\\nA: Because the memory footprint of vision transformers exceeds the budget of MCUs and lacks operator library for transformer inference on MCUs, existing network compression methods such as AutoFormer cannot deploy the model on the MCU. Without quantization and optimization, for STM32F746 320kB constraint, peak memory of AutoFormer reaches over 3MB which is 9.7 times to the constraint. Our technical contribution is listed as follows. \\\n(1)\tsearch algorithm. We first optimize the search space of hypernetworks that is composed of low-rank ratio and token number to reduce the dimension of tensor and peak memory. Due to the extremely high training cost, we evolve the search space by considering the relation among the performance, memory footprint and the search space factors including token numbers and the low-rank decomposition ratio. According to the score function, we get the final candidate hypernetwork and search in this hypernetwork. \\\n(2)\thardware design. We extend operator library including Layernorm and GeLU since existing library is not feasible for vision transformer inference. We decompose the patch embedding layer by transforming the convolution with large receptive field to multiple convolution operations with small receptive field to reduce the memory usage. To add the results from multiple convolution operations, we leverage small memory buffer to record the intermediate value of accumulation. \\\nFor STM32F746 320kB constrains, the peak memory of MCUFormer is 319kB, accuracy is 73.6% and latency is 641ms. For deploying AutoFormer on MCU, using extended operator library, peak memory is 1.24MB and the accuracy is 74.7%.\n"}, {"review_id": "4lhR68mcfY", "reviewer": "Reviewer_N3hf", "summary": "This work proposed a hardware-algorithm co-design method called MCUFormer to enable vision transformer deployment on MCUs with limited memory, which includes both model design and compilation optimization to meet the memory constrain of MCUs. Specifically, the authors utilize one-shot NAS to search the model that fits the memory budget and optimizes the performance at the same time. In addition, they optimize the inference with a set of techniques including operator integerization, patch embedding decomposition, and token overwritting for higher performance. Finally, the experiments shows competitive accuracy improvement given the memory constrain compared to prior CNN based solutions.", "questions": "1) Since MCUformer includes both cutomized NAS framework and model inference and particularly prior NAS framework i.e. AutoFormer can also produce models that are close to this work, an end-to-end comparison of the two NAS frameworks will be helpful. In addition, given the model optimized with AutoFormer, will it be possible to deploy it on MCUs using the proposed inference optimization techniques.\n\n2) There is always a large design space where performance and accuracy can be compromised. While the resulting transformer models show much higher model accuracy, could you further provide the performance comparison over CNN based solutions. \n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed framework generally addresses the deployment problem of transformers on MCUs for the first time and covers the major toolchain on MCUs including model architecture design, compilation, and operation library implementation.  The experiments confirm the feasibility of the proposed framework and shows promising accuracy improvement compared to prior CNN based counterparts.\n\n2) The proposed NAS framework can be aware of the memory limits of MCUs and constructs a compiling library with efficient memory scheduling to enable model deployment on MCUs.", "weaknesses": "1) Some of the key concepts utilized in this paper are confusing. For instance, this paper mentions compiling library and memory scheduling in the proposed framework. Compilation and transformer operation library are totally different things. They need to be clarified in this work especially when MCUs typically do not have a clear AI toolchain definition. As memory scheduling, it is also well defined in computer science and it is a bit misleading when they are utilized as a different consept. \n\n2) There is a lack of ablation study and some of the major technical contributions including the proposed NAS and the model implementation optimization need more evidence. \n\n3) End-to-end performance metrics like fps are not presented. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for careful reading and valuable comments. We will check the paper carefully, and modify the presentation of the ambiguous parts in the final version. We provide answers to the questions as follows: \n\n\nQ1: The confusion of compiling library and memory scheduling and the definition of memory scheduling\n\nA1: \n\nFor compiling library:\n\nWe are sorry for the ambiguity. We extend operator library including Layernorm and GeLU since existing library is not feasible for vision transformer inference. We formulate surrogate equations with fixed-point iterative methods to calculate the output of the square root operator when inferencing Layernorm. We approximate the activation function with quantization by multiplication and softmax function which can both implemented with int8 arithmetics. (We also replace the batchmatmul operator by concatenation, matmul and unpack operators which can implemented on MCU). We will replace the compiling library by operator library in the final version.\n\nFor memory scheduling:\n\nWe have the same definition in [1] which current memory usage contains tembuffer, input tensors and output tensors. Tembuffer storages the tensors which computed in the previous layers and will use in the following layers. For patch embedding layer, the receptive field of the decomposed patch is contained in the memory as well. We have the different definition with the traditional memory scheduling which focus on task priorities and scheduling order including priority setting, interrupt management, interrupt management and task queuing.\n\n[1] MCUNet: Tiny Deep Learning on IoT Devices\n\nQ2: Ablation study of NAS and the model implementation optimization.\n\nA2: Firstly, we compare our search space evolution method with other selection criteria including randomly sampled search space, the maximal search space and composing the search space selection into the supernet learning which you can refer to Table1 in the article. Secondly, we investigate the influences of the neighborhood size on tendency fitting, and depicts the results for data point sampling with different numbers of nearest neighbors, which you can refer to Table2 in the article. Moreover, to compare, we deploy the initial vision transformer, MCUFormer, the network architecture search replace by AutoFormer and missing one of hardware design techniques on STM32F4(memory constraint is 256kB). You can refer to the following table.\n| Model           | Accuracy | Memory Usage |\n|-----------------|----------|--------------|\n| AutoFormer      | 74.7%    | 3MB          |\n| Without NAS     | 74.7%    | 319kB        |\n| Without overwriting | 71.1% | 373kB        |\n| Without token decomposition | 71.1% | 545kB  |\n| MCUFormer       | 71.1%    | 218kB        |\n\nQ3: End-to-end performance\n\nA3: We deploy the vision transformers with our algorithm-hardware framework in different microcontrollers with different resource constraint. We evaluate framework to acquire the accuracy, latency and memory. Our method achieves higher top-1 accuracy and fits the memory constraint.\n\n|           | STM32F4 | STM32F7 | STM32H7 |\n|-----------|---------|---------|---------|\n| Constrain | 256kB   | 320kB   | 512kB   |\n| Accuracy  | 71.1%   | 73.6%   | 74.0%   |\n| Latency   | 706ms   | 641ms   | 798ms   |\n| Memory    | 218kB   | 319kB   | 505kB   |\n\nQ4: Deploying AutoFormer on MCU\n\nBecause the memory footprint of vision transformers exceeds the budget of MCUs and lacks operator library for transformer inference on MCUs, existing network compression methods such as AutoFormer cannot deploy the model on the MCU. Conventional operator library with quantized operators is not feasible for vision transformer inference because of the specific operators including the GeLU activation and layer normalization. So transformer model cannot be inferenced on MCUs. Using our operator library, for STM32F746 320kB constrains, peak memory of AutoFormer reaches 814kB. \n\nQ5: CNN based solutions\n\nBoth MCUNet and EMQ are CNN based solution. We evaluate three models on STM32F7 to measure the accuracy and memory usage. \n\n| Model      | MCUNetV1 | MCUNetV2 | EMQ    | MCUFormer |\n|------------|---------|---------|--------|-----------|\n| Accuracy   | 61.8%   | 63.5%   | 68.2%  | 73.6%     |\n| Memory     | 293kB   | 282kB   | 308kB  | 319kB     |\n\n\n\n\n\n"}, {"review_id": "2pHmHNTZo5", "reviewer": "Reviewer_67yL", "summary": "The paper introduces a hardware-algorithm co-design framework, MCUFormer, that enables the deployment of vision transformers on microcontrollers (MCUs). This framework jointly designs transformer architectures and constructs an inference compiler in order to accommodate the constraints of memory resources. MCUFormer broadens the search space by considering the low-rank decomposition ratio and token numbers, thereby identifying a model topology that offers the highest accuracy within a given resource budget. It also proposes memory scheduling techniques for inference that achieves significant memory reduction, enabling vision transformer on MCU. Experimental results demonstrate MCUFormer achieves higher accuracy on ImageNet compared to previous CNN-based networks on MCU.", "questions": "1. Could you please elaborate on the improvements achieved by the proposed techniques, such as low-rank decomposition, in reducing the model size?\n2. Why were the results of inference latency not included in the experimental results, and could you provide those measurements?\n3. How much search cost is necessary for MCUFormer to accommodate different combinations of models and resource constraints? Could you explain how the search cost was measured or calculated in your study?\n4. Is there a plan to make the MCUFormer framework open-source for further research and development in the community?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Timely contribution: The paper offers a timely contribution to the field by introducing a hardware-algorithm co-design framework capable of deploying vision transformers on MCUs.\n- Methodological soundness: The proposed techniques are reasonable and intuitive. They effectively address the challenges of enabling vision transformers on MCUs, given the limited memory. This complements existing research on efficient ViT, which primarily focuses on reducing computational complexity and storage costs.\n- Practical Implementation: The authors have operationalized their proposed framework by implementing an inference library. They have measured memory usage and latency on real devices, thereby attesting to the practicality of the framework.\n- Proven effectiveness: The evaluation indicates that the proposed framework significantly outperforms CNN-based state-of-the-art networks on MCUs. It delivers a substantial improvement in accuracy (+5.4%) with a similar memory footprint.", "weaknesses": "- Lack of ablation study on model size reduction: The evaluation shows the reduction in model size, i.e., flash memory usage, but it doesn't clearly quantify the improvements achieved through the proposed techniques, such as low-rank decomposition. An ablation study could further enhance the paper's quality.\n- Missing latency measurements: While the experimental results include memory usage and accuracy metrics achieved by the framework, they do not report on inference latency. As many vision tasks on IoT devices are time-sensitive, results of latency measurement are expected.\n- Lack of clarity on search cost: It remains unclear how much search cost is necessary for MCUFormer to support a different combination of models and resource constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for careful reading and valuable comments. We will check the paper carefully, and modify the presentation of the ambiguous parts in the final version. We provide answers to the questions as follows: \n\nQ1: How to reduce the memory\n\nA1: Since the memory footprint of vision transformers exceed MCUs constraint and lack of operator library for transformer affects inference on MCUs, existing network compression methods cannot deploy the model on the MCU. And existing NAS model only focus on parameter instead of memory constraint. So we jointly design transformer architectures within the memory resource constraint to reduce the peak memory.\n\n(1)\tsearch algorithms.\n\nFirstly, Low-rank decomposition and token number reduce the dimension of tensor and peak memory. For example, the dimension of input tensor in the MLP block is token number * (embedding dimension * MLP ratio * rank ratio). We first optimize the search space of hypernetworks composed of low-rank ratio and token number to reduce the dimension of tensor and peak memory. But new hypernetwork compositing two factors contains more than 8×10¬¬30 candidate architectures.\n\nSecondly, because of the convergence difficulty of training the hypernetwork we learn the hypernetwork in the fixed search space consisting of given low-rank decomposition ratios and token numbers, and evolve the search space of hypernetwork optimization by predicting the correlation between the task performance and the search space.\n\nThirdly, due to the strong correlation between the task errors and the continuous choices, we evolve the search space by considering the relation among the performance, memory footprint and the search space factors including token numbers and the low-rank decomposition ratio.\n\nWe compared the details of AutoFormer, AutoFormer composed with two factors and MCUFormer on STM32F4 which can refer in the following table.\n\n| Model      | Accuracy | Peak Memory | GPU Hours |\n|------------|----------|-------------|-----------|\n| AutoFormer | 73.7     | 1.24MB      | 200       |\n| Composed   | 68.6     | 218kB       | 600       |\n| MCUFormer  | 71.1     | 218kB       | 360       |\n\n(2)\thardware design. We extend operator library including Layernorm and GeLU since existing library is not feasible for vision transformer quantized inference. In addition, we decrease the receptive field size of parch embedding layer to reduce the memory usage. We also use overwriting sharing input and output tensors buffer to reduce the memory during fully-connect layer inferencing. We compress the model from 3MB to 319kB, fitting the memory constraint. We compare the details of hardware design effect on STM32F7 which can refer in the following tablei.\n\n|                   | Peak Memory | Quantization Approximation | Parch Embedding Decomposition | Overwriting |\n|-------------------|-------------|----------------------------|------------------------------|-------------|\n| Add               | 319kB       | 319kB                      | 319kB                        |             |\n| Unuse             | Cannot Inference | 474kB                  | 484kB                        |             |\n\nQ2: Search cost and results of inference latency\n\nA2: We deploy the vision transformers with our algorithm-hardware co- optimizations framework in different microcontrollers with different resource constraint. We evaluate framework to acquire the accuracy, latency and memory. Our method achieves higher top-1 accuracy and fit the memory constraint.\n\n| MCU      | STM32F4 | STM32F7 | STM32H7 |\n|------------|----------|---------|---------|\n| Constrain  | 256kB    | 320kB   | 512kB   |\n| Accuracy   | 71.1%    | 73.6%   | 74.0%   |\n| Latency    | 706ms    | 641ms   | 798ms   |\n| Memory     | 218kB   | 319kB   | 505kB   |\n\nFor our method, 320kB constraint as example, we measured on NVIDIA GeForce RTX 3090 and evolve the search space for 360 GPU hours. We also compare with Autoformer and Autoformer composited with low-rank ratio and token number without evolution which can refer to Q1 Table.\n\nQ3: How the search cost was measured\n\nA3: We measure the training time and search time. Train time include evolution and training the selected hypernetwork. \n\nQ4: MCUFormer framework open-source\n\nA4: Thank you for your interest in our work. If our paper is accepted, our code will be made open source.\n\n\n\n\n\n"}, {"review_id": "6jmALyCoUc", "reviewer": "Reviewer_77fa", "summary": "This paper introduces MCUFormer, which aims to deploy vision transformers on micrcontrollers. MCUFormer utilizes two key techniques: 1. search space evolution to search vision transformer architectures within the memory limits; and 2. the construction of a compiling library with efficient memory scheduling for deployment on real-world microcontrollers. MCUFormer achieves 73.62% top-1 accuracy with only 329KB of memory.", "questions": "1. The paper mentions that supplementary materials can be referenced, but it seems that the authors may have forgotten to upload them.\n2. Can you provide more details about your proposed search space evolution algorithm? For example, what is the total search cost, and what memory constraints were used? How did you evaluate the memory consumption of candidate sub-networks during the search? This is a crucial setting that seems to have been missed.\n3. Can you comment on the comparison between using model compression and token pruning algorithms on pre-trained transformers and your proposed search space evolution algorithm? \n\nI will do the final decision based on your rebuttal answers.", "limitations": "The proposed search space evolution algorithm appears to be incremental and the paper’s claim of a system-algorithm co-design approach is unclear. The algorithm is complex and may not significantly reduce memory footprint. \nI'm not sure the novelty of the techniques used in the compiling library is enough.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow\n2. The compiling library, which incorporates several memory optimization techniques, is both effective and novel\n3. The final achieved performance is competitive.", "weaknesses": "1. The proposed search space evolution algorithm appears to be an incremental improvement compared to the work[1].\n2. The proposed search space evolution algorithm is quite complex, and Figure 3 suggests that it may not be the most significant contributor to reducing memory footprint. It is not clear why simpler methods, such as model pruning followed by token pruning, were not used to compress a pre-trained vision transformer.\n3. The paper claims to have a system-algorithm co-design approach. However, the two main techniques proposed in the paper seem to be independent (i.e.: the first one is for algorithm and the second is for system optimization). It is not clear how these two techniques are integrated in a co-design approach.\n4. Some key details are missing. For example, the total search cost of the proposed search space evolution algorithm; the memory constraints used during the search, and the final architecture search processes, etc.\n\n\n[1] Chen, Minghao, et al. \"Searching the search space of vision transformer.\" Advances in Neural Information Processing Systems 34 (2021): 8714-8726.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for careful reading and valuable comments. We will check the paper carefully, and modify the presentation of the ambiguous parts in the final version. We provide answers to the questions as follows: \n\nQ1: Difference between AutoFormerV2[1] and MCUFormer\n\nA1: Traditional vision transformers including ViT, AutoFormer, Q-ViT are trained or inferenced on GPU. MCUFormer inferences on MCU which SRAM and flash are much lower than CPU. Compared with AutoFormerV2, our method uses the peak memory as the limitation instead of parameter. Our main contribution is letting the vision transformer model meet memory constraint.\n\n(1)\tsearch space enlargement\n\nExisting transformer model cannot deploy on MCUs due to the memory constraint. AutoFormerV2 finds the optimal window size, but larger embedding kernel size will increase the buffer usage. Since the dimension of located tensor affects peak memory[2], low-rank decomposition reduces the large matrix dimensions and lower token number reduces the size of intermediate features, we compose two factors into initial search space to find optimal search space. \n\n(2)\tsearch space decomposition. \n\nComposed with low-rank ratio and token number, hypernetwork contains more than 8×10¬¬30 candidate architectures. Due to the convergence difficulties of hypernetwork composited with two factors, the accuracy of network is much lower. Therefore, we optimize the search space of hypernetworks that are composed of the token numbers and the low-rank decomposition ratio. For the detail information of other selection criteria, you can refer to the Table1 in article.\n\n(3)\tevolution mode difference. \n\nAutoFormerV2 also proposes automatic search space design method which could modify the number of blocks contained in stage. But these cannot change the peak memory. For MCUFormer, enumerating all search space for architecture search is not feasible due to the extremely high training cost for one hypernetwork in the given search space. We perform architecture search in the updated search space of hypernetworks. Since two factors influence the trade-off between accuracy and memory constrains, we fit the correlation weight by current accuracy of hypernetwork and the ratio of eligible subnets whose memory footprint is within the SRAM limit. Moreover, We train hypernetworks in these search space for limited epochs for accuracy update to fairly estimate the correlation matrix for the given search space.\n\nDue to AutFormerV2[1] based on swin transformer, we compared with AutoFormer, AutoFormer composed with two factors and MCUFormer on STM32F4 which could refer to following table. \n\n|           | Accuracy | Peak Memory | GPU Hours |\n|-----------|----------|-------------|-----------|\n| AutoFormer| 73.7     | 1.24MB      | 200       |\n| Composed  | 68.6     | 218kB       | 600       |\n| MCUFormer | 71.1     | 218kB       | 360       |\n\nQ2: Effect of search space evolution algorithm in the whole deploying process\n\nA2: For STM32F4, which memory constraint is 256kB, the search space evolution algorithm reduces the peak memory from 329kB to 218kB. We add some ablation study in the following table to demonstrate the effect of search algorithms and hard ware design. Moreover, fixed low-rank ratio and token number cannot get the optimal search space optimal architectures.\n\n|                       | Memory  |\n|-----------------------|---------|\n| ViT + quant           | 811kB   |\n| ViT + quant + evolution| 487kB   |\n| ViT + quant + hardware design| 329kB |\n| MCUFormer             | 218kB   |\n\n\nQ3: Difference between search space evolution algorithm and model compression and token pruning algorithms\n\nA3: For token pruning algorithms, including token pooling(TOME) or dynamic thinning (DynamicViT), it will not change the peak memory of the model because the usage of first block do not decrease. To deploy DynamicViT on MCU, peak memory is over 800kB which exceeds the memory constraint. So we compose low-rank ratio and token number to hypernetwork and evolute in the search space of hypernetworks.\n\nQ3: Co-design approach\n\nA3: We will replace the co-design by co-optimizations in the final version. For algorithm, we first optimize the search space of hypernetworks which is composed of the token number and the low-rank ratio, and then perform architecture search in the updated search space of hypernetworks. For hardware design, we extend the operator library, use overwriting to share buffer and patch embedding decomposition to reduce the memory.\n\nQ4: Supplementary materials\n\nA4: Due to our negligence, we did not submit supplementary materials. We will submit in the final version.\n\n\nQ5: Details of search space evolution\n\nA5:We present the search space evolution framework for effective architecture search of vision transformers within the memory limits. We leverage the one-shot network architecture search method to discover the optimal architecture, where we enlarge the search space by considering low-rank decomposition ratios and token numbers that sizably influences the peak memory during the forward pass.\n\n(1)\ttotal search cost. We deploy the vision transformers with our algorithm-hardware co-optimizations framework in different microcontrollers with different resource constraint. For our method, as 320kB as example, we measured on NVIDIA GeForce RTX 3090 and evolve the search space for 360 GPU hours.\n\n(2)\tmemory constraints and evaluation. We have the same definition in [2]. Current memory usage contains tembuffer, input tensors and output tensors. Taking MLP block as example, the dimension of this tensor is token number * (embedding dimension * MLP ratio * rank ratio). \n\n(3)\tMemory consumption evaluation during search subnet. The equation is peak memory = embedding dimension * ( token number * (1+ MLP ratio) + embedding dimension * MLP ratio * rank ratio)\n\n[1] Chen, Minghao, et al. \"Searching the search space of vision transformer.\"\n[2] MCUNet: Tiny Deep Learning on IoT Devices\n\n\n\n"}, {"review_id": "mk2IdRYT3I", "reviewer": "Reviewer_9JB9", "summary": "This paper focuses on the efficient deployment of vision transformers on microcontroller platforms with limited resources. Vision transformers face significant challenges when deployed on resource constrained MCUs due to their large volume of data movement and intensive computation size. The paper explores algorithmic and systematic optimizations by modifying model architectures and reducing memory footprints. In summary, this paper proposes a hardware-algorithm co-design method called MCUFormer that enables the deployment of vision transformers on microcontrollers. This is achieved through joint design of transformer architectures and construction of an inference compiler that fits within the memory constraints.", "questions": "Please see above for detailed comments. Additionally, I have some questions:\n\n- The paper evaluates the framework on STM32F746 to measure accuracy, latency, and memory usage. However, I cannot find any information regarding latency or speedup results in the main paper. Has this been discussed elsewhere?\n- It would be beneficial if the author could clarify whether the accuracy data was collected from a desktop platform or an MCU device.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work targets an important research problem, and the proposed method generally makes sense. The evaluation shows promising results in terms of accuracy and memory consumption.\n- The proposed method is extensively evaluated on three kinds of resource constraint platforms and an ImageNet dataset.\n- The discussion on the breakdown of memory reduction is good.", "weaknesses": "- Some helpful details regarding speed comparison are missing, which would be useful in demonstrating the performance gain against other optimization methods and existing work.\n- The paper claims to have a hardware-algorithm co-design, but it is unclear where the Hardware design exists. It is more like the co-optimizations between algorithm and compiling system.\n- Existing work has widely discussed NAS with different constraints (e.g., latency, memory footprint, energy), so it would be helpful to further clarify the differences among them.\n- Regarding systematic optimizations, there seems to be limited novelty. Efficient quantized operator or integer-only inference implementation has already been discussed in existing work, e.g., [1-3]. It appears that operator integration provides another implementation for specific quantized operators/kernels. Temporary buffer reusing is also widely used in existing inference frameworks such as TFLite and MNN.\n- Minor issues include improving the writing style and some typos: \"Operator integerization\" to \"Operator integration.\"\n\n[1] A Microcontroller is All You Need: Enabling Transformer Execution on Low-Power IoT Endnodes\n\n[2] FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer\n\n[3] I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for careful reading and valuable comments. We will check the paper carefully, and modify the presentation of the ambiguous parts in the final version. We provide answers to the questions as follows: \n\nQ1: Compare with existing work.\n\nA1: Due to lack of operator library for vision transformer inference, we cannot deploy transformer on MCU based on existing NAS work. So we compared with CNN based solution. Since the article only trains the model on GPU and do not deploys the model on GPU, we can not get the latency. We will compared with MCUNet in the following table. All the latency is normalized to STM32F7. Compared with CNN based model, MCUFormer archieves higher accuracy and have the similiar latency. \n\n|           | Peak Memory | Accuracy | Latency |\n|-----------|-------------|----------|---------|\n| MCUNetV1  | 293kB       | 61.8%    | 890ms   |\n| MCUNetV2  | 282kB       | 63.5%    | 501ms   |\n| MCUFormer | 319kB       | 73.6%    | 641ms   |\n\nQ2: The results of inference latency\n\nA2: We deploy vision transformers using our algorithm-hardware framework across diverse microcontrollers characterized by varying resource constraints. Our assessment of the framework encompasses a comprehensive evaluation, encompassing accuracy, latency, and memory usage. Our method achieves higher top-1 accuracy and fit the memory constraint.\n\n| MCU      | STM32F4 | STM32F7 | STM32H7 |\n|------------|----------|---------|---------|\n| Constrain  | 256kB    | 320kB   | 512kB   |\n| Accuracy   | 71.1%    | 73.6%   | 74.0%   |\n| Latency    | 706ms    | 641ms   | 798ms   |\n| Memory     |  218kB   | 319kB   | 505kB   |\n\nQ3: co-design system\n\nA3: We are sorry for the ambiguity. For algorithm, We first optimize the search space of hypernetworks that is composed of the token numbers and the low-rank decomposition ratio, and then performs architecture search in the updated search space of hypernetworks. The evolution algorithm gets the balance between accuracy and memory constrains. For hardware design, we extend the operator library, use overwriting to share buffer and patch embedding decomposition to reduce the memory. Initially, co-design means both the algorithm and hardware design reduce the memory to fit the constraints on MCU. We will replace the co-design by co-optimizations in the final version.\n\nQ4: Limited novelty\n\nA4: Compared with [4], we have less error due to less approximation rewrite. And we have the co-optimization system to reduce the memory. Compared with [5][6], we construct a compiling library with efficient memory scheduling to enable practical deployment. \n\n(1)\tefficient quantized operator or integer-only inference\n\nFirstly, compared with[4], they approximate softmax operator which will influence the accuracy. To get the balance between the inference latency and the prediction precision, we only make Layernorm and GeLU approximation and we achieve 8-bit inference on GPU. \nSecondly, Since we should use existed operators to translate model and the translation of layers influences latency, FQ-ViT[2] and I-ViT[3] cannot get better performance when deploying on MCU. During the process of inference, [5][6] use the power function, which will greatly increase the latency, and also contain operators that cannot be translated. But for MCUFormer, we construct surrogate equations with fixed-point iterative methods for 3 times to calculate the output of the square root operators which can be translated.\n\n(2)\texisting inference frameworks\n\nThough it is easy to support cross-platform development, TF-Lite requires extra runtime memory, to store the meta-information (such as model structure parameters). For overwriting, TF-Lite will use additional temporary buffer which will increase the peak memory. Our method writes the output of matrix multiplication for each token in the same memory space of the input activations and share the same memory. \n\nQ5: Typos\n\nA5: We are sorry for typos, and we will make the correction and replace ‘Operator integerization’ by ‘Operator integration’ in the final version. Thank you for pointing it out.\n\nQ6: Accuracy data collection\n\nA6: Following the settings in MCUNet and EMQ, we report the accuracy acquired on inference with GPUs. \n\n[1] MCUNet: Tiny Deep Learning on IoT Devices\n[2] Entropy-Driven Mixed-Precision Quantization for Deep Network Design \n[4] A Microcontroller is All You Need: Enabling Transformer Execution on Low-Power IoT Endnodes\n[5] FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer\n[6] I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference\n\n\n\n\n\n\n\n"}]

Author rebuttal:

Meta-review: The paper builds a Vision Transformer (MCUFormer) that is able to run under very limited memory constraints on microcontrollers, and retain good accuracy. The do this, the paper proposes a generalized one-shot NAS method, alongside additional techniques to save memory. The paper received unanimous accept recommendations, however, 3/5 of these were only "Borderline". The reviewers highlight the importance of the work, as well as the strong results. The additional analysis in the rebuttal was also appreciated. One valid concern could be that the MCUFormer combines a combination of ML and engineering tricks, and it is not clear what the key generalizable scientific takeaway of the paper is. However, achieving strong results on a practical problem can still be of great value to the community, so I recommend acceptance.
Decision: Accept (poster)