Title: MCUFormer: Deploying Vision Tranformers on Microcontrollers with Limited Memory
Venue: NeurIPS 2023
Accepted: yes
Keywords: Vision transformer

microcontroller

network architecture search
Abstract: Due to the high price and heavy energy consumption of GPUs, deploying deep models on IoT devices such as microcontrollers makes significant contributions for ecological AI. Conventional methods successfully enable convolutional neural network inference of high resolution images on microcontrollers, while the framework for vision transformers that achieve the state-of-the-art performance in many vision applications still remains unexplored. In this paper, we propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory, where we jointly design transformer architecture and construct the inference operator library to fit the memory resource constraint. More specifically, we generalize the one-shot network architecture search (NAS) to discover the optimal architecture with highest task performance given the memory budget from the microcontrollers, where we enlarge the existing search space of vision transformers by considering the low-rank decomposition dimensions and patch resolution for memory reduction. For the construction of the inference operator library of vision transformers, we schedule the memory buffer during inference through operator integration, patch embedding decomposition, and token overwriting, allowing the memory buffer to be fully utilized to adapt to the forward pass of the vision transformer. Experimental results demonstrate that our MCUFormer achieves 73.62\% top-1 accuracy on ImageNet for image classification with 320KB memory on STM32F746 microcontroller. Code is available at https://github.com/liangyn22/MCUFormer.

MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory

Abstract

Due to the high price and heavy energy consumption of GPUs, deploying deep models on IoT devices such as microcontrollers makes significant contributions for ecological AI. Conventional methods successfully enable convolutional neural network inference of high resolution images on microcontrollers, while the framework for vision transformers that achieve the state-of-the-art performance in many vision applications still remains unexplored. In this paper, we propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory, where we jointly design transformer architecture and construct the inference operator library to fit the memory resource constraint. More specifically, we generalize the one-shot network architecture search (NAS) to discover the optimal architecture with highest task performance given the memory budget from the microcontrollers, where we enlarge the existing search space of vision transformers by considering the low-rank decomposition dimensions and patch resolution for memory reduction. For the construction of the inference operator library of vision transformers, we schedule the memory buffer during inference through operator integration, patch embedding decomposition, and token overwriting, allowing the memory buffer to be fully utilized to adapt to the forward pass of the vision transformer. Experimental results demonstrate that our MCUFormer achieves 73.62% top-1 accuracy on ImageNet for image classification with 320KB memory on STM32F746 microcontroller. Code is available at https://github.com/liangyn22/MCUFormer.

1 Introduction

Deep neural network deployment for realistic applications usually requires high-performance computing devices such as GPUs and TPUs [16, 12]. Due to the high price and energy consumption of these devices, the unacceptable deployment expenses strictly restrict the utilization of deep models in a widely variety of tasks[17, 39], especially in the scenarios without sufficient battery support. Deploying deep neural networks on cheap IoT devices with low energy consumption becomes a practical solution in many industrial applications such as maritime detection [34, 24], agricultural picking [31, 43] and robot navigation [2, 29].

Recently, the hardware-algorithm co-design framework [20, 19, 30] is widely studied to deploy deep convolutional networks with high storage and computational complexity to microcontroller units (MCU) with extremely limited memory (SRAM) and storage (flash). By searching the optimal architectures and arranging the memory buffer according to the overall network topology, deep convolutional neural networks with large numbers of parameters and high FLOPs are successfully deployed on MCUs with comparable accuracy as inferencing on high-performance computing devices. Vision transformers [22, 33] have surpassed convolutional neural networks in a wide variety of tasks, which can further push the limit of intelligent visual perception in resource-constrained devices.

However, the memory footprint of vision transformers significantly exceeds the budget of MCUs due to the matrix multiplication with large sizes, and the lack of operator library for transformer inference on MCUs also obscure the practical deployment.

In this paper, we present a hardware-algorithm co-optimizations framework to deploy vision transformers on MCUs with extremely limited memory budget. We jointly design transformer architectures that achieve the highest task performance within the memory resource constraint and construct an operator library for transformer inference with efficient memory scheduling. More specifically, we leverage the one-shot network architecture search method to discover the optimal architecture, where we enlarge the search space by considering low-rank decomposition ratios and token numbers that significantly influence the peak memory during the forward pass. To efficiently acquire effective supernets in the huge search space, we learn the supernet in the fixed search space consisting of given low-rank decomposition ratios and token numbers, and evolve the search space of supernet optimization by predicting the correlation between the task performance and the search space. To construct the operator library within the memory budget, we approximate full-precision operators such as softmax and GeLU layers with integer arithmetics. Moreover, we decompose the patch embedding layers with large convolution kernels to multiple patchification steps with small receptive fields, and overwrite the input token features in the matrix multiplication during the inference for further memory reduction. Extensive experimental results show our method can achieve 73.62% top-1 accuracy on the large-scale image classification dataset ImageNet [11] with the 320KB SRAM STM32F746 microcontrollers, and significantly outperforms the CNN based state-of-the-art networks on MCUs by 5.4%. Our contributions can be summarized as follows:

To the best of our knowledge, we propose the first hardware-algorithm co-optimizations framework that successfully deploys vision transformers on MCUs with the competitive performance in challenging computer vision tasks

We present the search space evolution framework for effective architecture search of vision transformers within the memory limits, and construct a operator library with efficient memory scheduling to enable practical deployment.

We conduct extensive empirical studies to show the feasibility of deploying vision transformers on MCUs with extremely limited memory.

2 Related Work

Efficient vision transformers: Vision transformers mine the global dependencies via self-attention that outperform convolutional neural networks on a wide variety of tasks such as image classification [22, 37], object detection [5, 40] and instance segmentation [35, 36]. To reduce the computational and storage complexity, efficient architecture design, sparse attention and parameter quantization are presented for efficiency enhancement. For efficient architecture design, Chen et al. [6] shared the self-attention in consecutive layers to avoid redundant calculation in dependency discovering, and Fan et al. [14] fused multiscale intermediate features to reduce the resolution of top layers. For sparse attention, Rao et al. [26] dynamically pruned uninformative tokens during inference according to the computation budget and sample features. For parameter quantization, Liu et al. [23] extended the post-training quantization framework by mimicking the self-attention rank in the full-precision model. However, existing methods usually focus on reducing the computational complexity measured by FLOPs and the storage cost evaluated by the number of parameters, and ignore the memory limit in inference that is the main resource bottleneck of MCUs.

Network architecture search: Network architecture search aims to discover the topology including operators and connections to achieve the optimal trade-off between the accuracy and complexity, which can be divided into two categories according to the search algorithms. Non-differentiable methods [3, 7, 44] utilize reinforcement learning or evolutionary algorithms to update the selection regarding the defined reward or fitness, where the candidates can be updated from scratch or from a pre-trained supernet. Differentiable methods [21, 41, 42] construct a large supernet that contains all possible choices represented by different parallel branches. By optimizing the importance weights of different choices, the operators and connections with the highest importance weight are selected to form the final architecture. In this paper, we focus on learning the supernet in the optimal search space defined by the low-rank decomposition ratio and token numbers, which influence the inference memory significantly.

↑Score

Search Space K

rxn

(rk,tk)

New Search

Space

Evolution

wx+yo

nxr

x(t+1)=x(t-[]·n

Original Search

Space

Evolving

nxn

nxr2

r2xn

Path

Low-rankWeightDecomposition

(r3,t3)

Search Space 3

Low-rankRatio/Token Number

SearchSpace2

TendencyRegression

(r2,t2)

Choices

Embed

Dim

SearchSpace 1

(r,t1)

MLP

Ratio

txtTokenst2xt2Tokens

.··

CustomTokenization

SearchSpace Evolution

Deep learning on microcontrollers: Due to the low price and energy consumption, employing microcontrollers for deep learning inference and training has been comprehensively researched in recent years. Existing frameworks contain TensorFlow Lite Micro [1], CMSIS-NN [18], CMix-NN [4], MicroTVM [10] and TinyEngine [20]. Nevertheless, these compiling libraries are designed for convolutional neural networks and lack operators of vision transformers such as LayerNorm and GeLU. Meanwhile, the layer-wise optimization of memory footprint cannot fully utilize the SRAM budget of the microcontrollers and fails to deploy vision transformers with high task performance. For the memory-efficient network architecture design, Lin et al. reduced the image resolution and the width multiplier for peak memory reduction. Sun et al. leveraged the mixed-precision quantization that assigned the optimal bitwidth for each layer to achieve the optimal accuracy-memory trade-off, and Saha et al. [27] decreased the resolution of the activation after the proposed RNNPool layer. However, the memory reduction techniques are designed for convolutional neural networks, which are not feasible for vision transformers with different operators.

3 Approach

In this section, we detail the hardware-algorithm co-optimizations framework to deploy vision transformers on microcontrollers. We first introduce the network architecture search of vision transformers with limited memory constraint, and demonstrate the operator library construction for vision transformer inference on microcontrollers.

3.1 Network Architecture Search with Memory Constraint

Conventional architecture search methods usually consider the computational complexity (FLOPs) and the storage cost (the number of parameters), while the peak memory obscures the deployment of vision transformers on microcontrollers due to the extremely limited SRAM. We extend the one-shot NAS to search the optimal architecture, where we first learn some of supernets randomly which containing all possible topology choices and employ evolutionary algorithms to seek for the optimal supernet given the resource budget. Figure 1 demonstrates the pipeline of the network architecture search with memory constraint. To fully leverage the achievements of vision transformer NAS methods, we build the search space of sub-networks on the basis of AutoFormer [8] which searches the embedding dimension, QKV dimension, MLP ratio, head number and depth number with trained hypernetworks. The search space of supernet is defined by the the low-rank decomposition ratio and token numbers that can significantly influence the memory footprint. The token number impacts the size of intermediate features during inference and can be modified by varying the receptive

field in the patch embedding layers. Different from the multiply-accumulate (MAC) operations of convolutional neural networks, the matrix multiplication in large sizes causes heavy memory burden. To address this, we generalize low-rank decomposition for the multi-layer perceptron (MLP) layers in the following form:

where x ∈ R n × d means the input features with n tokens and d elements of each token. U r ∈ R d × r and V r ∈ R d × r are two low-rank matrices approximating the original weights in the MLP layer, and b means the bias term. Since the dimension of low-rank matrices affects the trade-off between memory footprint and task performance, we are required to find the optimal low-rank decomposition ratio in our search space selection.

Directly integrating the token numbers and the low-rank decomposition ratio into the original search space in AutoFormer causes convergence difficulties of supernets because of the large discrepancy among optimal architectures of different low-rank decomposition ratios. Therefore, we first optimize the search space of hypernetworks that is composed of the token numbers and the low-rank decomposition ratio, and then performs architecture search in the updated search space of hypernetworks. The above two steps are iteratively implemented until convergence or achieving the maximum search cost. Enumerating all search space for architecture search is not feasible due to the extremely high training cost for a single supernet in the given search space. Inspired by the observation in [9] that there is a strong correlation between the task errors and the continuous choices, we evolve the search space by considering the relation among the performance, memory footprint and the search space factors including token number and the low-rank decomposition ratio. The score function S to evaluate the optimality of different hypernetwork search space is defined as follows:

where A is the accuracy of the hypernetwork in the search space and η is a hyperparameter. R is the ratio of sub-networks whose memory footprint is within the SRAM limit. Meanwhile, enumerating all sub-networks in the supernet to acquire the average memory is prohibitive due to the extremely large numbers of candidates in the subspace, and we randomly sample N sub-networks to approximately evaluate the value of R for the given hypernetwork. The linear fitting functions are employed to estimate the tendency between the score and the factors of the search space as S = w T x + S 0 with the intercept S 0 , where x = [ x 1 , x 2 ] consists of low-rank decomposition ratio and token numbers. The search space is updated with the following rules:

where x ( t ) i means the i th element in x at the t th step and w i represents the i th element in w . The optimization step ∆ i is set to the interval of adjacent factor values in our method, and the update threshold h i are hyperparameters for the optimization. [ x ] demonstrates the largest integer smaller than x .

Training the hypernetwork in the evolving search space until convergence requires unacceptable optimization cost, and we fit the correlation between task performance and the continuous choices of search space factors in the training process. For each evolved search space, we train the supernet for t epochs from the resume points or scratch if pre-trained model does not exist. To fairly discover the tendency between the task performance and the continuous choices, we fit the correlation weight matrix w with the data points in the same total training epochs. Since the linear fitting function only holds in a small local region for a given search space, we utilize the piecewise linear fitting function that only considers top-k nearest neighbors for the search space factor x . The data point sampling strategy for tendency fitting can be represented as follows:

where Φ( x ) means the set of data points consisting of task accuracy with memory and continuous choices of low-rank decomposition ratio and token numbers for fitting the linear function in the evolving search space x . The neighborhood for x with k nearest neighbors is denoted as N k ( x ) , and the total training epochs for the supernet in the search space x are represented by E ( x ) . When the neighborhood of the given search space x is empty, we construct the neighborhood by considering the top-k nearest neighbors whose total training epochs are E ( x ) -t . We train hypernetworks in

LargeKernel

InputFeatures

Temporary

Buffer

MAC

TokenFeatures

Weights

ConvolutionResults

Multiple Small

Kernel 1

Kernel 2

Kernels

Kernel3

Kernel4

Record

Accumulation

TemporaryFeatures

Output

Overwrite

TokenFeatures

OutputFeatures

(a)PatchEmbeddingDecomposition

(b)TokenOverwriting

these search space for t epochs for accuracy update to fairly estimate the correlation matrix w for the given search space x .

The selected supernet in the optimal search space is used for subsequent training until full convergence and is employed for the network architecture search of vision transformers. In this process, the evolutionary algorithm is applied with the aim of maximizing accuracy while adhering to the memory constraint.

3.2 Operator Library Construction for Vision Transformers

Deploying the vision transformer on microcontrollers for inference requires the operator library to transform the model into executable functions. Existing operator libraries for deep learning are comparable with convolutional neural networks, while fail to implement operations in vision transformers such as the GeLU activation function and the layer normalization. Meanwhile, the inference efficiency of the library cannot fully utilize the memory budget of microcontrollers to achieve higher task performance with larger vision transformer models. Despite of the efficient memory scheduling techniques that are general to all network architectures in [20], we present the following memory utilization enhancement techniques to enable deployment of vision transformers on microcontrollers.

Patch embedding decomposition: Images are patchified by the patch embedding layer in vision transformers for inference. The input samples are transformed into patches by convolution operations, where both the kernel size and stride are both equal to the patch size. Since the kernel size in the patch embedding layer (larger than 16 × 16 ) is much larger than common convolutional neural networks, most memory cost of the patch embedding layer results from the filter weights instead of the input and output activation. In order to effectively address this challenge, we decompose the patch embedding layer by transforming the convolution with large receptive field to multiple convolution operations with a small receptive field. To add the results from multiple convolution operations, we leverage a small memory buffer to record the intermediate value of accumulation, effectively reducing the peak memory usage during image patchification. Figure 2a demonstrates the comparison between the original patch embedding operation and that with our decomposition. As decomposing the patch embedding layer into excess convolutional operators with small receptive fields can obviously increase the inference latency due to the multiple forward passes, we select the receptive field of the decomposed patch embedding layer as 4 × 4 to achieve the satisfying trade-off between the memory footprint and the inference latency.

Operator integration: Quantizing network parameters and substituting MAC operations with integer arithmetics have been proven beneficial in reducing the memory of convolutional neural networks without causing performance degradation. However, conventional operator library with quantized operators is not feasible for vision transformer inference because of the specific operators including the GeLU activation and layer normalization. Since GeLU requires the cumulative distribution

function (CDF) of Gaussian distribution, we approximate the activation function with quantization by multiplication and softmax function which can both implemented with int8 arithmetics. Similarly, the square root operators in the layer normalization is not supported by int8 arithmetics in operator library. We construct surrogate equations with fixed-point iterative methods to calculate the output of the square root operators. Considering the inference latency of layer normalization operator, we only iterate the surrogate equation for 3 times with balance between the inference latency and the prediction precision. We detail the pseudo algorithm of integer-only square root operation in the supplementary material.

Token overwriting: The matrix multiplication in vision transformer inference that results in high memory consumption impose element interaction only within tokens, and the interaction among tokens is enabled in self-attention computation whose memory cost is far less than the SRAM limit. Therefore, we can overwrite the intermediate features during inference of fully-connected layers because the input activation of each token will no longer be used after acquiring the output features of the corresponding token. Figure 2b illustrates the memory scheduling of matrix multiplication in existing inference operator libraries and our token overwriting techniques, where our method writes the output of matrix multiplication for each token in the same memory space as the input activation. Since the peak memory usage occurs in the matrix multiplications, we assign the maximum memory size M for vision transformer inference based on the network architecture, according to the following criteria:

where h l f , h l i and h l o represent the height of weights, input features, output features in the l th fullyconnected layers, and w l f , w l i and w l o demonstrate the width of the above tensors. Since the memory consumed by the input activation can be overwritten by the output features, we only assign the memory buffer with the same size as the larger one between input and output tensors.

4 Experiments

In this section, we first introduce our hardware configuration, dataset and the implementation details of our hardware-algorithm co-optimizations framework, and then we conduct thorough performance analysis including the task performance and the memory consumption with respect to the presented techniques in network architecture search and operator library construction. Finally, we compare the performance in accuracy and efficiency with existing network architecture search methods and operator libraries for deep model deployment on microcontrollers.

4.1 Setups

Hardware configuration: We deploy the vision transformers with our hardware-algorithm cooptimizations framework in different microcontrollers with various resource constraint including STM32F427 (Cortex-M4/256KB memory/1MB flash), STM32F746 (Cortex-M7/320KB memory/1MB flash) and STM32H743 (Cortex-M7/512KB memory/2MB flash). In the performance analysis, we evaluate our framework on STM32F746 to acquire the accuracy and memory.

Dataset: We conduct the experiments on ImageNet for image classification, which contains 1.2 million training images and 50k validation images from 1000 classes. All images are scaled and biased into the range [-1,1] for normalization. For the training process, we resize the images with the shorter side as 256 and randomly crop a 240 × 240 region. For inference, we utilize the central crop with the size of 240 × 240.

Implementation details: For the network architecture search of vision transformers, our choices for the search space consisting of low-rank decomposition ratio r and the token size c can be selected from r ∈ [0 . 4 : 0 . 05 : 0 . 95] and c ∈ { 16 , 20 , 24 , 28 , 32 } . The supernet design and the evolutionary search in the learned hypernetwork keep the same with those in [8] to fully leverage the potential in architecture search with high degrees of freedom. During the search space evolution, we only select 5 nearest neighbors with the same total training epochs to fit the piecewise linear function for acquiring the slope. For each time of evolving, the supernet in the selected search space is trained for 30 epochs from the resumed points if existed or from scratch. If the neighborhood set for learning the tendency function is empty, we select the top-5 nearest neighbors whose training epochs are 30 less than the given hypernetwork. Then we train hypernetworks in these search space to acquire the

accuracy for tendency fitting. The initialized data points for tendency fitting are randomly sampled continuous choices. For operator library construction, we utilize int8 quantization for all tensors in the vision transformer during inference. The filter size of the decomposed patch embedding layer is set to 4 × 4 with multiple forward passes to reduce the peak memory, and we iterate the surrogate assignment from the fixed-point iterative methods for 4 times to calculate the square root in the layer normalization operators. The hyperparameter assignment and approximation method is listed in the supplementary material.

4.2 Performance Analysis

Effects of search space evolution: Low-rank decomposition ratio and token number influence the memory significantly for vision transformer deployment. In order to discover the optimal search space with satisfying tradeoff between task performance and memory, we evolve the search space efficiently according to the estimated tendency. We compare our search space evolution method with other selection criteria including randomly sampled search space, the maximal search space and composing the search space selection into the supernet learning. Table 1 shows the top1 accuracy, the memory footprint and the flash of different methods, where our approach outperforms others in both the accuracy and efficiency. Randomly selecting the search space fails to discover the optimal trade-off between the task performance and the efficiency of storage and memory, while choosing the search space with the highest complexity underperforms the evolution methods because of the low ratio of sub-networks within the memory limit. Di-

↑PatchSize

2

忆

30epochs

0.5

0.6

0.7

0.8

0.9

Low Rank Ratio

PatchSize

PatchSize

+

120 epochs

0.85

60epochs

90epochs

0.8

0.9

0.85

0.95

Low Rank Ratio

Low Rank Ratio

rectly composing the search space selection in supernet training amplifies the search complexity by 15-16 orders of magnitude, and the search deficiency prevents the acquisition of the optimal network architectures.

Effects of memory scheduling in the operator library: To enforce the vision transformers to be executable on microcontrollers, we construct an operator library to include the lack functions required by transformers. Moreover, we also reschedule the memory during inference to fully utilize the memory constraints of microcontrollers through patch embedding decomposition, Operator integration and token overwriting. Figure 3 illustrates the comparison of peak memory for the operator library with different memory scheduling techniques. Operator integration makes the most significant contribution to memory reduction (23.8%) because the memory to represent the int8 tensor only costs 25% of that for a float one. Patch embedding decomposition is necessary because the large

convolutional kernels requires large memory space for the MAC operation in convolution. Token overwriting also decreases the memory cost by reusing the buffer for each step of multiplication, since the memory for both input and output can be replaced by that only represents the larger one of them. In conclusion, integrating all above techniques into the operator library reduces the memory cost by 9% and enables deployment of large-scale vision transformers on microcontrollers with satisfying task performance.

Influence of tendency fitting: The search space evolves according to the fitted tendency between the continuous choices and the task performance and memory, and effective tendency estimation is important to acquiring the optimal network architectures. Considering both the simplification and the fitting precision, we leverage the piecewise linear function to fit the tendency with the neighboring data points near the given search space. Large neighborhoods are more robust to outliers of task performance and continuous choices, while small neighborhoods generate more precise fitting functions for the selected search space. We investigate the influences of the neighborhood size on tendency fitting, and Table 2 depicts the results for data point sampling with different numbers of nearest neighbors. The assignment of moderate numbers of data points in the neighborhood results in the architecture with the highest accuracy within the memory constraints due to effective search space evolution. We also provide a visualization of a search space evolution example in Figure 4, where the search space that is selected more frequently during evolution is verified to be more optimal. Different colors represent the evolutionary paths of various candidates, and the arrows indicate the direction of evolution. Double arrows signify that the candidates after evolution are trained with t more times than current total training epochs.

Influence of hyperparameters in search space evolution: Crucial hyperparameters in search space evolution contains the update threshold h 1 and h 2 , which are varied in the ablation study to investigate the influence. Figure 5 demonstrates the top-1 accuracy for different value assignment of thresholds, where both medium thresholds result in the highest performance. Large thresholds cause frequent search space change without sufficient confidence, while the small ones enforce the evolution process to stuck at the local optimum. Meanwhile, the performance is more sensitive to the update threshold for low-rank decomposition ratio, which significantly changes the optimal architecture.

4.3 Evaluation on Large-scale Image Classification

In this section, we compare the network architecture search techniques in MCUFormer with state-of-the-art architecture design methods including the vision transformer based NAS methods including AutoFormer [8].We also compare our operator library with existing ones designed for convolutional neural network inference. Since the conventional operator library such as CMSIS-NN [18] and TinyEngine [20] cannot be directly utilized for vision transformer inference due to the lack of executable functions for specific design such as GeLU and layer normalization, we supplemented the missing functions to the conventional library to ensure a fair comparison of memory scheduling in vision transformer inference.

Image classification on ImageNet is highly challenging, primarily because of its large scale and high diversity, particularly when using lightweight models deployed on microcontrollers. Table 3 illustrates the top-1 classification accuracy, the memory footprint and flash of different deep neural networks including CNN based and transformer based architectures and various operator libraries. The gray number in the table indicates the the required memory or flash exceeds the memory constraints or flash constraints. For the combination of architectures and operator libraries that exceeds the memory constraint of the microcontrollers, we report the accuracy achieved during inference with GPUs. For the CNN based architectures[28, 30], we employ the TinyEngine [20] for compilation. MCUFormerA and MCUFormer-T respectively represent our method that replaces the network architecture search with AutoFormer and utilizes the TinyEngine with supplemented layers of vision transformers for compilation. Except for MCUFormer-T, we employ our proposed engine for compilation to evaluate all transformer based architectures.

Since vision transformers outperform CNNs by a sizable margin on large-scale vision tasks, deploying transformers on microcontrollers is very desirable for practical applications. Compared with the

state-of-the-art CNN based model EMQ on microcontrollers, our method achieves 5.4% higher top-1 accuracy (68.2% vs. 73.6%) with the 320KB memory limit. Both the search space evolution in architecture search and the memory scheduling in the constructed operator library contribute to deploying vision transformers on microcontrollers with high accuracy by comparing MCUFormer-T, MCUFormer-A and MCUFormer, because the redundant representation that consumes the memory of MCUs in the inference is removed. Although the state-of-the art NAS methods AutoFormer [8] effectively discover the topology of vision transformers with the optimal accuracy-complexity trade-offs, they fail to consider the extremely memory constraints of microcontrollers which are not a concern for GPUs. Directly supplementing the low-rank decomposition ratio and token numbers to the original search space usually causes search deficiency because of the convergence problems in the supernet training. Meanwhile, TinyEngine optimizes the memory scheduling for general deep network deployment while still ignores the specific architectures in vision transformers for further memory utilization ratio enhancement. Generally speaking, the presented search space evolution in network architecture search and memory scheduling in library construction respectively boost the top-1 accuracy by 4.3% (69.3% vs. 73.6%) and reduce the peak memory by 63% (319kB vs. 872kB) with the STM32F746 which get better result.

We also deploy our hardware-algorithm co-optimizations framework in different controllers with various resource constraint, and Table 3 also demonstrates the top-1 accuracy for various methods. With 256KB memory and 1MB flash, our vision transformer still achieves 71.1% top-1 accuracy on ImageNet and outperforms the EMQ method by 6.9%. The task performance is boosted with the increase of the constraint of memory and flash, where the acquired the 74.0% top-1 accuracy on STM32H743 with 512KB memory and 2MB flash even outperforms the Deit [32] 1.8% inferenced on GPUs. Hence, our vision transformer can be applied in practical applications with high requirement of accuracy.

5 Conclusion

In this paper, we have presented MCUFormer for hardware-algorithm co-optimizations that successfully deploys the vision transformer on microcontrollers with satisfying task performance. We generalize the network architecture search with enlarged search space considering the low-rank decomposition ratio of weight matrix and token number. The search space evolves for supernet training to discover the model topology with the highest accuracy given the resource constraints, where the evolution leverages the tendency between task performance and the continuous choices with enhanced search efficiency. Meanwhile, we construct the operator library for vision transformer inference that converts the model into executable functions, and fully utilize the memory by patch embedding decomposition, Operator integration and token overwriting. Extensive experiments on image classification, person presence identification and key word spotting shows that our MCUFormer achieves competitive performance on microcontrollers with the state-of-the-art vision transformer models. Our work currently contains the following limitation. Considering transformer deployment on MCUs in more diverse vision tasks such as DETR [5] for object detection is important in realistic applications including autonomous driving[38, 25] and robot navigation[15, 13].

Broader Impacts

Deploying transformer on a microcontroller is particularly beneficial for applications that require quick decision-making or operate in resource-constrained environments. Unlike the traditional CNN, the peak memory of the transformer will not change with the number of channels, so it is relatively difficult to reduce the peak memory. The MCUFormer we designed reduces the peak memory from the structural design and improves the accuracy at the same time.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2022ZD0160102, and in part by the National Natural Science Foundation of China under Grant 62321005, Grant 62336004, and Grant 62125603.

References

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. { TensorFlow } : a system for { Large-Scale } machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16) , pages 265-283, 2016.

Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. IEEE Robotics and Automation Letters , 7(2):4606-4613, 2022.

Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 , 2018.

Alessandro Capotondi, Manuele Rusci, Marco Fariselli, and Luca Benini. Cmix-nn: Mixed low-precision cnn library for memory-constrained edge devices. IEEE Transactions on Circuits and Systems II: Express Briefs , 67(5):871-875, 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

Boyu Chen, Peixia Li, Baopu Li, Chuming Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Psvit: Better vision transformer via token pooling and attention sharing. arXiv preprint arXiv:2108.03428 , 2021.

Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12-21, 2021.

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12270-12280, 2021.

Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, and Haibin Ling. Searching the search space of vision transformer. Advances in Neural Information Processing Systems , 34:8714-8726, 2021.

Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. arXiv preprint arXiv:1802.04799 , 2018.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Heming Du, Xin Yu, and Liang Zheng. Vtnet: Visual transformer network for object goal navigation. arXiv preprint arXiv:2105.09447 , 2021.

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6824-6835, 2021.

Tianrui Guan, Divya Kothandaraman, Rohan Chandra, Adarsh Jagan Sathyamoorthy, Kasun Weerakoon, and Dinesh Manocha. Ga-nav: Efficient terrain segmentation for robot navigation in unstructured outdoor environments. IEEE Robotics and Automation Letters , 7(3):8138-8145, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9223-9232, 2023.

Liangzhen Lai, Naveen Suda, and Vikas Chandra. Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus. arXiv preprint arXiv:1801.06601 , 2018.

Ji Lin, Wei-Ming Chen, Han Cai, Chuang Gan, and Song Han. Mcunetv2: Memory-efficient patch-based inference for tiny deep learning. arXiv preprint arXiv:2110.15352 , 2021.

Ji Lin, Wei-Ming Chen, Yujun Lin, Chuang Gan, Song Han, et al. Mcunet: Tiny deep learning on iot devices. Advances in Neural Information Processing Systems , 33:11711-11722, 2020.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems , 34:28092-28103, 2021.

Sebastian Moosbauer, Daniel Konig, Jens Jakel, and Michael Teutsch. A benchmark for deep learning based object detection in maritime environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 0-0, 2019.

Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7077-7087, 2021.

Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems , 34:13937-13949, 2021.

Oindrila Saha, Aditya Kusupati, Harsha Vardhan Simhadri, Manik Varma, and Prateek Jain. Rnnpool: Efficient non-linear pooling for ram constrained inference. Advances in Neural Information Processing Systems , 33:20473-20484, 2020.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Dhruv Shah, Bła˙ zej Osi´ nski, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning , pages 492-504. PMLR, 2023.

Zhenhong Sun, Ce Ge, Junyan Wang, Ming Lin, Hesen Chen, Hao Li, and Xiuyu Sun. Entropy-driven mixed-precision quantization for deep network design. Advances in Neural Information Processing Systems , 35:21508-21520, 2022.

Yunchao Tang, Mingyou Chen, Chenglin Wang, Lufeng Luo, Jinhui Li, Guoping Lian, and Xiangjun Zou. Recognition and localization methods for vision-based fruit picking robots: A review. Frontiers in Plant Science , 11:510, 2020.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. ArXiv , abs/2012.12877, 2020.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Leon Amadeus Varga, Benjamin Kiefer, Martin Messmer, and Andreas Zell. Seadronessee: A maritime benchmark for detecting humans in open water. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2260-2270, 2022.

Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8741-8750, 2021.

Ziyi Wang, Yongming Rao, Xumin Yu, Jie Zhou, and Jiwen Lu. Semaffinet: Semantic-affine transformation for point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11819-11829, 2022.

Ziwei Wang, Changyuan Wang, Xiuwei Xu, Jie Zhou, and Jiwen Lu. Quantformer: Learning extremely low-precision vision transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundocc: Multicamera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21729-21740, 2023.

Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848 , 2023.

Zhenyu Wu, Xiuwei Xu, Ziwei Wang, Chong Xia, Linqing Zhao, Jiwen Lu, and Haibin Yan. Anyview: Generalizable indoor 3d object detection with variable frames. arXiv preprint arXiv:2310.05346 , 2023.

Han Xiao, Ziwei Wang, Zheng Zhu, Jie Zhou, and Jiwen Lu. Shapley-nas: discovering operation contribution for neural architecture search. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11892-11901, 2022.

Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737 , 2019.

Jin Yuan, Yang Li, Xuemei Liu, Xinxue Zhao, and Linfei He. Structure design of egg auto-picking system and manipulator motion planning. Transactions of the Chinese Society of Agricultural Engineering , 32(8):48-55, 2016.

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the efficient deployment of vision transformers on microcontroller platforms with limited resources. Vision transformers face significant challenges when deployed on resource constrained MCUs due to their large volume of data movement and intensive computation size. The paper explores algorithmic and systematic optimizations by modifying model architectures and reducing memory footprints. In summary, this paper proposes a hardware-algorithm co-design method called MCUFormer that enables the deployment of vision transformers on microcontrollers. This is achieved through joint design of transformer architectures and construction of an inference compiler that fits within the memory constraints.\n\nSTRENGTHS:\n- This work targets an important research problem, and the proposed method generally makes sense. The evaluation shows promising results in terms of accuracy and memory consumption.\n- The proposed method is extensively evaluated on three kinds of resource constraint platforms and an ImageNet dataset.\n- The discussion on the breakdown of memory reduction is good.\n\nWEAKNESSES:\n- Some helpful details regarding speed comparison are missing, which would be useful in demonstrating the performance gain against other optimization methods and existing work.\n- The paper claims to have a hardware-algorithm co-design, but it is unclear where the Hardware design exists. It is more like the co-optimizations between algorithm and compiling system.\n- Existing work has widely discussed NAS with different constraints (e.g., latency, memory footprint, energy), so it would be helpful to further clarify the differences among them.\n- Regarding systematic optimizations, there seems to be limited novelty. Efficient quantized operator or integer-only inference implementation has already been discussed in existing work, e.g., [1-3]. It appears that operator integration provides another implementation for specific quantized operators/kernels. Temporary buffer reusing is also widely used in existing inference frameworks such as TFLite and MNN.\n- Minor issues include improving the writing style and some typos: \"Operator integerization\" to \"Operator integration.\"\n\n[1] A Microcontroller is All You Need: Enabling Transformer Execution on Low-Power IoT Endnodes\n\n[2] FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer\n\n[3] I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference\n\nQUESTIONS:\nPlease see above for detailed comments. Additionally, I have some questions:\n\n- The paper evaluates the framework on STM32F746 to measure accuracy, latency, and memory usage. However, I cannot find any information regarding latency or speedup results in the main paper. Has this been discussed elsewhere?\n- It would be beneficial if the author could clarify whether the accuracy data was collected from a desktop platform or an MCU device.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MCUFormer, which aims to deploy vision transformers on micrcontrollers. MCUFormer utilizes two key techniques: 1. search space evolution to search vision transformer architectures within the memory limits; and 2. the construction of a compiling library with efficient memory scheduling for deployment on real-world microcontrollers. MCUFormer achieves 73.62% top-1 accuracy with only 329KB of memory.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow\n2. The compiling library, which incorporates several memory optimization techniques, is both effective and novel\n3. The final achieved performance is competitive.\n\nWEAKNESSES:\n1. The proposed search space evolution algorithm appears to be an incremental improvement compared to the work[1].\n2. The proposed search space evolution algorithm is quite complex, and Figure 3 suggests that it may not be the most significant contributor to reducing memory footprint. It is not clear why simpler methods, such as model pruning followed by token pruning, were not used to compress a pre-trained vision transformer.\n3. The paper claims to have a system-algorithm co-design approach. However, the two main techniques proposed in the paper seem to be independent (i.e.: the first one is for algorithm and the second is for system optimization). It is not clear how these two techniques are integrated in a co-design approach.\n4. Some key details are missing. For example, the total search cost of the proposed search space evolution algorithm; the memory constraints used during the search, and the final architecture search processes, etc.\n\n\n[1] Chen, Minghao, et al. \"Searching the search space of vision transformer.\" Advances in Neural Information Processing Systems 34 (2021): 8714-8726.\n\nQUESTIONS:\n1. The paper mentions that supplementary materials can be referenced, but it seems that the authors may have forgotten to upload them.\n2. Can you provide more details about your proposed search space evolution algorithm? For example, what is the total search cost, and what memory constraints were used? How did you evaluate the memory consumption of candidate sub-networks during the search? This is a crucial setting that seems to have been missed.\n3. Can you comment on the comparison between using model compression and token pruning algorithms on pre-trained transformers and your proposed search space evolution algorithm? \n\nI will do the final decision based on your rebuttal answers.\n\nLIMITATIONS:\nThe proposed search space evolution algorithm appears to be incremental and the paper’s claim of a system-algorithm co-design approach is unclear. The algorithm is complex and may not significantly reduce memory footprint. \nI'm not sure the novelty of the techniques used in the compiling library is enough.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a hardware-algorithm co-design framework, MCUFormer, that enables the deployment of vision transformers on microcontrollers (MCUs). This framework jointly designs transformer architectures and constructs an inference compiler in order to accommodate the constraints of memory resources. MCUFormer broadens the search space by considering the low-rank decomposition ratio and token numbers, thereby identifying a model topology that offers the highest accuracy within a given resource budget. It also proposes memory scheduling techniques for inference that achieves significant memory reduction, enabling vision transformer on MCU. Experimental results demonstrate MCUFormer achieves higher accuracy on ImageNet compared to previous CNN-based networks on MCU.\n\nSTRENGTHS:\n- Timely contribution: The paper offers a timely contribution to the field by introducing a hardware-algorithm co-design framework capable of deploying vision transformers on MCUs.\n- Methodological soundness: The proposed techniques are reasonable and intuitive. They effectively address the challenges of enabling vision transformers on MCUs, given the limited memory. This complements existing research on efficient ViT, which primarily focuses on reducing computational complexity and storage costs.\n- Practical Implementation: The authors have operationalized their proposed framework by implementing an inference library. They have measured memory usage and latency on real devices, thereby attesting to the practicality of the framework.\n- Proven effectiveness: The evaluation indicates that the proposed framework significantly outperforms CNN-based state-of-the-art networks on MCUs. It delivers a substantial improvement in accuracy (+5.4%) with a similar memory footprint.\n\nWEAKNESSES:\n- Lack of ablation study on model size reduction: The evaluation shows the reduction in model size, i.e., flash memory usage, but it doesn't clearly quantify the improvements achieved through the proposed techniques, such as low-rank decomposition. An ablation study could further enhance the paper's quality.\n- Missing latency measurements: While the experimental results include memory usage and accuracy metrics achieved by the framework, they do not report on inference latency. As many vision tasks on IoT devices are time-sensitive, results of latency measurement are expected.\n- Lack of clarity on search cost: It remains unclear how much search cost is necessary for MCUFormer to support a different combination of models and resource constraints.\n\nQUESTIONS:\n1. Could you please elaborate on the improvements achieved by the proposed techniques, such as low-rank decomposition, in reducing the model size?\n2. Why were the results of inference latency not included in the experimental results, and could you provide those measurements?\n3. How much search cost is necessary for MCUFormer to accommodate different combinations of models and resource constraints? Could you explain how the search cost was measured or calculated in your study?\n4. Is there a plan to make the MCUFormer framework open-source for further research and development in the community?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a hardware-algorithm co-design method called MCUFormer to enable vision transformer deployment on MCUs with limited memory, which includes both model design and compilation optimization to meet the memory constrain of MCUs. Specifically, the authors utilize one-shot NAS to search the model that fits the memory budget and optimizes the performance at the same time. In addition, they optimize the inference with a set of techniques including operator integerization, patch embedding decomposition, and token overwritting for higher performance. Finally, the experiments shows competitive accuracy improvement given the memory constrain compared to prior CNN based solutions.\n\nSTRENGTHS:\n1) The proposed framework generally addresses the deployment problem of transformers on MCUs for the first time and covers the major toolchain on MCUs including model architecture design, compilation, and operation library implementation.  The experiments confirm the feasibility of the proposed framework and shows promising accuracy improvement compared to prior CNN based counterparts.\n\n2) The proposed NAS framework can be aware of the memory limits of MCUs and constructs a compiling library with efficient memory scheduling to enable model deployment on MCUs.\n\nWEAKNESSES:\n1) Some of the key concepts utilized in this paper are confusing. For instance, this paper mentions compiling library and memory scheduling in the proposed framework. Compilation and transformer operation library are totally different things. They need to be clarified in this work especially when MCUs typically do not have a clear AI toolchain definition. As memory scheduling, it is also well defined in computer science and it is a bit misleading when they are utilized as a different consept. \n\n2) There is a lack of ablation study and some of the major technical contributions including the proposed NAS and the model implementation optimization need more evidence. \n\n3) End-to-end performance metrics like fps are not presented.\n\nQUESTIONS:\n1) Since MCUformer includes both cutomized NAS framework and model inference and particularly prior NAS framework i.e. AutoFormer can also produce models that are close to this work, an end-to-end comparison of the two NAS frameworks will be helpful. In addition, given the model optimized with AutoFormer, will it be possible to deploy it on MCUs using the proposed inference optimization techniques.\n\n2) There is always a large design space where performance and accuracy can be compromised. While the resulting transformer models show much higher model accuracy, could you further provide the performance comparison over CNN based solutions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a co-design method (MCUFormer) that enables the deployment of vision transformers on microcontrollers with limited memory. The method enlarges the search space of vision transformers by considering low-rank decomposition dimensions and patch resolution for memory reduction. The inference library of vision transformers is constructed by scheduling the memory buffer during inference through operator integerization, patch embedding decomposition, and token overwriting.

STRENGTHS:
1. The compiling library for Vision Transformers marks a great contribution to the field. This may provide valuable insights and tools for future research within this domain.
2. The results showcased in Table 3 are remarkable.

WEAKNESSES:
While the Compiling Library Construction does provide an element of novelty, the search algorithm essentially leverages the AutoFormer. A deeper discussion and analysis would be beneficial, particularly low-rank decomposition dimensions and patch resolution for memory reduction.

QUESTIONS:
See weakness.

LIMITATIONS:
The authors have addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06