SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details: For the network architecture search of vision transformers, our choices for the search space consisting of low-rank decomposition ratio  $r$  and the token size  $c$  can be selected from  $r \in [0.4 : 0.05 : 0.95]$  and  $c \in \{16, 20, 24, 28, 32\}$ . The supernet design and the evolutionary search in the learned hypernetwork keep the same with those in [8] to fully leverage the potential in architecture search with high degrees of freedom. During the search space evolution, we only select 5 nearest neighbors with the same total training epochs to fit the piecewise linear function for acquiring the slope. For each time of evolving, the supernet in the selected search space is trained for 30 epochs from the resumed points if existed or from scratch. If the neighborhood set for learning the tendency function is empty, we select the top-5 nearest neighbors whose training epochs are 30 less than the given hypernetwork. Then we train hypernetworks in these search space to acquire the
Table 1: Comparison of flash, memory and top-1 accuracy variation across different search space selection method.
Search Space	Flash	Memory	Top-1
Random	1.03MB	332kB	65.2%
Maximal	1.11MB	271kB	69.2%
Composition	0.96MB	218kB	68.6%
Ours	0.89MB	218kB	71.1%
Table 2: The effects of different neighborhood sizes in tendency fitting, where we report the inference efficiency and the top-1 accuracy.
Neighborhood	Flash	Memory	Top-1
2	0.79MB	207kB	65.4%
3	0.84MB	231kB	68.2%
5	0.89MB	218kB	71.1%
8	0.89MB	218kB	71.1%
MCUFormer	Search Space Evolution	Token overwriting	Patch Embedding decomposition	Operator integerization
218kB	111kB	155kB	327kB	2400kB
accuracy for tendency fitting. The initialized data points for tendency fitting are randomly sampled continuous choices. For operator library construction, we utilize int8 quantization for all tensors in the vision transformer during inference. The filter size of the decomposed patch embedding layer is set to  $4 \times 4$  with multiple forward passes to reduce the peak memory, and we iterate the surrogate assignment from the fixed-point iterative methods for 4 times to calculate the square root in the layer normalization operators. The hyperparameter assignment and approximation method is listed in the supplementary material.
4.2 Performance Analysis
Effects of search space evolution: Low-rank decomposition ratio and token number influence the memory significantly for vision transformer deployment. In order to discover the optimal search space with satisfying tradeoff between task performance and memory, we evolve the search space efficiently according to the estimated tendency. We compare our search space evolution method with other selection criteria including randomly sampled search space, the maximal search space and composing the search space selection into the supernet learning. Table 1 shows the top-1 accuracy, the memory footprint and the flash of different methods, where our approach outperforms others in both the accuracy and efficiency. Randomly selecting the search space fails to discover the optimal trade-off between the task performance and the efficiency of storage and memory, while choosing the search space with the highest complexity underperforms the evolution methods because of the low ratio of sub-networks within the memory limit. Di
Figure 3: The peak memory reduction is achieved through the operator library with memory scheduling techniques, including patch embedding decomposition, Operator integration, and token overwriting. The peak memory of the final model adheres to the SRAM constraint, which is  $256\mathrm{kB}$ .
Figure 4: Visualization of search space evolution.
directly composing the search space selection in supernet training amplifies the search complexity by 15-16 orders of magnitude, and the search deficiency prevents the acquisition of the optimal network architectures.
Effects of memory scheduling in the operator library: To enforce the vision transformers to be executable on microcontrollers, we construct an operator library to include the lack functions required by transformers. Moreover, we also reschedule the memory during inference to fully utilize the memory constraints of microcontrollers through patch embedding decomposition, Operator integration and token overwriting. Figure 3 illustrates the comparison of peak memory for the operator library with different memory scheduling techniques. Operator integration makes the most significant contribution to memory reduction (23.8%) because the memory to represent the int8 tensor only costs 25% of that for a float one. Patch embedding decomposition is necessary because the large
convolutional kernels requires large memory space for the MAC operation in convolution. Token overwriting also decreases the memory cost by reusing the buffer for each step of multiplication, since the memory for both input and output can be replaced by that only represents the larger one of them. In conclusion, integrating all above techniques into the operator library reduces the memory cost by  $9\%$  and enables deployment of large-scale vision transformers on microcontrollers with satisfying task performance.
Influence of tendency fitting: The search space evolves according to the fitted tendency between the continuous choices and the task performance and memory, and effective tendency estimation is important to acquiring the optimal network architectures. Considering both the simplification and the fitting precision, we leverage the piecewise linear function to fit the tendency with the neighboring data points near the given search space. Large neighborhoods are more robust to outliers of task performance and continuous choices, while small neighborhoods generate more precise fitting functions for the selected search space. We investigate the influences of the neighborhood size on tendency fitting, and Table 2 depicts the results for data point sampling with different numbers of nearest neighbors. The assignment of moderate numbers of data points in the neighborhood results in the architecture with the highest accuracy within the memory constraints due to effective search space evolution. We also provide a visualization of a search space evolution example in Figure 4, where the search space that is selected more frequently during evolution is verified to be more optimal. Different colors represent the evolutionary paths of various candidates, and the arrows indicate the direction of evolution. Double arrows signify that the candidates after evolution are trained with  $t$  more times than current total training epochs.
Influence of hyperparameters in search space evolution: Crucial hyperparameters in search space evolution contains the update threshold  $h_1$  and  $h_2$ , which are varied in the ablation study to investigate the influence. Figure 5 demonstrates the top-1 accuracy for different value assignment of thresholds, where both medium thresholds result in the highest performance. Large thresholds cause frequent search space change without sufficient confidence, while the small ones enforce the evolution process to stuck at the local optimum. Meanwhile, the performance is more sensitive to the update threshold for low-rank decomposition ratio, which significantly changes the optimal architecture.
4.3 Evaluation on Large-scale Image Classification
In this section, we compare the network architecture search techniques in MCUFormer with state-of-the-art architecture design methods including the vision transformer based NAS methods including AutoFormer [8]. We also compare our operator library with existing ones designed for convolutional neural network inference. Since the conventional operator library such as CMSIS-NN [18] and TinyEngine [20] cannot be directly utilized for vision transformer inference due to the lack of executable functions for specific design such as GeLU and layer normalization, we supplemented the missing functions to the conventional library to ensure a fair comparison of memory scheduling in vision transformer inference.
Figure 5: The performance variation with respect to the threshold  $h_1$  and  $h_2$  in search space evolution.
Image classification on ImageNet is highly challenging, primarily because of its large scale and high diversity, particularly when using lightweight models deployed on microcontrollers. Table 3 illustrates the top-1 classification accuracy, the memory footprint and flash of different deep neural networks including CNN based and transformer based architectures and various operator libraries. The gray number in the table indicates the required memory or flash exceeds the memory constraints or flash constraints. For the combination of architectures and operator libraries that exceeds the memory constraint of the microcontrollers, we report the accuracy achieved during inference with GPUs. For the CNN based architectures[28, 30], we employ the TinyEngine [20] for compilation. MCUFormer-A and MCUFormer-T respectively represent our method that replaces the network architecture search with AutoFormer and utilizes the TinyEngine with supplemented layers of vision transformers for compilation. Except for MCUFormer-T, we employ our proposed engine for compilation to evaluate all transformer based architectures.
Since vision transformers outperform CNNs by a sizable margin on large-scale vision tasks, deploying transformers on microcontrollers is very desirable for practical applications. Compared with the
Table 3: The flash, memory and top-1 accuracy on ImageNet for different network architecture methods and deep learning libraries for microcontrollers, where we utilize three MCUs with various SRAM limits for evaluation.
Model	STM32F4 256KB	STM32F7 320KB	STM32H7 512KB
Mem.	Fla.	Acc.	Mem.	Fla.	Acc.	Mem.	Fla.	Acc.
MBV2+CMSIS	-	-	-	308kB	0.72MB	49.0%	-	-	-
MCUNetV1	238kB	0.70MB	60.3%	293kB	0.70MB	61.8%	452kB	1.65MB	68.5%
MCUNetV2	233kB	0.67MB	62.0%	282kB	0.67MB	63.5%	498kB	1.56MB	70.7%
EMQ	253kB	0.73MB	66.5%	308kB	0.71MB	68.2%	507kB	1.67MB	72.8%
AutoFormer	-	-	-	681kB	1.24MB	74.7%	-	-	-
MCUFormer-A	328kB	0.86MB	70.4%	411kB	0.98MB	69.3%	842kB	2.11MB	72.7%
MCUFormer-T	816kB	3.81MB	73.3%	872kB	3.90MB	75.4%	1.5MB	7.45MB	76.4%
MCUFormer	218kB	0.89MB	71.1%	319kB	0.90MB	73.6%	505kB	1.95MB	74.0%
state-of-the-art CNN based model EMQ on microcontrollers, our method achieves  $5.4\%$  higher top-1 accuracy (68.2% vs. 73.6%) with the 320KB memory limit. Both the search space evolution in architecture search and the memory scheduling in the constructed operator library contribute to deploying vision transformers on microcontrollers with high accuracy by comparing MCUFormer-T, MCUFormer-A and MCUFormer, because the redundant representation that consumes the memory of MCUs in the inference is removed. Although the state-of-the-art NAS methods AutoFormer [8] effectively discover the topology of vision transformers with the optimal accuracy-complexity trade-offs, they fail to consider the extremely memory constraints of microcontrollers which are not a concern for GPUs. Directly supplementing the low-rank decomposition ratio and token numbers to the original search space usually causes search deficiency because of the convergence problems in the supernet training. Meanwhile, TinyEngine optimizes the memory scheduling for general deep network deployment while still ignores the specific architectures in vision transformers for further memory utilization ratio enhancement. Generally speaking, the presented search space evolution in network architecture search and memory scheduling in library construction respectively boost the top-1 accuracy by  $4.3\%$  (69.3% vs. 73.6%) and reduce the peak memory by  $63\%$  (319kB vs. 872kB) with the STM32F746 which get better result.
We also deploy our hardware-algorithm co-optimizations framework in different controllers with various resource constraint, and Table 3 also demonstrates the top-1 accuracy for various methods. With 256KB memory and 1MB flash, our vision transformer still achieves  $71.1\%$  top-1 accuracy on ImageNet and outperforms the EMQ method by  $6.9\%$ . The task performance is boosted with the increase of the constraint of memory and flash, where the acquired the  $74.0\%$  top-1 accuracy on STM32H743 with 512KB memory and 2MB flash even outperforms the Deit [32]  $1.8\%$  inferred on GPUs. Hence, our vision transformer can be applied in practical applications with high requirement of accuracy.
In this paper, we have presented MCUFormer for hardware-algorithm co-optimizations that successfully deploys the vision transformer on microcontrollers with satisfying task performance. We generalize the network architecture search with enlarged search space considering the low-rank decomposition ratio of weight matrix and token number. The search space evolves for supernet training to discover the model topology with the highest accuracy given the resource constraints, where the evolution leverages the tendency between task performance and the continuous choices with enhanced search efficiency. Meanwhile, we construct the operator library for vision transformer inference that converts the model into executable functions, and fully utilize the memory by patch embedding decomposition, Operator integration and token overwriting. Extensive experiments on image classification, person presence identification and key word spotting shows that our MCUFormer achieves competitive performance on microcontrollers with the state-of-the-art vision transformer models. Our work currently contains the following limitation. Considering transformer deployment on MCUs in more diverse vision tasks such as DETR [5] for object detection is important in realistic applications including autonomous driving[38, 25] and robot navigation[15, 13].
Deploying transformer on a microcontroller is particularly beneficial for applications that require quick decision-making or operate in resource-constrained environments. Unlike the traditional CNN, the peak memory of the transformer will not change with the number of channels, so it is relatively difficult to reduce the peak memory. The MCUFormer we designed reduces the peak memory from the structural design and improves the accuracy at the same time.
This work was supported in part by the National Key Research and Development Program of China under Grant 2022ZD0160102, and in part by the National Natural Science Foundation of China under Grant 62321005, Grant 62336004, and Grant 62125603.
[42] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737, 2019.
[43] Jin Yuan, Yang Li, Xuemei Liu, Xinxue Zhao, and Linfei He. Structure design of egg auto-picking system and manipulator motion planning. Transactions of the Chinese Society of Agricultural Engineering, 32(8):48-55, 2016.
[44] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.