SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: The amount of MACs (Millions) used in StreamNet and MCUNetv2
Model	n_batch	splitidx	MNv2	MNv2+	SN-1D	SN-2D	Speedup over MNv2
MNv2+	SN-1D	SN-2D
MB2	18	13	77.4	73.1	36.9	23.5	1.06	2.10	3.29
PL	22	13	349.6	326.5	88.6	38.4	1.07	3.95	9.11
MI0	12	06	15.5	14.4	8.5	6.4	1.08	1.82	2.43
MI1	12	11	27.0	25.4	16.6	12.8	1.06	1.62	2.11
MI2	20	21	998.8	920.2	189.2	67.4	1.09	5.28	14.82
MI3	22	21	1,784.7	1,650.7	271.7	81.9	1.08	6.57	21.78
MI4	20	17	1,942.1	1,787.8	353.8	126.0	1.09	5.49	15.41
MV0	08	13	23.1	20.0	9.7	6.0	1.15	2.39	3.87
MV1	20	14	96.1	89.1	24.2	11.6	1.08	3.97	8.28
MV2	18	21	826.8	754.3	153.3	55.9	1.10	5.39	14.79
GMEAN	1.08	3.45	7.18
finds the one that can result in the minimal usage of SRAM memory space in patch-based inference. Thus, results in Table 2 show the amount of MACs used in the TinyML model with the lowest peak SRAM memory usage through all possible combinations of n_batch and splitidx. The amount of the MACs in patch-based inference increases with the reduction of the usage on the SRAM memory. Therefore, MCUNetv2 (MNv2) obtains a significant amount of the MACs over the other alternatives in Table 2. In addition, MCUNetv2+ (MNv2+) reduces roughly  $8\%$  of MACs on average over the baseline MCUNetv2. The output padding bypassing of MCUNetv2+ removes overhead on the padding data and achieves such a reduction on the amount of MACs. Moreover, StreamNet-1D (SN-1D) performs 1D stream_y processing and saves the redundant computation through its stream buffer. Thus, StreamNet-1D reduces  $65\%$  of MACs on average over the baseline. To completely remove the redundant computation on patch-based inference (MCUNetv2), StreamNet-2D (SN-2D) performs 2D streaming processing to increase more data reuse than StreamNet-1D. Hence, StreamNet-2D reduces  $83\%$  of MACs on average over MCUNetv2 across 10 TinyML models. StreamNet-2D saves  $93\%$  and  $95\%$  of MACs on MI2 and MI3, since these models consist of many patch-based layers with the large splitidx value and the amount of MACs on the redundant computation is large.
5.4 Reveal the Latency of StreamNet
This evaluation aims to figure out if StreamNet can shorten the execution time of TinyML models by reducing the amount of their MACs. Table 3 compares the latency of each TinyML model across different implementations on MCUNetv2(MNv2) and StreamNet. The n_batch and splitidx in Table 3 and Table 2 use the same value. Table 3 shows the execution time when TinyML models use minimum SRAM memory space through patch-based inference. MCUNetv2+ (MNv2+), StreamNet-1D (SN-1D), and StreamNet-2D (SN-2D) achieves a geometric mean of 1.1X, 3.4X, 7.3X speedup
Table 3: The latency (ms) of TinyML models on StreamNet and MCUNetv2
Model	MNv2	MNv2+	SN-1D	SN-2D	Speedup over MNv2
MNv2+	SN-1D	SN-2D
MB2	1,514	1,391	687	417	1.09	2.20	3.63
PL	6,296	5,795	1,610	676	1.09	3.91	9.31
MI0	238	219	131	99	1.09	1.81	2.40
MI1	449	413	258	188	1.09	1.74	2.39
MI2	16,545	15,055	3,281	1,168	1.10	5.04	14.17
MI3	29,700	27,331	4,858	1,444	1.09	6.11	20.57
MI4	26,816	24,747	5,153	1,762	1.08	5.20	15.22
MV0	415	355	169	101	1.17	2.45	4.10
MV1	1,817	1,667	473	225	1.09	3.84	8.09
MV2	14,106	13,040	2,778	961	1.08	5.08	14.68
GMEAN	1.10	3.40	7.28
over the baseline MCUNetv2. The speedup of Table 3 presents the consistent results over the amount of MACs in Table 2. Hence, this result provides StreamNet the evidence to exploit the MAC count obtained at compile time to infer the final latency in StreamNet auto-tuning framework.
(a) SRAM memory consumption
(b) Auto-tuning framework
Figure 7: Results of SRAM Memory consumption and StreamNet's auto-tuning framework
5.5 Analysis of Peak SRAM Memory Usage
Figure 7(a) presents the peak SRAM memory usage of TinyML models on different implementations of MCUNetv2 (2) and StreamNet. MCUNetv2 (2), StreamNet-1D, and StreamNet-2D save  $66\%$ ,  $62\%$ , and  $47\%$  SRAM memory usage on average over the layer-wise inference baseline, respectively. Unlike patch-based inference, StreamNet-1D reuses the data within the stream buffer that occupies the SRAM memory space. Thus, StreamNet-1D consumes additional  $4\%$  SRAM memory space on average to store the stream buffer over the MCUNetv2 (2). To increase the ratio of the data reuse, StreamNet-2D requires about  $22\%$  extra SRAM memory space on average for the stream buffer to cover more data reuse than StreamNet-1D. StreamNet provides a trade-off between the performance and the peak usage of the SRAM memory. Unlike patch-based inference, StreamNet improves the performance of TinyML models without sacrificing too much SRAM memory space.
5.6 Results of StreamNet Auto-tuning Framework
This evaluation aims to validate if StreamNet auto-tuning framework can create a case that satisfies its memory constraint while achieving the best performance. Figure 7(b) presents the performance speedup results when adding  $0\%$  to  $15\%$  memory space on each TinyML model based on the minimal SRAM memory usage of StreamNet-1D. In addition, we also design an auto-tuning framework for MCUNetV2 (2) that scans through its n_batch and splitidx parameters to obtain a case with the best performance while meeting the memory constraint. Unlike MCUNetV2 (2) auto-tuning framework, StreamNet searches for the combination of StreamNet-1D and StreamNet-2D network to fit the memory constraint while achieving the best performance. As illustrated in Figure 7(b), StreamNet achieves a geometric mean of 1.63X, 1.61X, 1.8X speedup over MCUNetV2 (2) when the memory constraint is the base memory space plus  $5\%$ ,  $10\%$ , and  $15\%$ , respectively. In Figure 7(b), MIO
achieves only a speedup of 1.07X to 1.09X over MCUNetV2 (2), where the value of the split_idx and n_patches is small, leading to minor recomputation in MCUNetV2 (2). Unlike MI0, MI3 uses larger split_idx and n_patches values to meet the memory constraints. The large value of split_idx and n_patches allows StreamNet to eliminate the significant amount of redundant computation.
Tiny Machine Learning (TinyML) TinyML has gained significant attention in recent years. Techniques such as model compression (9; 10), pruning (11; 12; 13; 14; 15) and quantization (11; 16; 17; 18; 19; 20; 21; 22), are employed to reduce model size and complexity while maintaining accuracy. Existing frameworks like TFLM (3) and microTVM (4) optimize the execution time of TinyML models on resource-constrained devices by utilizing CMSIS-NN (6) and CMix-NN (23) libraries for kernel implementation. MCUNet (1) further optimizes kernel implementation through vectorization, loop reordering, loop unrolling, and data layout transformation, resulting in more efficient performance.
Memory-Efficient Inference The peak usage of SRAM memory is a critical factor in deploying TinyML models on microcontrollers due to the tight SRAM constraints in MCUs. However, existing works (3; 24; 25) employ per-layer inference, which restricts model deployment on severely memory-constrained MCUs. Several approaches have been proposed to address this issue in different ways. Some methods rearrange operators (26; 27) or use tensor memory planners (28) to save memory usage. Nonetheless, a single intermediate tensor may still be too large for the MCUs. As an alternative, models can be rewritten to reduce memory requirements (29; 30). However, this requires network modifications and may affect accuracy, reducing adaptability. MCUNetV2 (2) and TVM's cascade scheduling (31) address this problem by fusing multiple layers and introducing redundant computation. Prior work (32; 33; 34; 35; 4) also tackles similar problems as MCUNetv2 (2) in the ML inference. However, too many fused layers increase the amount of redundant computation.
Remove Recomputations To address recomputations introduced by the aforementioned approach, the main idea is to cache the computed result and reuse it when necessary. Halide (35) introduces sliding window optimization to remove recomputation, but it lacks flexibility in adjusting temporary buffers to reduce memory requirements across different dimensions. Prior work such as Fused-layer CNN accelerators (36) and streaming methods (37; 38; 39; 40; 41) offer solutions to reduce execution time in the data streaming manner, but none of them provides a method to mix 1D and 2D data streaming. Hence, our proposed StreamNet builds upon and extends existing streaming approaches by introducing a hybrid method that mixes different dimensional streaming to form 1D and 2D streaming techniques, skips the redundant computation shown in patch-based inference (2), and offers an auto-tuning framework to select StreamNet configurations.
Deploying TinyML models exposes challenges on MCUs with the stringent memory and power constraints. Unlike layer-wise inference, patch-based inference significantly decreases the usage of the SRAM memory and helps the deployment of TinyML models on MCUs. However, the redundant computation of patch-based inference enormously raises the latency of Machine Learning inference. Thus, this work proposes StreamNet that aims to eliminate such a performance bottleneck of patch-based inference. StreamNet designs 1D and 2D streaming processing to significantly remove the performance overhead of patch-based inference. Thus, StreamNet-2D achieves a geometric-mean speedup of 7.3X over patch-based inference across 10 TinyML models. Furthermore, StreamNet also skips the unnecessary computation for the padding data. Finally, StreamNet provides an auto-tuning framework to automatically search for the network that can satisfy the memory constraint of MCUs while achieving better performance over the existing patch-based inference.
The authors gratefully acknowledge the support from the National Science and Technology Council in Taiwan under grant number 111-2221-E-A49-131-MY3
[35] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, F. Durand, and S. Amarasinghe, “Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines,” *Acm Sigplan Notices*, vol. 48, no. 6, pp. 519–530, 2013.
[36] M. Alwani, H. Chen, M. Ferdman, and P. Milder, "Fused-layer cnn accelerators," in 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 1-12, IEEE, 2016.
[37] O. Rybakov, N. Kononenko, N. Subrahmanya, M. Visontai, and S. Laurenzo, “Streaming keyword spotting on mobile devices,” arXiv preprint arXiv:2005.06720, 2020.
[38] A. Coucke, M. Chlieh, T. Gisselbrecht, D. Leroy, M. Poumeyrol, and T. Lavril, "Efficient keyword spotting using dilated convolutions and gating," in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6351-6355, IEEE, 2019.
[39] Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Y. Wu, R. Pang, et al., "Streaming end-to-end speech recognition for mobile devices," in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6381-6385, IEEE, 2019.
[40] X. Chen, Y. Wu, Z. Wang, S. Liu, and J. Li, “Developing real-time streaming transformer transducer for speech recognition on large-scale dataset,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5904-5908, IEEE, 2021.
[41] Y. Wang, H. Lv, D. Povey, L. Xie, and S. Khudanpur, “Wake word detection with streaming transformers,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5864–5868, IEEE, 2021.