msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML
Authors: Zhaolan Huang
Emmanuel Baccelli
Venue: NeurIPS 2025 poster ()
Keywords: TinyML
DNN
Fusion
Microcontroller
Random access memory
Primary area: infrastructure
TL;DR: msf-CNN efficiently uses patch-based fusion to optimize CNN inference execution on MCUs, reducing RAM usage by up to 50% compared to prior methods while offering greater design flexibility.

Abstract: AI spans from large language models to tiny models running on microcontrollers (MCUs). Extremely memory-efficient model architectures are decisive to fit within an MCU's tiny memory budget e.g., 128kB of RAM. However, inference latency must remain small to fit real-time constraints. An approach to tackle this is *patch-based fusion*, which aims to optimize data flows across neural network layers. In this paper, we introduce *msf-CNN*, a novel technique that efficiently finds optimal fusion settings for convolutional neural networks (CNNs) by walking through the fusion solution space represented as a directed acyclic graph. Compared to previous work on CNN fusion for MCUs, msf-CNN identifies a wider set of solutions. We published an implementation of msf-CNN running on various microcontrollers (ARM Cortex-M, RISC-V, ESP32).  We show that msf-CNN can achieve inference using 50% less RAM compared to the prior art (MCUNetV2 and StreamNet). We thus demonstrate how msf-CNN offers additional flexibility for system designers.

msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML
Zhaolan Huang * and Emmanuel Baccelli 1,
Freie Universität Berlin
Einstein Center Digital Future
IInii, France
Abstract
AI spans from large language models to tiny models running on microcontrollers (MCUs). Extremely memory-efficient model architectures are decisive to fit within an MCU's tiny memory budget e.g., 128KB RAM. However, inference latency must remain small to fit real-time constraints. An approach to tackle this is patch- based fusion, which aims to optimize data flows across neural network layers. this paper, introduce msf-CNN, novel technique that efficiently finds optimal fusion settings for convolutional neural networks (CNNs) by walking through the fusion solution space represented directed acyclic graph. Compared previous work on CNN fusion for MCUs, msf-CNN identifies a wider set of solutions. We published implementation of msf-CNN running various microcontrollers (ARM Cortex-M, RISC-V, V, ESP32). We show that msf-CNN can achieve inference using 50% less RAM compared to the prior (MCUNetV2 and StreamNet).. We thus demonstrate how msf-CNN offers additional flexibility for system designers.
Introduction
Artificial Intelligence of Things (AloT) is a domain aiming embed AI in the smallest networked devices [T]. As such AloT is pushing the miniaturization Deep Neural Networks (DNNs) to hardware, which enables various applications the edge of the network.
fit microcontroller-basee Use-cases include vision/audio recognition, environmental monitoring, personalized medical care, etc. However, imbalance between the increasing resource requirements of DNNs and the very limited computation capacity (CPU in MHz) and memory resource of Microcontroller Units (MCUs) remains Internet of Things (IoT) devices. For instance, in RFC7228 (5), billions of 1oT devices are resource-constrained devices, with Random Access Memory (RAM) smaller than 50 KiB, and Flash memory smaller than 250 KiB. On the other hand, even a single convolutional layer in quantized ResNet-34 [2] [4] consumes around 414.72 KiB in RAM. This example highlights the huge gap between memory budgets IoT devices and RAM usage of DNNs.
challenge in deploying DNNs
described
technique aimed decreasing this gap is patch- patch-based layer fusion introduced in 22. Initially targeting FPGAs, patch-based fusion reduces off-chip Dynamic RAM (DRAM) requirements and communication bus transfer costs for inference with CNNs. Fusion is great for low-memory de- vices because it can save up to 95% of RAM usage. Moreover, Fusion decouples input size from memory usage, allowing for larger input. Recent work has thus explored the of fusion MCUs, for example, to improve the memory consumption of the first few convolutional layers of MobileNetV2 (26).
"Corresponding Author. E-mail: whaolan.huang @ fu-herlin.de
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Nevertheless, we observe that significant issues linger on MCUs. First, intermediate feature maps inside the fusing block incur high (re)compute cost. Second, input size limits hamper many use- cases such as medical image processing, sequence time series analysis (e.g. audio application), etc. Third, implementations of fusion on MCUs have so far been very hardware-specific (e-g. bound to the ARM-Cortex-MF instruction set) and model-specific (e.g. bound to CNN mobile inverted blocks).
Contributions With the goal of improving on the above issues, we report on following work:
We propose msf-CNN, fusion-based approach achieve ultra-low RAM footprint of neural network inference and we open-source its implementation 2
We formulate the problem of finding optimal fusion settings that minimize peak RAM usage or compute cost of neural networks as a variant shortest path problem.
We provide graph models representing multi-stage fusion neural networks, which encode peak RAM usage and compute cost of single and fused layers.
We designed a pruning strategy to squeeze the search space and use graph- based algorithm to find solutions in reasonable time complexity (from O(2N-2) to polynomial time).
We improved global pooling and dense operators further squeeze RAM usage without compute overhead.
We released preliminary evaluation results on MCU-based loT boards. We compared common CNN, StreamNet, MCUNetV2 and msf-CNN on a variety of microcontrollers. We show that msf-CNN allows new trade-off between memory saving and compute overhead.
While our main focus is on microcontrollers, msf-CNN is not limited to them. Its cost estimator and C-code backend also support general CPU platforms (e.g., x86, Cortex-A), enabling broader use such as memory-optimize msf-CNN can be extended to accelerators (GPUs, FPGAs, ASICs). We leave non-MCU experiments to future work to maintain focus on devices with tighter RAM constraints.
cloud inference. Moreover, with appropriate cost models and backends,
2 Background
Patch-based Fusion for DNN on FPGA & GPU - Patch-based fusion was initially proposed in 2 as fusion scheme for Convolutional Neural Network (CNN) deployed on Field Programmable Gate Array (FPGA) to reduce the off-chip DRAM usage and I/O overhead. Instead of computing the complete feature maps for each layer, it fuses convolutional layers into single block (pyramid structure) and computes only one or a few output elements. This approach requires only small portions (tiles) of the feature maps loaded onto DRAM. However, the reduction of RAM is at the cost of re-computing the overlapped elements in feature maps required by adjacent fused layers. DeFiNES (28), another fusion framework, explored different cache strategies within fused layers to alleviate the re-computation issue. (Fully-recompute, H-Cached & V-recompute, and Fully-cache). Fully-recompute eliminates caching entirely, requiring all overlapping input tensor elements to be recalculated; H-cached & V-recompute caches elements along the horizontal axis while recomputing vertical overlaps; and Fully-cache retains all overlapping elements in memory. These approaches illustrate critical trade-off-enhanced caching progressively reduces compute redundancy but proportionally increases RAM usage, with cached element quantity inversely correlating to compute overhead and directly scaling with memory demands. Additional work has also applied fusion on GPUs, for instance (31] used it for cancer detection in medical pictures. Note that patch-based fusion is fundamentally different from kernel fusion techniques. We elaborate on this distinction and its implications in Appendix[A) A
Patch-based Fusion MCUs Work on MCUNetV2 26 has applied fusion on MobileNetV2 to reduce the peak RAM usage. It revealed that layers at the head of the model dominate the RAM usage. Hence these layers were fused into one block to reduce RAM usage significantly. The recompute issue was mitigated by redistributing the receptive field, so the receptive field inside the fusion block was decreased and regained at later stage. Work StreamNet [46] introduced two-dimensional tensor cache to significantly reduce re-compute operations in a fusion block and applied brute force to search for optimal fusion position and cache depth. Nevertheless, no prior work explored the potential of multiple fusion blocks in CNNs.
2 Pleaase check ht https::///// httpe//hww...cm/aaaa//////////a.
2

---

Representing DNNs as Inverted Dataflow Graphs Dataflow graph have been widely used for modeling DNN, as pioneered by TensorFlow and PyTorch [0]. The data (tensor) flows alongside the directed edge between nodes which indicates the operations (convolution, pooling, addition, etc.) applied on the incoming edges (tensors). This representation shows the producer-consumer relations among operations and has great expressiveness and flexibility, enabling automatic differentiation and concurrent execution independent operations.
High-Level Idea
Inspired by the above previous works, msf-CNN aims to answer the following questions: (1) Where fuse and how determine the fusion position/depth? (2) Under specific resource constraints, how to find the optimal fusion settings?
As depicted in Figure Ta msf-CNN determines fusion settings (fusion position and depth), transforms layers accordingly into fusion blocks and rewrites global pooling and dense layers as their iterative implementation, which can further squeeze RAM usage without any computation overhead.
To guide us in doing so, we use inverted dataflow graphs to model CNNs, where tensors are represented as nodes, and operations are depicted as edges connecting them. On this graph, we encode into the edges the resource usage of the operations, and use additional edges to represent fusion blocks. This allows us to design graph-based strategies to find optimal solutions with lower computational complexity using proven graph algorithms.
Memory Efficient
Original
RAM Usages
Vo
Optimal (min-max): (es, ea. e.) KB
es
ConwiPnnlig
ConwPooling
VD
e
V1
e2
V2
e3
V3
e4
V4
ConwPooling
Conwolu- tion Block with Layer Cache
15
11 kB
Conu/Pooling
Fused
ConuiPnning
es
2 KB
ConwPooling
MAC Operations
ConwPooingg
Optimal (shortest): (ei, Ca, ei)
Rewrite
V3
es
GlohalPool ng/ Derse
Rerrave Pooling
ConwPooligg
40 M
45 M
e1
e2
e3
e.
V1
V2
V3
Va
Va
15
MLP
Cu
Optimal Path
Output
Output
M
(a) Overview
(b) Neural Network as DAG
Figure 1: (a) Overview of msf-CNN. The convolutional layers are fused into several fusion blocks based the optimal setting found by optimizer. We let global Pooling and dense layers compute the outputs iteratively to further squeeze RAM usage. (b) The neural network modeled as directed acyclic graph (DAG), Nodes v,. denote the tensors that are produced and consumed by the operators or possible fusion blocks. Edges cI...., ea represent individual operators, while edge cs és represent two candidate fusion blocks. Edges are annotated with the RAM usage and multiply accumulate (MAC) amounts of their corresponding operators and fusion blocks.
Problem Definitions & Assumptions
We aim to solve pair of dual optimization problems. Let x be the set of all possible configurations for fusion blocks. We define P1 as the problem minimizing peak RAM usage subject to computation cost limit:
min P(x, S
s.t.
F(, < Fmaz
(2)
3

---

where P is the peak RAM usage, and F is the computation overhead for inference under fusion setting S. relatively inference without fusion (thereafter denoted vanilla). The compute cost limit and RAM limit are annotated by maz and P manz respectively. Dually, we define P2 as the problem of minimizing computation cost subject to RAM footprint limit:
min F(,,S) P Pmaz
s.t. P(x, S)
Without loss of generality, we only discuss fusion blocks of convolutions. We assume a H-Caache scheme, which we chose to be a good trade-off between buffer size and recompute cost on MCUs.
In Appendix and Appendix C we further detail the analysis of the cache buffer size and the amount of MAC operations.
DNN Graph Representation & Formulation
We interpret the optimization problems described in Dy modeling the DNN as data-nodes graph. We transform the problem as shortest path problem [35] and use off-the-shelf graph algorithms to find a solution that minimizes the peak memory usage as well as compute cost during inference regarding specified constraints.
5.1 DNN Representation
As described in Section 2 we model a DNN as a DAG G = (V, with data nodes Vo. Un representing input/output tensors of consecutive layers and edges c1,..., Em that represent single layers or fusion blocks. Each edge is also encoded with resource requirements by layer or fusion block. Specifically, the first (vo) and the last node (vn) are the input and output tensor of the neural network, respectively.
In general, the edge represents the input/output relation of nodes and also indicates the fusion depth inside the neural network. For example, an edge that connects consecutive vertices e = on + Un+1 is a single layer that consumes v, input tensor and outputs tensor Un+1, while an edge jumps over multiple vertices e = on + Un-mm m >> 1 represents a fusion block with m layers. Each complete compute path from by on represents a fusion setting S.
typical example depicted in Figure Tb explains how to use DAG for representing a simple neural network. Tensors are transformed into nodes, operators and fusion blocks are edges. Edges are encoded with RAM usages and MAC amounts of their corresponding operators. Hence, the problem is transformed to find an optimal path from the input node to the output node of the graph.
5.2 Encoding RAM Usage
We first calculate the RAM usages Pa, of all single layers and all possible fusion blocks inside the neural network by
Pa. = O Buf
where / and O are the size of input and output tensor, respectively. Buf represents the cache buffer size of the fusion block, which is determined by the chosen cache scheme. In this work it is given in Equation [I]. Trivially, for non-fused layers Buf is always set to zero since no fusion cache is needed.
Thereafter, the calculated RAM usages are attached to the corresponding edges for further analysis. For complete compute path contains edges 5 = (Cig...., Ci,, we can then calculate the overall peak RAM usage Ps by
Ps
max Pat j=1...
5.3 Encoding Compute Cost
The encoding steps of compute cost are similar to encoding peak memory usage. Here we use MAC operations as the indicator of compute cost. In this paper, the MAC amount of fusion block given in Equation [4] and Equation 15

---

After attaching the calculated MACs to the edges, the total compute cost of a complete compute path
C3
Cet
Therefore, the compute overhead factor F' representing the ratio of the MAC amount after fusion to the vanilla, common one without fusion is expressed as F = DSs/Connlla.. For the constraints Equation (2) users can set a maximum compute overhead factor Fmax expressed as Fmax = Cmaz/ / Cvanitla. In the following sections, we will discuss several graph-based algorithms solve the optimization problem.
Searching for Optimal Fusion Settings
After building an inverted dataflow graph of a DNN with all possible fusion combinations (edges), the two dual problems are indeed transformed into classic graph problems: finding an optimal complete compute path from the input tensor node 01 the output tensor node 2, under specific constraints.
Impact of Search Space Size - we consider the unconstrained optimization, the solution is trivial: the setee--uuur---s-e---eeeeeeee-ee-
shortest path, which can be found by classical graph algorithm like Dijkstra's 's 8 with the time complexity of O(E log (V)). However, when considering the constraints, it is necessary first to explore all possible complete compute paths that meet the conditions, which can potentially explode the complexity to O - [32] in the worst case. Hence, we need smarter strategy squeeze the search space and avoid horrendous complexity.
Note that such shortest path computations do not take place on the microcontroller at runtime. Instead, they are computed offline, PC, which expands the realm of what can be assessed as bearable computation.
6.1 Problem Pl: Minimizing Peak RAM Usage
The unconstrained optimization is to find a complete compute path with minimal peak RAM usage, which is equivalent to finding the path that minimizes the maximum weight edges (minimax path problem). As mentioned above, this can be solved by modified Dijkstra algorithm. An example path with minimal peak RAM usage is presented in Figure [b]
For the constraint of compute cost limit (Equation (2). the pruning strategy needs co-design with its optimization problem (Equation (T). We noticed that all possible peak RAM usages have already been encoded into the edges. Therefore, the problem can be transformed into the following: we first construct a candidate solution set (Su, , S..... with
S arg min S), 5
(8)
G; == subgraph of G,-1, obtained by removing all edges in G with the maximal RAM usage,
(9)
Go
(10)
where C(G,, S) is the MAC amount of fusion setting S in graph G.. The candidate solution St can be obtained by applying the shortest path algorithm. We then filter the candidate solutions to find those that satisfy the constraints and select the one with the smallest RAM usage as the optimal solution.
In this way. we avoid constructing a search space with complexity of O(2V-2). Instead, we iteratively eliminate subgraphs and solve for candidate solutions, reducing the complexity to OV33). For most deep neural networks running MCUs, this process can be done in few seconds.
6.2 Problem P2: Minimizing Compute Cost
We first discuss the unconstrained variant, which is identical to Pmaz oc. In this case, finding the solution is equivalent to finding the shortest complete compute path - the path with a minimal sum of MAC of the graph, which can be again solved by classical algorithm like Dijkstra's g Figure Tb shows example with optimal path marked in red.
5

---

When bringing back the constraint of RAM limit, the pruning step is simple: eliminating all edges with encoded RAM usage exceeding the limit. So, all paths in the graph will automatically fulfill the limitation.
6.3 Analytical Results
To explore the capability of these two dual optimizers, here we choose three variants of MobileNetV2 and MCUNet [3 [6] with different scales for the pilot study: MobileNetV2 with width multiplier 0.35 and input size of 144 x 144 x 3 (MBV2-w0.355 MCUNetVV-VVW-5hf with input size of 80 80 (MN2-vvw5),
with input size of 176 176 (MN2- 320K). For optimizer of minimizing peak RAM usage, the maximal compute overhead factor ranges from 1.1 1.5 then jumps to Infinite, which represents unconstrained optimization. For optimizer of minimizing compute cost, the maximal peak RAM usage was set from 16 kB to 256 kB where each level represents popular RAM capacity of mainstream MCUs.
As shown in Table I both optimizers can indeed theoretically suppress the peak RAM usage without violating all preset constraints. The high RAM usage compression is achieved with increase of deep fusion blocks, thereby introducing a high compute overhead. The extreme cases lay on the unconstrained optimization minimizing the RAM usage by more than 90%, while reluctantly introducing 1.6x to 2.7x x of compute overhead. This is only suitable for time-intensive applications with high limited RAM budget.
the other hand, setting appropriate constraints can still lead to well-optimized configurations, with our tools offering flexibility to accommodate real-life scenarios. Under different thresholds on compute overhead factor or peak RAM usage, the solutions that optimizer found are all fulfill constraints and with RAM usage all lower than the vanilla, un-fused setting. In some cases, it is possible to compress RAM usage without incurring additional computational overhead. These pilot studies demonstrate the effectiveness of finding usable solutions under real-life constraints.
We also conducted a preliminary analysis of the heuristic strategy used MCUNetV2, which fuses only the early layers to minimize RAM usage. While this approach is simple and straightforward, tends to yield suboptimal fusion configurations. As shown in Table msf-CNN discovers better solutions and offers greater flexibility compared to the heuristic strategy. The analytical results were further validated by on-board experiments presented in Section
Table 1: Analytical results with msf-CNN under different constraints. Vanilla: un-fused models.
Heuristic: minimize RAM consumption by only fusing heading layers. SAA: Same as above.
Gray: msf-CNN beats heuristic.
MBV2-w0.35
MN2-vwW5
MN2-320K
Constraint
RAM (kB)
RAM (kB)
F
RAM (kB)
F
Vanilla Heuristic
194.44
1.00
96.00
1.00
309.76
1.00
32.08
1.59
24.00
1.56
90.31
3.25
1.1
67.91
1.10
32.79
1.04
190.10
1.04
1.2
(SAA)
26.13
1.11
186.74
1.19
1.3
21.29
1.30
17.76
1.30
186.03
1.25
Pl: Fmaz
1.4
15.34
1.38
13.38
1.35
156.67
1.37
1.5
(SAA)
(SAA)
94.18
1.45
Inf
7.89
1.68
12.00
1.96
42.64
2.69
16 kB
15.34
1.38
13.38
1.35
(No Solution)
32 kB
25.67
1.25
26.13
1.11
P2: Pmaz
64 kB
63.74
1.23
38.58
1.02
62.88
2.02
128 kB
83.07
1.02
89.60
1.00
94.18
1.45
256 kB
181.44
1.00
(SAA)
247.81
1.00
7 msf-CNN Implementation Details
We have implemented the msf-CNN fusion mechanism on top microTVM v0. 10.16.0 7 We use the TVM frontend to convert models into intermediate representation (IR), and rewrite the compute graph

---

and low-level routines of operators to fit the fusion settings. We used RIOT-ML [17 to benchmark the fused models: the models are transformed into C code using MVC and integrated in dedicated firmware leveraging common loT operating system (RIOT which we can run on various boards shown in Table
Sequential RAM Usage- - We have optimized the RAM usage of the global pooling and fully connected (Dense) layers. We observed that the outputs of these two basic blocks can be computed iteratively, and in most scenarios, their input dimensions are much larger than their output dimensions. As result, we can temporally divide the input and sequentially process through the iterative global pooling dense layers, which further minimizes memory usage. If their upstream is fusion block, this perfectly matches the feature of temporally split inputs, enabling them be fused seamlessly.
Iterative Computation of Global Pooling As illustrated in Figure 2 standard glohal pooling requires that all elements of the input tensor stored in RAM. In approach, the global pooling layer receives one or few input elements at each step and iteratively updates the result. For a 7 x global pooling layer, this allows us to compress the RAM usage 2% of the original size, without introducing any redundant computations or computation overhead.
Iterative Computation of Dense Layer- We noted that the matrix multiplication in dense layers can be implemented by splitting the input vector into individual elements, multiplying each element with its corresponding weight column, and iteratively summing the results, as shown in Figure 3 Unlike the original approach, which requires the complete input tensor, this method processes only one element of the input tensor per iteration. For (024-+250 dense layer, this approach compresses memory usage to 20% the original.
Common Global Pooling
Iterative
Common Dense
Iterative Dense
Global Pooling
Flatten
IterPool
Pool
+..
(Iter)Pool E (min, max, average)
Elements in memory
Elements in memory
IterPool
n-th Hte Iteration
Figure Comparison of common and iterative dense layer. The columns of the weight matrix are denoted as w(n).
Figure 2: Comparison of common and iterative global pooling.
Experiments on Microcontrollers
In this section we report on experiments running msf-CNN on various MCUs, aiming to validate both the correctness of our optimization strategies and their versatility when applied on diverse hardware.
More concretely, we measured peak RAM usage and compute latency based on the fusion settings in Section 6.3] as reported in the following, As shown in Table 2 we carried out our experiments on the relevant 32-b bit microcontroller architectures: Arm Cortex-N -M, Espressif Xtensa, and RISC-V. For our model zoo, we chose MB VBV2-w0.35, MN2-vww5 and MN2-320K as they are good representatives of backbones for applications in AloT 33 as also used in prior works 226 46)
We compare msf-CNI
performance to the closest related work: MCUNetV2 26 and StreamNet-2D
[6), more simply
denoted StreamNet in the following.
8.1 Minimal Peak RAM Usage
First, we evaluated solutions P1 while relaxing Equation (2). i.e. the fusion settings with minimum peak RAM usage, without considering time constraint. Results are shown in Table We observe that, compared to prior art (StreamNett-2D and MCUNeIV2), msf-CNN can further reduce the peak RAM usage by 65% to 87%. We could even deploy the MBV2-w0.33. model onto the SiFive board that provides only 16 kB RAM (!). However, achieving this high compression ratio comes at the expense of increased computational latency, which measured in Table Interestingly, while
7

---

Table 2: The different microcontrollers & boards used in our experiments. The RAM and Flash
capacity are presented in kB.
Board
MCU
Core
RAM
Flash
Nucleo-f7677i
STM32F76721
Cortex-M7 @ 216 MHz
512
2048
Stm321744g-dissco
STM32F746NG
Cortex-M7 @ 216 MHz
320
1024
Nucleo-f4122g
STM32F4122G
Cortex-M4 @ 100 MHz
256
1024
esp32s3-devkit
ESP32-S3-WROOM-ING
Xtensa @ 240 MHz
512
8192
esp32c3-devkit
ESP32c3-I-MINII-AN
RISC-V @ 160 MHz
384
4096
hifivelb
SiFive FE310-G002
RISC-V @ 320 MHz
16
4096
clock frequency plays decisive role, MCU architecture also have crucial effect, for larger models. For instance, notice latency with Xtensa esp32s3 at 240MHz versus RISC-V esp32c3 at 160 MHz, for the MN2-320K model (in Table Nevertheless, measured that latency increases 2x to 5x compared to vanilla (non-fused) CNN. Hence, such minimal RAM settings are only suitable for latency-tolerant applications the smallest devices.
Table 3: Minimal peak RAM use, measured in
Table 4: Inference execution time, measured in ms, with msf-CNN tuned with minimal peak RAM. (OOM: Out-of-Memory)
kB. (Vanilla: un-fused model)
MBV2
MN2
MN2
(Fusion) -w0.35
-www5
-320K
MBV2
MN2
MN2
(MCU)
Vanilla 194.44
96.00
309.76
-w0.35
-wWW5
-320K
MCUNetva
63.00
45.00
215.00
stm321767
1996.8
1723.0
19329,9
StreamNet
66.00
44.00
208.00
(vs. vanilla)
2.5x
3.4x
4.4x
msf-CNN
8.56
15.37
51.16
stm321746
1379.6
1727.5
16261.9
stm32f412
5270.1
4943.4
56979.0
esp32s3
6748.2
5974.1
76763.6
csp32c3
6792.7
6248.9
73713.8
SiFive
10000.0
OOM
OOM
8.2 Impact of RAM Budget Limit
As shown in Figure and T the measured peak RAM usage consistently obeys the given constraints, thereby validating the correctness of the optimizer and corroborating our analytical results. Based on these, we observe that higher RAM budgets result shorter compute latency for the optimal fusion configurations identified by msf-CNN. This is because the optimizer tends to favor configurations with either no fusion shallow fusion depths, which correspond higher peak RAM usage but lower computational costs.
For the MBV2-w0.. and MN2-vww5 models, method outperforms MCUNetV2 when the RAM limit is set to 32kB and 64kB. Although our method does not surpass StreamNet-22 across the board, msf-CNN does demonstrate its flexibility, enabling users select the optimal fusion configuration under varying memory budgets.
8.3 Impact of Computation Cost Limit
When capping computation cost as a constraint, the relation between compute latency and peak RAM usage is consistent (dual) with the previous section, such that higher compute overhead budgets result in longer compute latency and smaller peak RAM usage. We also observe that the ratio F measuring the overhead compared to vanilla CNN (no fusion) is bigger than the Fmaz we for. This discrepancy comes from the fact that the optimizer computes the amount of MAC operations. whereas the full latency includes not only MAC operations but also I/O delays. mainstream MCUs, model weights are stored in Flash rather than RAM, which introduces substantial additional latency during read operations, thereby contributing to higher compute latency. Specifically, when recomputation occurs, the weights must be refetched from flash memory, which could disrupt cache hits and lead to increased overall latency. Despite this discrepancy, our method still generates fusion configurations for the
and MN2-vww5 models that outperform MCUNetV2.

---

MBV2-w0..
MN2-Vww5
MN2-320K
2.0
20
Latency (s)
1.5
15
1.0
10
0.5
5
0.0
0
0
100
200
0
50
100
0
200
400
RAM (kB)
RAM (kB)
RAM (kB)
Vanilla
StreamNet
P2
P2 CMSIS
MCUNetv2
P1
P1 CMSIS
Figure 4: Trade-off between RAM and latency of different optimal fusion settings on Nucleo-f767zi. P1: Minimize RAM s.t. compute cost limit. P2: Minimize compute cost s.t. RAM limit. For more detailed results, please refer to Table 6n Appendix F
Particularly for memory-sensitive but time-insensitive applications, we can set the constraint Fmaz to infinity, thereby obtaining novel fusion configurations with minimal RAM usage.
msf-CNN underperforms StreamNet because our implementation does not use bardware-dependee libraries or acceleration instructions, while StreamNet is optimized for ARM platforms via the CMSIS [22 and employs 2-D cache (ours uses I-D), which further mitigates recomputation. To enable fairer comparison, we added an optional CMSIS backend to msf-CNN for ARM devices The results show msf-CNN's Pareto curves of CMSIS variants are much closer to StreamNet's. We are also extending msf-CNN with 2-D cache support to provide greater flexibility across hardware platforms.
and conducted additional measurements, as shown in Figure 4
Discussion
Our experiments demonstrate msf-CNNG capability optimize resource usage with diverse CNN models, under user-specified constraints emphasizing either compute latency or RAM footprint. Furthermore, msf-CNN generates code that is deployable across diverse MCUs' ISAs. Users thus produce optimal CNN fusion configurations tailored to specific industrial hardware requirements. However, some limitations remain, on which our future work will focus next:
Parameter Space- The current optimization scope is limited to fusion block positioning and depth selection, with the number of output elements per iteration fixed at one. This parameter significantly impacts both memory footprint and compute overhead, which warrants further exploration.
Caching Paradigm The search space currently incorporates only the H-cache paradigm. Future implementations should integrate alternative caching strategies to enhance optimization flexibility.
Neural Network Architecture - The work currently focuses exclusively on convolutional neural network architectures (CNNs). The analysis of other prevalent structures, particularly attention mechanisms and recurrent neural networks (RNNs), remains an open research direction. Please check Appendix G for the current state of extension.
10 Related Work
Machine Learning Compilers for MCUs Compilers such as Tensor Virtual Machine (TVM) 7)
IREE 39, FlexTensor [48). and Buddy 44 offer automated transpilation and compilation for models produced by major Machine Learning (MI.) frameworks, including TensorFlow and Py Torch.. Other
prior work such RIOT-ML [17] combine small general-purposs OS with microTVM (extension of

---

TVM orienting to MCU), for comprehensive support for ML frameworks and operator implementation on divers MCUs. However, none of the above tools provide CNN fusion optimization mechanisms, in contrast to msf-CNN.
Efficient Neural Network Structure For models to operate low-power IoT devices, they must be compact and computationally efficient. Studies have demonstrated the use of lightweight CNNs for speech recognition and age classification [27]. water leakage detection [3] fall detection for the elderly [9] and other tasks [8 [50] Tiny vision transformers have also been employed for
classification tasks in various studies 20
Besides handcrafting a lightweight structure 16 34 [5] also re-designed the basic blocks
by reducing layer number or kernel size, people 19
38
to replace common convolutions for lower memory footprint and compute latency.
Tiny Neural Architecture Search (NAS) This technique is employed to automatically search for model structures with optimal accuracy under the constraints of memory, flash footprint and compute latency. TinyNAS (25.. ,NNS (24] and the Once-for-All Network [6] leverage Neural Architecture Search (NAS) design CNNs with exceptionally small memory requirements for MCUs. The resulting networks require only few hundred kilobytes of RAM for execution. However, contrary to msf-CNN, these methods necessitate retraining or fine-tuning of pre-existing networks.
Memory Optimization for CNN layers- - Memory optimization strategies can be broadly catego- rized into scheduling-basee based and fusion- based methods. Scheduling-base based methods, such as those implemented in frameworks like Ansor [47), vMCU [4). MoDEL [37] and TinyEngine [26]. focus on the efficient reuse of memory pools minimize peak memory usage by leveraging the different lifetimes of inter- and intra-layer tensors. Although both methods achieve peak memory reduction exceeding 50%, they still generate complete output tensor for each layer. This requirement remains problematic for low-power MCUs with limited RAM. Prior work on fusion was covered in Section 2 Contrary to msf-CNN, these methods do not fully exploit the potential of multiple fusion blocks.
11 Conclusion
Convolutional neural networks (CNNs) must not only execute in the cloud or on edge computing gateways, but also on the smaller, more energy-efficient microcontroller-baase part in our cyber-physical systems. Microcontrollers pose a great challenge for CNNs regarding the joint optimization of RAM memory consumption and inference latency. In this context, we presented msf-CNN, a technique and heuristics able to identify pools of practical patch-based fusion optimizations for CNN inference, which jointly satisfy memory and latency constraints. Compared to previous work on CNN fusion for microcontrollerr. msf-CNN identifies a wider set of applicable solutions, on much more diverse hardware. Our experimental evaluation using the open source implementation we provide for common microcontrollers (ARM Cortex-M, RISC-V, and ESP32) show that msf-CNN can achieve inference with less than 50% the peak RAM usage compared to state-of-tte the-art. art. As such, msf-CNN provides a new level flexibility for embedded system designers, which can now better tune the trade-off between peak RAM and model inference latency various hardware.
devices which take
Acknowledgment
The authors thank Adrien Tousnakhoff for useful discussions. Work contributing to these results was partially funded by the ANR France 2030 Programme (ANR-22-PTCC-0001 and ANR-22-PEFT: 0007).
References
[1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A.,, Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. [TTessoFllo]] J: a system for (Large-Scale) machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp. 265-283, 2016.
[2] Alwani, M., Chen, H., Ferdman, M., and Milder, Fused-layer CNN accelerators. 2016 49th Annual IEEE/ACM International Symposium on Microarchiteecture (MICRO), pp. 1-12, Taipei,
10

---

Taiwan, October 2016. IEEE. ISBN 978-1-5090-3508-3, doi: 10.1109/MICRO.221..77888
URL httt::/ieeeepplore/ ieee. org/document/7788725/
[3] Atanane, O., Mourhir, A., Benamar, N., and Zennaro, M. Smart Buildings: Water Leakage Detection Using TinyML. Sensors, 23(22):9210, November 2023. ISSN 1424-8220. doi: 10.3390/s23222210,
httL:htps:////
wmpi. com/ 1424 /2/0/22/22//22 2////22222222/// /210
[4] Baccelli, E., Gündogan, C., Hahm, O., Kietzmann, Lenders, M. S., Petersen, H., Schleiser, K., Schmidt, T. C., and Wählisch, M. RIOT: An Open Source Operating System for Low-end Embedded Devices in the IoT. IEEE Internet of Things Journal, 44):4442-4440,, 2018.
[5] Bormann, C. et al. Terminology for Constrained-Node Networks. RFC 7228, May 2014.
[6] Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S. Once-for-all: Train network and specialize it for efficient deployment. arXiv preprint
09791, 2019.
[7] Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Cowan, M., Shen, H., Wang, L., Hu, Y., Ceze, L.,, et al. Tvm: An automated end-to-end optimizing compiler for deep learning. arXiv preprint 1802:0802004999 2018.
[8] Dijkstra, E. W. A note on two problems connexion with graphs. In Edsger Wybe Dijkstra: his life, work, and legacy, pp. 287-290. 2022.
[9] Fang, K., Xu, Z., Li, Y., and Pan, J. A Fall Detection using Sound Technology Based on TinyML. 2021 1 11th International Conference on Information Technology Medicine and Education (ITME), pp. 222-225, Wuyishan, Fujian, China, November 2021. IEEE. ISBN 978-1-66540-679-6. doi: 10. 109/TT 212.390011.222.000555 URL httpps//ieeeeepoorr
leee dww.mentccmttt/////m/m aeee.org/ddccumnn/99555555 99700588/
[10] Fredman, M. L. and Tarjan, R. E. Fibonacci heaps and their uses in improved network optimization algorithms. Journal of the ACM (JACM), 34(3):596-615, 1987.
[11] Ghosh, A., Chakraborty, D., and Law, A. Artificial intelligence in internet things. CAAI Transactions on Intelligence Technology, 3(4):208-21) -18, 2018.
[12] Guo, M.-H., Lu, C.-Z., Hou, Q.. Liu, Z., Cheng, M.-M., and Hu, S.-M. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in neural information processing systems, 35: 1111110115555 1156, 2022.
[13] Guo, M.-H., Lu, C.-z., Liu, Z.-N., Cheng, M.-M., and Hu, S.-M. Visual attention network. Computational visual media, 9(4::733-752, 2023.
[14] He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[15] Howard, A., Sandler, M., Chu. G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q V., and Adam, H. Searching for MobileNetV3. pp. 1314 1324, URL htttp:://ppeaceee Searching for www.eeeeeeet__ecooooo.0... 2o1ppp______9ppppp.pppr.. paper. peaper.hmml htmll
thecvf
http://ottettccooeeee 2ttp222199hhmmme/ /Howard
[16] Howard, A. G., Zhu, M., Chen, B., Kalenichenko,, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. MobileNets: Efficient convolutional neural networks for mobile vision applications,
URI. htt.:httt//aaaa http://xxivv..rr/ waarrg/bb/177044. 04861
[17] Huang, Z., Zandberg, K., Schleiser, K., and Baccelli, RIOT-ML: toolkit for over-the-air secure updates and performance evaluation of TinyML models. Annals of Telecommunications, pp. 1-15, 2024.
[18] Hussain, M. S. and Haque, M. A. SwishNet: A Fast Convolutional Neural Network for Speech, Music and Noise Classification and Segmentation. 2018. doi: 10.44550/ARXX.11120000449 URL htt.:ttpps//arxx.oo///aaaa////
00149
[19] landola, N., Han, S., Moskewiez, M. W., Ashraf, K., Dally, W.J., and Keutzer, SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size.
11

---

[20] Jinyang Yu, Zikai Song, Jiahao Ji, Lixian Zhu, Kele Xu. Qian, K., Dou. Y., and Hu, B. Tiny Audio Spectrogram Transformer: Mobilevit for Low- Com-olxxyy Acoustic Scene Classification with Decoupled Knowledge Distillation. 2023. doi: 10.13140/RG..22224400111
URL
htttp::/rgdoi.nnn net/10. 101111..11//RGGG. 2. 24001. 12646
[21] Koonce, B. Resnet 34. Convolutional neural networks with swift for tensorflow: image recognition and dataset calegorization, pp. 51-61, 2021.
[22] Lai, L.,, Suda, N., and Chandra, V. Cmsis-nn: Efficient neural network kernels for arm cortex-m cpus. arXiv preprint arXiv: 1801.06601, 2018.
[23] Liang, Y., Wang. Z., Xu, X., Tang, Y., Zhou, J., and Lu, J. MCUFormer: Deploying Vision Transformers on Microcontrollers with Limited Memory. 2023. doi: 1044850/ARXIV.23100 16898. URL httpttttta/aaaat org. abs/ 2310. 16898
[24] Liberis, E., Dudziak, E.,, and Lane, D. pnas: Constrained neural architecture search for microcontroollers In Proceedings of the 1st Workshop Machine Learning and Systems, pp. 70-79, 2021.
[25] Lin, J., Chen, W.-M,, Lin, Y., Gan, C., Han, S., et al. Mcunet: Tiny deep learning on iot devices. Advances in neural information processing systems, 33:11711-11722 2020.
[26] Lin, J.. Chen, W.-M., Cai, H., Gan, C., and Han, S. Memory--fficient Patch- Inference for Tiny Deep Learning. In Advances in Neural Information Processing Systems, volume 34, pp. 2346 2358. Curran Associates, Inc., 2021. URL http:::://roccceiiinggg www.ss.....ppp.p cc/ paper/ http://////.ess./cccccc/////aaa.
html
[27] Maayah, M., Abunada, A., Al-Janahi, K., Ahmed, M. E., and Qadir, J. LimitAccess: on- device TinyML based robust speech recognition and age classification. Discover Artificial Intelligence, 3(1):8, February 2023. ISSN 2731-0809. doi: 10. attps://1ink.spriiing
URL
10. 1007 0000000//44133--0 0220040006-000000---2----
(28) Mei, L.., Goetschalekx, K., Symons, A., and Verhelst, M. Defines: Enabling fast exploration of the depth-first scheduling space for dnn accelerators through analytical modeling. In 2023 IEEE International Symposium on Hligh-Performanc Computer Architecture (HPCA), pp. 570-583. IEEE, 2023.
[29] Niu, W., Guan, J., Wang, Y., Agrawal, G., and Ren, B. Dnnfusion: accelerating deep neural networks execution with advanced operator fusion. Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pp. 883-898, 2021.
[30] Paszke, A., Gross, S.. Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.. Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, tigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[31] Pinckaers, H., van Ginneken, B., and Litjens, G. Streaming Convolutional Neural Networks for EnddttoEEnd Learning With Multi-Megapixx Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1581-1590 March 2022. ISSN 1939-3533 3539. doi: 10.1 109/TTAMI 2020.301 33639366..... URI. https: //ipp/////eeeeeprrr/ ieee. org/ /dccmmen Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
[32] Robert, S. Algorithms in c, part 5: Graph algorithms, 2002.
[33] Saha, S. S., Sandha, S. S., and Srivastava, M. Machine learning for microcontroller-class hardware: A review. IEEE Sensors Journal, (122)):1332-21399) 2022.
[34] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mo- bileNetV2: Inverted residuals and linear bottlenecks. pp. 4520-4520. URL. https //openaccess. thecvf com/ content_cttt cvpr 2018/html/Sandler Mobiileettt.1 Ceaprrtee_RRiiduuls_e_CCCC CVWR_22RR11188 https_221_ppap_e__ae1_e_
[35] Sedgewick, R. Algorithms c, part graph algorithms, third edition. Addison- -Weley Professional, third edition, 2001. ISBN 9780768685329.
12

---

[36] Shen, Z., Zhang, M., Zhao, H., Yi, S., and Li, H. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3531-3539, 2021.
[37] Steiner, B., Elhoushi, M., Kahn, J., and Hegarty, J. Model: memory optimizations for deep learning. In International Conference on Machine Learning, pp.
PMLR, 2023.
[38] Tan, M. and Le, Q. V. Efficienteet: Rethinking model scaling for convolutional neural networks. Uttphtp::/ http:://axxiv..//o wrg.:iiiioorrorooo httgg//bb//aa/ 1905. 11946
[39] The IREE Authors. IREE, September 2019. URL htLLAatpp////ihbb..o//rrreeeee/p
[40] Wang. G., Lin, Y., and Yi, W. Kernel fusion: An effective method for better power efficiency on multithreaded gpu. In 2010 IEEE/ACM Int't Conference on Green Computing and Commu- nications Int'l Conference on Cyber, Physical and Social Computing, pp. 344-350. IEEE, 2010.
[41] Wyatt, S., Elliott, D., Aravamudan, A., Otero, C. E., Otero, L. D., Anagnostopoulos, G. C., Smith, A. O., Peter, A. M., Jones, W., Leung, S., and Lam, E. Environmental Sound Classifica- tion with Tiny Transformers in Noisy Edge Environments. In 2021 IEEE 7th World Forum on In- ternet of Things (WF-IoT), pp. 309-314 314,, June 2021. doi: URL
1TT.W/WIIOTT136.022222 leee. http://attaccttecccceee/e///e///
[42] Xingjian, S. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28:1, 2015.
[43] Yao, Z. and Liu, X. A CNN-Transformer Deep Learning Model for Real-time Sleep Stage Wireless Device In 2023 11th International IEEF/EMBS Conference on Neural Engineering (NER), pp. 1-4, Baltimore, MD, USA, April 2023. IEEE ISBN https://ieeexple ieee hrtg//occcmmmet/111123335
Classification in an
doi: 10. 1099NER5222111222 221. URL
[44] Zhang, H., Xing. M., Wu, Y., and Zhao, C. Compiler technologies in deep learning co-design: A survey. Intelligent Computing. 2:0040, 2023.
[45] Zhao, J., Gao, X., Xia, R., Zhang, Z., Chen, D., Chen, L.,, Zhang, R., Geng, Z., Cheng, B., and Jin, X. Apollo: Automatic partition-based operator fusion through layer by layer optimization. Proceedings of Machine Learning and Systems, 4:1-19, 2022.
[46] Zheng, H.-S., Liu, Y.-Y., Hsu, C.-F., and Yeh, T. T. Streamnet: memory-efficient streaming tiny deep learning inference on the microcontrollerr. Advances Neural Information Processing Systems, 36, 2024.
[47] Zheng, I.., Jia, C., Sun, M., Wu, Z., Yu, C. H., Haj-Ali, A., Wang, Y., Yang, J., Zhuo, D., Sen, K., et al. Ansor: Generating Nrigh-Performance/ tensor programs for deep learning. In 14th USENIX symposium operating systems design and implementation (OSDI 20), pp. 863-879, 2020.
[48] Zheng, S., Liang, Y., Wang, S., Chen, R., and Sheng, K. Flextensor: An automatic schedule exploration and optimization framework for tensor computation on heterogeneous system. In Proceedings of the "wenty-Fftth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 859-873, 2020.
[49] Zheng, S., Chen, R., Li, M., Yc, Z., Ceze, L., and Liang, Y. vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs. Proceedings of Machine Learning and Systems, 6:452 464, May 2024.
[50] Zhu-Zhou, F., Computationally constrained audio-based violence detection through transfer learning and data augmentation techniques. Applied Acoustics, 113:10 October 2023. ISSN 0003-682X. 109638
D., Gil-Pita, R., Utrilla-Manso, M., and Rosa-Zurera, M.
doi: 10. 10.10066/..ppacoosss222.... 109638. URL https: //www
arriccle/p///00006222220333
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction reflect the contribution and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are presented in Section 9
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: We transformed the optimization problems presented in Section 4 into shortest path problems (or the variants) and solved them by classic graph algorithm like Dijkstra's 's, whose correctness is proven by predecessors. The time complexity is given and proven in Appendix [
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All technical details are provided in the experimental parts and in the supple- mental appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer:
[Yes]
Justification:
We open-sourced the code a anonymous repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. htcp//blcc//uddeeeeeeeeeeeeeeeee
ssiobiioiiiioocccyyy. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We released all technical details throughout the paper and the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
All results are deterministic. No need to report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is provided Section g and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips. htt.eerrpp.....pppppaaaaaaaaaaaa
Answer:
[Yes]
Justification:
The research is aligned with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is focused on technical improvements the core algorithmic frame- work and does not directly address an application area with immediate societal implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All used assets are properly cited in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We released our code in a anonymous repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLMs were only used for wordsmiths.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: hattaa///aaarrppsscc///offeereee for what should or should not be described.
20

---

A Relation between msf-CNN and kernel fusion
We note that readers might potentially confuse patch-based multi-stage fusion (msf-CNN, approach) on the one hand, and on the other hand traditional kemne fusion techniques. These two approaches are orthogonal and can be applied concurrently for maximum benefit.
While kernel fusion optimizes computation overhead, msf-CNN instead targets memory efficiency, the latter being the critical first hurdle on small edge devices. As such, we disambiguate this further:
focuses primarily on reducing redundant data movements between GPU and RAM by combining multiple primitive operators (e.g. Batch Normalization. ReLU, Softmax, etc.) with a primary, memory- nemory-bound operator (e.g. conv, pooling) into a single kernel. While kernel fusion improves compute latency and data throughput, it does not address the fundamental memory usage issue that arises when processing multiple primary operators sequentially.
Kernel fusion [40 29 45
Patch-based multi-stage fusion (msf-CNN, our approach) extends the idea of patch-based fusion 2 28
More specifically, msf-CNN:
Fuses multiple layers (i.e. primary operators like convolution and pooling) into a single computational stage. Implements patch-based partial computation, which drastically reduces peak memory usage by processing input data in smaller patches while maintaining accuracy.
Introduces a compute-memory trade-off mechanism that allows users to prioritize either memory consumption computational efficiency based on their deployment constraints.
This makes msf-CNN fundamentally different from traditional kernel fusion techniques.
To the best of our knowledge, the closest related works to msf-CNN are StreamNet and MCUNetv2, which have been the state-of-the-as for patch-based fusion on microcontrollers so far. Compared to this state of the art, based on our measurements, msf- Bsff-NN achieves to 50% reduction of peak RAM usage model inference, for the same inference accuracy, which thus enables more models to fit smaller devices.
B Analysis of the H-cache buffer size
For fusion block containing n layers, the cache buffer size of the i-th layer under H-cache scheme is given by
Buf, = 4 x k2 x
(11)
where k and ct"h the tile size, kernel size and input channels number, respectively. Obviously, the first layer of the fusion block does not need any input cache, thus Buf; O. The total cache size of the fusion block Buf E. Buf,.
C Analysis of the amount of MAC operations
Analyzing the number of MAC operations the fusion block is quite complex. The input tensor for each layer is sliced into overlapped tiles, and the kernel performs convolution the data within each tile. Here, the number of overlapped tiles tile each layer is
h + 2p -
+ 1JL in + 2p
Artile
+ 1J,
(12)
tile
slayer
where h" in are the height and width of input tensor, tiile, slayer are the stride of tile and layer, represents the input padding. Recall that I, & are the tile size and kernel size respectively.
And the output size of each tile determined as:
Ote
glayer + I
(13)
whereby cat is the number of output channels. We can therefore derive the number Clayer of MAC operations of a fused convolutional layer as:
Clayer
Atile Otile x 12t x out.
(14)
21

---

Finally, we can derive C the total MAC operations of the entire fusion block as:
C
Clayer
(15)
D Complexity analysis of the search algorithm
We provide a quick preliminary analysis of the worst-case scenario. We also highlight that these shortest path computations do not take place on the microcontroller at runtime, but offline on a PC (which expands the realm of what can assessed as bearable computation).
First, we consider the lower-bound of the search algorithm. As shown in Section 6, both Problems PI and P2 without constraints be transformed into a multiple single-source-single- path problem, which can by solved by Dijkstra's algorithm with Fibonacci heap [00 with complexity:
shortest
O(E + VVog(V)),
where and denote the edges (possible fusion blocks) and vertices (layers) of the DAG. In the worst case E Ex nn - 1), leading the overall lower-bound to O(V2).
Concerning Problem P1 with constraints: if we don't 't prune the search space (iteratively), need to brute force all possible fusion settings of the DAG to form a subspace that fulfills the latency constraints. This leads enumerating all simple paths from input layer the output layer. In the worst-case, starting from the input layer, we obtain -2 fusion combinations, which becomes unbearable for deep neural network. We can prove this complexity by simple induction:
In worst-case scenario, we assume complete DAG, where each node connected to all its predeces- sors.
Base Case: For V 2, there is only one complete compute path:
2
Inductive Step: Assume a complete DAG with nodes has 2 complete compute paths. To construct complete DAG with & nodes:
Add a new node with the same incoming edges as the last node. This contributes 2 paths.
2. Connect the new node the duplicated node, adding another 2 paths.
Total paths:
2k + 2k 2k-1
Therefore, by induction, a complete DAG with V nodes has 2 complete compute paths.
Hence, apply pruning strategy (see Equ. 9-11) reduce the complexity from - to 0(V3). The idea: we erase the edges with maximal RAM usage per iteration. In the worst-case, only edge is erased in each iteration, with complete DAG with V(V-1) edges. Thus, the worst-case complexity of our search algorithm for constrained Problem P1 is O(V3).
E
Accuracy Evaluation
msf-CNN is a computation scheduling and memory optimization technique that does not alter the model's architecture, parameters, or the mathematical operations performed. Like MCUNet and StreamNet, msf-CNN only changes when and how intermediate results are computed and stored to minimize peak memory. Therefore, the final output, and consequently the model's accuracy, remains identical to the original, unfused model. Hence, standard ML performance benchmarks focusing on accuracy irrelevant here.
Nevertheless, to make sure, we conducted extra experiments on imagenet and vww dataset, as MCUNet did. We reused the model weights from the pre-trained MCUNet models, and selected
22

---

Table 5: Model Top-1 1 Accuracy
Method
MBV2-w0.35
MN2-vww5
MN2-320K
Vanilla
48.94 %
88.89 %
61.76
msf-CNN
48.94 %
88.89 %
61.76
Pmex= = 64kB fusion constraints. The results show in Table 5that that no Top-1 accuracy drop was found between vanilla models and the msf-CNN variants.
F
Experiment Details
Here, we provide additional details the experiments, including results not presented in the main text.
F.1 Supplemental Results
Table provides the underlying data corresponding to Figure including the specific constraints applied to Pl and P2, respectively.
Table Optimal fusion settings on Nucleo-f767zi. RAM (kB), Latency (ms). SAA: Same as above.
Gray: msf-CNN beats MCUNetva
MB V2 -0.35
MN2-vww5
MN2-320K
RAM
Latency
RAM
Latency
RAM
Latency
Vanilla
194.44
807.60
96.00
509.70
309.76
4394.30
MCUNetva
63.00
1513.00
45.00
810.00
215.00
2777.00
StreamNet
66.00
417.00
44.00
225.00
208.00
1444.00
Pl: Min. RAM s.t.
Compute Cost Limit
1.1
68.00
961.90
45.28
696.00
199.60
4171.00
1.2
(SAA)
26.24
769.20
196.07
4525.10
1.3
21.39
1313.80
20.57
922.70
195.33
4680.70
F moo
1.4
15.20
1412.30
17.90
931.30
156.86
5128.90
1.5
(SAA)
(SAA)
94.22
5370.30
Inf
8.56
1996.80
15.37
1723.00
51.16
19 329.90
P2: Min. Compute Cost s.t. RAM Limit
16 kB
15.20
1412.30
17.90
931.30
(No Solution)
32 kB
25.80
1266.30
26.24
769.20
mma
64 kB
63.60
1121.70
45.28
684.60
63.46
9458.60
128 kB
83.13
947.00
89.60
683.40
94.22
5370.30
256 kB
181.44
879.20
(SAA)
247.81
3923.20
F.2
Impact on Power Consumption
We used the Nordic Semiconductor Power Profiler Kit II (PPK2) on nr152840 boards to provide preliminary measurements of the power draw. We observed that across all optimization configura- tions, inference current and deep sleep current remained consistently around 5.4 mA and 1.9 mA, respectively, which hints straightforward link between latency and energy consumption. We note, however, that the precision the inference current we measured may be limited due to our measurement setup and should be subject to further investigations.
G Network Architecture Extension
We are currently extending msf-CNN to support RNN- and Transformer-base architecture.
23

---

G.1 msf-CNN on RNNILSTM/GRU
For I-D sequence input, it trivial to fuse the cascade RNN cells into one block, avoid outputting the complete sequence between RNNs during inference. This eliminates RAM usage for storing intermediate output, especially when facing long input sequence. The compute cost remains the same as the original models, since no recompute was introduced compared to 2-D inputs.
For 2-D input 42), all matrix-vector multiplications Hadamard product is applied on two matrices inside the RNN cells. In this case, we can analyze the compute graph check which convolution operators (inter- or intra RNN cells) can be fused into blocks, and calculate their RAM usage, compute cost as we did in the original msf-CNN.
replaced by 2-D convolutions and the
G.2 msf-CNN on Transformer
We only considered decoder-only architecture far.
For 1-D sequence input, there is not much optimization space, since the transformer/attentiin blocks are full of dense layers or heavy matrix-matrix multiplications, where patch-based fusion cannot be applied to reduce RAM usage. On the other hand, we could apply msf-CNN on the covolutional layers before or after transformer blocks of hybrid model (such as in ViT [3),, and integrate Efficient Attention 36 to further improve space and compute complexity of attention layer.
For 2-D image input, msf-CNN can be extended to the convoluional variants of common attention layers (such as VAN [33 and SegNeXt (12)) which contain cascade convolutional layers inter- or intra the attention layers. In this case, msf-CNN can be applied on deciding the fusion strategy.
24


Reviews:
[{"review_id": "NqFztwX2CB", "reviewer": "Reviewer_NmDR", "summary": "The paper describes an algorithmic optimization for fitting CNNs on microprocessors. It introduces the tight resourcing constraints that motivate the problem. It covers the limitations of related work as being model or architecture specific. It presents the basic idea, which is based on an inverted dataflow graph in which ops are nodes and stored weights are edges, and the optimization is cast as a graph-based algorithm. Edges are annotated with RAM usage if unfused and compute cost if fused; optimization then seeks to find a path that minimizes one cost potentially under a constraint on the other. The authors validate their work first with an analytical model, then with actual experiments. The authors show their algorithm is very flexible, and outperforms competitive algorithms in some or all cases. They also cover related work more broadly.", "questions": "My general question on this paper is around significance. It's good work, but limited to applying a specific optimization for a specific hardware class. Acceptance should depend on significance relative to the field of papers submitted. \n\nDetailed suggestions in the strengths/weaknesses analysis will improve clarity of the paper, but significance alteration would take substantial work. \n\nTwo ideas for leveraging the strength of the approach in this paper to a broader problem area would be:\n(1) Is there an opportunity to apply this technique outside MCUs? \n(2) Is there an opportunity to say something significant in a larger context around the shape of RAM -- Compute curves relative to model size for common model architectures?\n\nAddressing either one well would raise my score a level but would require substantial effort.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n-- Real problem solved with a useful technique, with a clean algotihmic formulation \n-- Good use of both analytical and experimental proofs to validate\n-- Thorough treatment of related work\n-- Good structure: sets up problem, solves problem, shows value\n-- Clearly written\n\nWeaknesses:\n-- Somewhat incremental and narrow in impact (MCUs)\n-- The graphs are hard to read, especially in grayscale\n-- The analytical result table uses random number of digits for results\n\nSection by section:\n-- Introduction: section would benefit from a sentence or two describing layer fusion up front, given the importance of the idea to the rest of the explanation.\n-- Introduction: these is are tradeoffs that limit the use of fusion (why not fuse everything) which could be more clearly introduced\n-- Background: suggest a more broad strokes intro to the work on fusion, which is more or less a loop transform -- it's worth establishing that this work is novel in both the AI domain and from a basic compiler perspective\n-- High-level idea: this section is a strength of the paper and something more papers should include. Suggest introducing the RAM and ops optimization criteria in the main text rather than through the caption though.\n-- Sections 4-5: no issues\n-- Section 6: The structure in this section could made clearer: there is a jump from the idea of dual problems to two corresponding optimizers, but this jump is not well supported in the text. Suggest conveying the structure of the section in the first couple of paragraphs.\n-- Section 7: This section would benefit from an introductory sentence or two. \n-- Section 8: The experimental setup clear enough, but not well connected to the structure of section 6 -- the reader must figure out which optimization strategy is being used when from context. Also, the tables here are should have a fixed numerical format with right hand justification for ease of digestion.\n-- Section 9: related work and background field duplicative IMO\n-- Section 10: conclusion is crisp and effective", "was_revised": true, "final_justification": "In light of the general quality of the work, and the author's coherent response about generality, I am raising my response to accept.", "rebuttal": "Thank you very much for the insightful comments and constructive suggestions on our work. Below, we address each of your questions in detail.\n\n**[Q1. Is there an opportunity to apply this technique outside MCUs? ]**\n\nWhile the focus and main motivation for our work was microcontrollers, the applicability of msf-CNN is indeed not limited to MCU-based systems. By design, msf-CNN also supports common CPU platforms such as mobile phones, PCs, and computing clusters: the same cost estimator can be applied and the same backend can be used to generate universal C-code-based fusion kernels. That means, the code generated by msf-CNN can actually run on any CPUs (such as x86 and Cortex-A). Thus, for instance, cloud computing may serve more clients concurrently by using RAM usage optimizations provided by msf-CNN.\n\nMoreover, msf-CNN can be extended to machines equipped with accelerators (e.g., GPUs, FPGAs, ASICs). As long as RAM usage and compute costs are accurately modeled in the optimizer, and a corresponding backend is implemented for fusion kernel generation, msf-CNN can be generalized to support a wide range of hardware architectures.\n\nTo avoid blurring our primary focus on MCUs, which have more RAM reduction requirements, we did not include experimental results for non-MCU platforms in this work.\n\n\n**[Q2. Is there an opportunity to say something significant in a larger context around the shape of RAM -- Compute curves relative to model size for common model architectures?]**\n\nLooking at Fig. 4, we can observe that all curves of msf-CNN share the quasi-hyperbola shape. That's because the more layers we fuse, the more RAM usage we save, and hence more recompute are introduced. The compute latency of vanilla - which do not fuse - serves as a *theoretical lower-bound* and thus \"controls\" the boundary behaviors of the curves.\n\nNow let's assume \"model size\" refers to the number of model parameters (weights, bias, etc., which can be seen as \"constants\" participating in computation during model inference). Then, the bigger the model size, the more complex and deeper the model is, and thus the more time inference takes, which moves the curves \"up\".\n\nHowever, model size as such is not the only factor that drags on the latency lower-bound. The input size also determines the compute latency. Using the same model but with different input sizes consumes different timespans. Furthermore, the model architecture plays another important role. For example, a $10 \\times 3$ dense layer and a conv layer with $k=3, c=10$ on a sequence with 10 elements have a huge gap in time consumption, though they share the same model size (30): while the dense layer incurs 30 MACs, the conv layer incurs 240 MACs.\n\n**[C1. Formatting Concerns and Section-by-section suggestions]**\n\nThank you for the insightful suggestions. We will ensure that the readability and accessibility of all tables and figures are thoroughly improved in the final submission. Furthermore, we will revise the manuscript text to align with your valuable feedback.\n\nThanks again for your feedback. Please let us know if you have further questions."}, {"review_id": "l3z1IR7wb6", "reviewer": "Reviewer_oCE3", "summary": "The paper presents msf-CNN, a patch-based multi-stage fusion technique for CNNs designed to run on memory-limited microcontrollers (MCUs). By modeling the fusion solution space as a directed acyclic graph (DAG), the method optimizes RAM usage and computational cost, reducing peak RAM by 50% compared to MCUNetV2 and StreamNet. Experiments on ARM Cortex-M, RISC-V, and ESP32 validate its efficiency, and the authors provide an open-source implementation.", "questions": "Have the authors explored adapting msf-CNN to non-CNN architectures such as transformers? If so, what challenges arise (e.g., handling attention layers), and how might the approach be adapted to these architectures? A positive response demonstrating feasibility would significantly strengthen the work's impact and generality.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\na. Practical Significance: The work addresses a critical challenge in TinyML—enabling deep neural networks on ultra-low-memory devices—which has broad implications for AIoT applications.\\\nb. Novel and Simple Scheme: The DAG representation and pruning strategy are both novel and straightforward, enabling systematic optimization of memory and computational usage.\\\nc. Convincing Results: The approach achieves a 65% to 87% RAM reduction compared to prior work across a variety of MCUs.\\\nd. Clarity and Rigor: The paper clearly explains the problem definition, graph modeling, and algorithm design, supported by mathematical formulations and complexity analysis.\n\nMinor Weaknesses\n\na. Limited Network Scope: The method focuses solely on CNNs, overlooking emerging architectures such as transformers. This limitation is acknowledged by the authors.\\\nb. Venue Fit: While technically solid, the paper may be more suited for machine learning systems venues, given its strong emphasis on system-level optimization rather than novel learning algorithms or architectural innovations.", "was_revised": true, "final_justification": "This paper presents msf-CNN, a patch-based multi-stage fusion technique for CNNs optimized for memory-constrained MCUs. By modeling fusion as a directed acyclic graph (DAG), it achieves 50% peak RAM reduction vs. MCUNetV2/StreamNet and 65–87% across ARM Cortex-M, RISC-V, and ESP32. Strengths include practical TinyML impact, a novel DAG strategy, robust results, and rigorous exposition, with open-source code. Though CNN-specific (not Transformers), it advances foundational computer vision via pragmatic memory reduction, a meaningful contribution. Therefore, my final rating remains an acceptance.", "rebuttal": "Thank you very much for the insightful comments and constructive suggestions on our work. We are currently extending msf-CNN to support RNN- and Transformer-based architecture. Below are some details that we can share so far. Please let us know if you have further questions.\n\n**(1) msf-CNN on RNN/LSTM/GRU**\n\nFor 1-D sequence input, it is trivial to fuse the cascade RNN cells into one block, to avoid outputting the complete sequence between RNNs during inference. This eliminates RAM usage for storing intermediate output, especially when facing a long input sequence. The compute cost remains the same as the original models, since no recompute was introduced compared to 2-D inputs.\n\nFor 2-D input [1], all matrix-vector multiplications are replaced by 2-D convolutions and the Hadamard product is applied on two matrices inside the RNN cells. In this case, we can analyze the compute graph to check which convolution operators (inter- or intra RNN cells) can be fused into blocks, and calculate their RAM usage, compute cost as we did in the original msf-CNN.\n\n**(2) msf-CNN on Transformer**\n\nWe only considered decoder-only architecture so far.\n\nFor 1-D sequence input, there is not much optimization space, since the transformer/attention blocks are full of dense layers or heavy matrix-matrix multiplications, where patch-based fusion cannot be applied to reduce RAM usage. On the other hand, we could apply msf-CNN on the covolutional layers before or after transformer blocks of hybrid model (such as in ViT [3]), and integrate Efficient Attention [2] to further improve space and compute complexity of attention layer. \n\nFor 2-D image input, msf-CNN can be extended to the convoluional variants of common attention layers (such as VAN [4] and SegNeXt) which contain cascade convolutional layers inter- or intra the attention layers. In this case, msf-CNN can be applied on deciding the fusion strategy.\n\n[1] Shi, Xingjian, et al. \"Convolutional LSTM network: A machine learning approach for precipitation nowcasting.\" Advances in neural information processing systems 28 (2015).\n\n[2] Shen, Zhuoran, et al. \"Efficient attention: Attention with linear complexities.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.\n\n[3] Liu, Yang, et al. \"A survey of visual transformers.\" IEEE transactions on neural networks and learning systems 35.6 (2023): 7478-7498.\n\n[4] Guo, Meng-Hao, et al. \"Visual attention network.\" Computational visual media 9.4 (2023): 733-752.\n\n[5] Guo, Meng-Hao, et al. \"Segnext: Rethinking convolutional attention design for semantic segmentation.\" Advances in neural information processing systems 35 (2022): 1140-1156."}, {"review_id": "WQbzzizyUI", "reviewer": "Reviewer_zzV4", "summary": "The proposed msf-CNN approach essentially explores trade-offs between recomputation and memory saving. The authors formulate the problem of identifying optimal solutions of applying patch-based layer fusion as a shortest path problem and leverage existing graph algorithms to devise a set of solutions that constitute Pareto curves (e.g., the ones depicted in Figure 4). The authors claim that the proposed pruning strategy squeezes the search space to reach the time complexity from exponential to polynomial time, and the results on MCU-based IoT boards are promising: msf-CNN provides richer trade-offs between memory-saving and recomputation over the alternatives.", "questions": "* To get a bigger picture, it would be great to report the energy consumption. It is not clear if squeezing the memory usage to the extreme brings any benefit to the energy perspective. Could you report energy consumption in the evaluation?\n* Can you perform an exhaustive search and report how far what you found are from the optimal configurations?\n* Can you refine msf-CNN such that it can outperform StreamNet (e.g., Figure 4)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ msf-CNN provides a spectrum of solutions that trade memory saving with recomputation, which can be effective to memory-capacity-constraint IoT devices\n+ The problem formulation (transforming it into a graph’s shortest path problem) is decent.\n\nWeaknesses:\n- msf-CNN does not seem to be clearly better than StreamNet. It would be great if the gist of msf-CNN can be applied to StreamNet, such as the new Pareto curves beat StreamNet.\n- In other words, the heuristic solution does not seem to guarantee being optimal (considering the StreamNet case).", "was_revised": true, "final_justification": "My original concern or comment that \"msf-CNN does not seem to be clearly better than StreamNet. It would be great if the gist of msf-CNN can be applied to StreamNet, such as the new Pareto curves beat StreamNet.\" has not been entirely satisfied, but found decent reasons why it is tough to be addressed in a short period of time. Considering the reviews of the fellow reviewers and the interaction process with the authors, I think it is okay to adjust my stance to be more positive. I do not have an expertise to champion the paper though.", "rebuttal": "Thank you very much for the insightful comments and your constructive suggestions on our work. We address your remarks point-by-point below.\n\n**[Q1: Could you report energy consumption in the evaluation?]**\n\nYes. The energy consumption is actually decoupled from the RAM usage, since we only used the on-chip memory which is always on, alongside the MCU. However, the RAM usage budget still affects the compute latency and hence the energy consumption (Power $\\times$ Time).\n\nTo get an accurate measurement, we use the Nordic Semiconductor Power Profiler Kit II (PPK2) on nrf52840 boards for measuring the energy consumption of SoC. It showed a steady working current around 5.4 mA regardless of the RAM usage of models. So the energy consumption is only linearly related to compute latency. We can coarsely say that, the more RAM we suppress, the more recompute time is introduced, and the more energy will be used.\n\n**[Q2: Can you perform an exhaustive search and report how far what you found are from the optimal configurations?]**\n\nThanks for this question which gives us the opportunity to disambiguate this aspect. We'd like to clarify that all configurations identified as solutions by msf-CNN under certain constraints are *theoretically* optimal -- this optimality is guaranteed by Dijkstra's algorithm as we've applied it. The apparent performance gap between StreamNet and msf-CNN comes from the way we implemented the fusion block. We provide the details on the belowing Q3.\n\n**[Q3: Can you refine msf-CNN such that it can outperform StreamNet (e.g., Figure 4)?]**\n\nOne reason why msf-CNN did not perform as well as StreamNet is that our implementation of msf-CNN didn't apply any hardware-dependent compute library nor any hardware acceleration instructions, while on the contrary, StreamNet focuses on ARM hardware, and uses [CMSIS-NN](https://github.com/ARM-software/CMSIS-NN) as its backend, which is ARM-specific. Also, StreamNet applied 2-D cache, compared to 1-D cache in msf-CNN. It further alleviates the recompute issue of fusion blocks.\n\nFor a fairer comparison on ARM hardware, we have added to msf-CNN the option to use CMSIS-NN as backend for ARM platform (this option can be turned off for non-ARM hardware). Using this alternative setup, we have carried out more measurements (see tables below) to complement those from Figure 4. It shows that although msf-CNN does not fully surpass StreamNet, the Pareto curves are much closer to it. We're also integrating 2-D cache into msf-CNN, which yields a new trade-off option for even more flexibility on more hardware, using msf-CNN.\n\nSpeed Up: Acceleration factors compared to the original results (msf-CNN without CMSIS-NN) from Figure 4.\n\nSAA: Same as Above.\n\nNS: No Solution under specified constraint.\n\nMBV2-w0.35\n---\n| Method       | Constraints | RAM (kB) | Latency (ms) | Speed Up |\n| ------------ | ----------- | -------- | ------------ | -------- |\n| StreamNet-2D | NA          | 66       | 417.0        | NA       |\n|              |             |          |              |          |\n| **msf-CNN**  |             |          |              |          |\n| $F_{max}$    | 1.1         | 68.0     | 860.8        | 1.1      |\n|              | 1.2         | (SAA)    |              |          |\n|              | 1.3         | 21.4     | 970.2        | 1.4      |\n|              | 1.4         | 15.2     | 973.9        | 1.5      |\n|              | 1.5         | (SAA)    |              |          |\n|              | Inf         | 8.6      | 1213.8       | 1.6      |\n| $P_{max}$    | 16 kB       | 15.2     | 973.9        | 1.5      |\n|              | 32 kB       | 25.8     | 950.1        | 1.3      |\n|              | 64 kB       | 63.6     | 967.7        | 1.2      |\n|              | 128 kB      | 83.1     | 806.1        | 1.2      |\n|              | 256 kB      | 181.4    | 828.4        | 1.1      |\n\nMN2-vww5 \n---\n| Method       | Constraints | RAM (kB) | Latency (ms) | Speed Up |\n| ------------ | ----------- | -------- | ------------ | -------- |\n| StreamNet-2D | NA          | 44       | 225.0        | NA       |\n|              |             |          |              |          |\n| **msf-CNN**  |             |          |              |          |\n| $F_{max}$    | 1.1         | 32.9     | 405.7        | 1.7      |\n|              | 1.2         | 26.2     | 428.0        | 1.8      |\n|              | 1.3         | 20.6     | 506.0        | 1.8      |\n|              | 1.4         | 17.9     | 513.0        | 1.8      |\n|              | 1.5         | (SAA)    |              |          |\n|              | Inf         | 15.4     | 845.4        | 2.0      |\n| $P_{max}$    | 16 kB       | 17.9     | 513.0        | 1.8      |\n|              | 32 kB       | 26.2     | 428.0        | 1.8      |\n|              | 64 kB       | 38.6     | 423.6        | 1.6      |\n|              | 128 kB      | 89.6     | 423.0        | 1.6      |\n|              | 256 kB      | (SAA)    |              |          |\n\nMN2-320K \n---\n| Method       | Constraints | RAM (kB) | Latency (ms) | Speed Up |\n| ------------ | ----------- | -------- | ------------ | -------- |\n| StreamNet-2D | NA          | 208      | 1444.0       | NA       |\n|              |             |          |              |          |\n| **msf-CNN**  |             |          |              |          |\n| $F_{max}$    | 1.1         | 190.4    | 3618.1       | 1.2      |\n|              | 1.2         | 186.9    | 3881.2       | 1.2      |\n|              | 1.3         | 186.1    | 3983.8       | 1.2      |\n|              | 1.4         | 156.9    | 4247.8       | 1.2      |\n|              | 1.5         | 94.2     | 4332.4       | 1.2      |\n|              | Inf         | 51.2     | 9976.7       | 1.9      |\n| $P_{max}$    | 16 kB       | (NS)     |              |          |\n|              | 32 kB       | (NS)     |              |          |\n|              | 64 kB       | 63.5     | 5901.3       | 1.6      |\n|              | 128 kB      | 94.2     | 4332.4       | 1.2      |\n|              | 256 kB      | 247.8    | 3436.6       | 1.1      |\n\n\nThanks again for your feedback. Please let us know if you have further questions."}, {"review_id": "CVNX7SF3Wm", "reviewer": "Reviewer_YE3p", "summary": "The paper presents msf-CNN, a memory-efficient CNN fusion method for microcontrollers. It explores fusion strategies using a directed acyclic graph, achieving up to 50% RAM savings over prior methods like MCUNetV2, while maintaining low latency. Tested on ARM Cortex-M, RISC-V, and ESP32, msf-CNN enhances real-time AI on resource-constrained devices.", "questions": "- If the authors can add the missing accuracy evaluation, I will increase my rating.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed multi-stage fusion technique and graph-based search framework look novel and sound.\n- Empirical results show significant memory saving.\n- The method is implemented and tested on diverse MCUs.\n- The implementation is publicly available.\n\nWeaknesses:\n- Limited to CNNs. The method currently targets only CNN architectures. \n- Compute Overhead Can Be High. Extreme fusion configurations achieve dramatic memory reduction at the cost of significant latency overhead, making them unsuitable for time-sensitive applications.\n- Missing Accuracy Evaluation. The paper lacks a comprehensive evaluation of the accuracy–memory, and accuracy–latency trade-offs compared to prior models.", "was_revised": true, "final_justification": "The authors' responses have addressed my concerns. I am inclined to recommend a borderline accept.", "rebuttal": "Thank you very much for the insightful comments and your constructive suggestions on our work. We address your remarks point-by-point below.\n\n**[C1: Missing accuracy evaluation]**\n\nWe appreciate the reviewer raising this, as it allows us to clarify a crucial aspect of our method. msf-CNN is a computation scheduling and memory optimization technique that does not alter the model’s architecture, parameters, or the mathematical operations performed. Like *patch-based fusion* in MCUNet and StreamNet, msf-CNN only changes when and how intermediate results are computed and stored to minimize peak memory. Therefore, the final output, and consequently the model’s accuracy, remains identical to the original, unfused model. Hence, standard ML performance benchmarks focusing on accuracy are irrelevant here. \n\nNevertheless, to make sure, we conducted extra experiments on imagenet and vww dataset, as [mcunet](https://github.com/mit-han-lab/mcunet/) did. We reused the model weights from the pre-trained mcunet models, and selected $P_{min}= 64 \\text{kB}$ as fusion constraints. The results show that no Top-1 accuracy drop was found between vanilla models and the msf-CNN variants.\n\n\n| Method      | MBV2-w0.35 | MN2-vww5 | MN2-320K |\n| ----------- | ---------- | -------- | -------- |\n| Vanilla     | 48.94%     | 88.89%   | 61.76%   |\n| **msf-CNN** | 48.94%     | 88.89%   | 61.76%   |\n\n\n\nAccordingly, we will add disambiguating text in the paper, explicitly mentioning the above.\n\n**[C2: Limited to CNNs]**\n\nWe note that this comment aligns with the feedback provided by Reviewer oCE3. For your convenience, we have re-posted our detailed response to Reviewer oCE3 below.\n\nWe are currently extending msf-CNN to support RNN- and Transformer-based architecture. Below are some details that we can share so far. Please let us know if you have further questions.\n\n**(1) msf-CNN on RNN/LSTM/GRU**\n\nFor 1-D sequence input, it is trivial to fuse the cascade RNN cells into one block, to avoid outputting the complete sequence between RNNs during inference. This eliminates RAM usage for storing intermediate output, especially when facing a long input sequence. The compute cost remains the same as the original models, since no recompute was introduced compared to 2-D inputs.\n\nFor 2-D input [1], all matrix-vector multiplications are replaced by 2-D convolutions and the Hadamard product is applied on two matrices inside the RNN cells. In this case, we can analyze the compute graph to check which convolution operators (inter- or intra RNN cells) can be fused into blocks, and calculate their RAM usage, compute cost as we did in the original msf-CNN.\n\n**(2) msf-CNN on Transformer**\n\nWe only considered decoder-only architecture so far.\n\nFor 1-D sequence input, there is not much optimization space, since the transformer/attention blocks are full of dense layers or heavy matrix-matrix multiplications, where patch-based fusion cannot be applied to reduce RAM usage. On the other hand, we could apply msf-CNN on the covolutional layers before or after transformer blocks of hybrid model (such as in ViT [3]), and integrate Efficient Attention [2] to further improve space and compute complexity of attention layer. \n\nFor 2-D image input, msf-CNN can be extended to the convoluional variants of common attention layers (such as VAN [4] and SegNeXt) which contain cascade convolutional layers inter- or intra the attention layers. In this case, msf-CNN can be applied on deciding the fusion strategy.\n\n[1] Shi, Xingjian, et al. \"Convolutional LSTM network: A machine learning approach for precipitation nowcasting.\" Advances in neural information processing systems 28 (2015).\n\n[2] Shen, Zhuoran, et al. \"Efficient attention: Attention with linear complexities.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.\n\n[3] Liu, Yang, et al. \"A survey of visual transformers.\" IEEE transactions on neural networks and learning systems 35.6 (2023): 7478-7498.\n\n[4] Guo, Meng-Hao, et al. \"Visual attention network.\" Computational visual media 9.4 (2023): 733-752.\n\n[5] Guo, Meng-Hao, et al. \"Segnext: Rethinking convolutional attention design for semantic segmentation.\" Advances in neural information processing systems 35 (2022): 1140-1156.\n\nThanks again for your feedback. Please let us know if you have further questions."}]

Author rebuttal:

Meta-review: The paper proposes msf-CNN, a memory-efficient CNN fusion method for microcontrollers. msf-CNN explores fusion strategies using a directed acyclic graph, achieving significant savings over prior methods, while maintaining low latency. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.

This paper received four reviews. After the rebuttal, all reviewers are positive about this paper, with the overall ratings of 2 accept and 2 borderline accept. They recognized the contributions and novelty of this paper, such as a useful technique, practical significance, a novel scheme, convincing results, and a decent problem formulation. Further, the reviewers are generally satisfied with the rebuttal, and their concerns are mostly well addressed. I recommend acceptance for its above advantages.
Decision: Accept (poster)