Title: msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML
Venue: NeurIPS 2025
Accepted: yes
Keywords: TinyML

DNN

Fusion

Microcontroller

Random access memory
Abstract: AI spans from large language models to tiny models running on microcontrollers (MCUs). Extremely memory-efficient model architectures are decisive to fit within an MCU's tiny memory budget e.g., 128kB of RAM. However, inference latency must remain small to fit real-time constraints. An approach to tackle this is *patch-based fusion*, which aims to optimize data flows across neural network layers. In this paper, we introduce *msf-CNN*, a novel technique that efficiently finds optimal fusion settings for convolutional neural networks (CNNs) by walking through the fusion solution space represented as a directed acyclic graph. Compared to previous work on CNN fusion for MCUs, msf-CNN identifies a wider set of solutions. We published an implementation of msf-CNN running on various microcontrollers (ARM Cortex-M, RISC-V, ESP32).  We show that msf-CNN can achieve inference using 50% less RAM compared to the prior art (MCUNetV2 and StreamNet). We thus demonstrate how msf-CNN offers additional flexibility for system designers.


Peer reviews:
comments: SUMMARY:
The paper presents msf-CNN, a memory-efficient CNN fusion method for microcontrollers. It explores fusion strategies using a directed acyclic graph, achieving up to 50% RAM savings over prior methods like MCUNetV2, while maintaining low latency. Tested on ARM Cortex-M, RISC-V, and ESP32, msf-CNN enhances real-time AI on resource-constrained devices.

STRENGTHS AND WEAKNESSES:
Strengths:
- The proposed multi-stage fusion technique and graph-based search framework look novel and sound.
- Empirical results show significant memory saving.
- The method is implemented and tested on diverse MCUs.
- The implementation is publicly available.

Weaknesses:
- Limited to CNNs. The method currently targets only CNN architectures. 
- Compute Overhead Can Be High. Extreme fusion configurations achieve dramatic memory reduction at the cost of significant latency overhead, making them unsuitable for time-sensitive applications.
- Missing Accuracy Evaluation. The paper lacks a comprehensive evaluation of the accuracy–memory, and accuracy–latency trade-offs compared to prior models.

QUESTIONS:
- If the authors can add the missing accuracy evaluation, I will increase my rating.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-17

comments: SUMMARY:
The proposed msf-CNN approach essentially explores trade-offs between recomputation and memory saving. The authors formulate the problem of identifying optimal solutions of applying patch-based layer fusion as a shortest path problem and leverage existing graph algorithms to devise a set of solutions that constitute Pareto curves (e.g., the ones depicted in Figure 4). The authors claim that the proposed pruning strategy squeezes the search space to reach the time complexity from exponential to polynomial time, and the results on MCU-based IoT boards are promising: msf-CNN provides richer trade-offs between memory-saving and recomputation over the alternatives.

STRENGTHS AND WEAKNESSES:
Strengths:
+ msf-CNN provides a spectrum of solutions that trade memory saving with recomputation, which can be effective to memory-capacity-constraint IoT devices
+ The problem formulation (transforming it into a graph’s shortest path problem) is decent.

Weaknesses:
- msf-CNN does not seem to be clearly better than StreamNet. It would be great if the gist of msf-CNN can be applied to StreamNet, such as the new Pareto curves beat StreamNet.
- In other words, the heuristic solution does not seem to guarantee being optimal (considering the StreamNet case).

QUESTIONS:
* To get a bigger picture, it would be great to report the energy consumption. It is not clear if squeezing the memory usage to the extreme brings any benefit to the energy perspective. Could you report energy consumption in the evaluation?
* Can you perform an exhaustive search and report how far what you found are from the optimal configurations?
* Can you refine msf-CNN such that it can outperform StreamNet (e.g., Figure 4)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents msf-CNN, a patch-based multi-stage fusion technique for CNNs designed to run on memory-limited microcontrollers (MCUs). By modeling the fusion solution space as a directed acyclic graph (DAG), the method optimizes RAM usage and computational cost, reducing peak RAM by 50% compared to MCUNetV2 and StreamNet. Experiments on ARM Cortex-M, RISC-V, and ESP32 validate its efficiency, and the authors provide an open-source implementation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\na. Practical Significance: The work addresses a critical challenge in TinyML—enabling deep neural networks on ultra-low-memory devices—which has broad implications for AIoT applications.\\\nb. Novel and Simple Scheme: The DAG representation and pruning strategy are both novel and straightforward, enabling systematic optimization of memory and computational usage.\\\nc. Convincing Results: The approach achieves a 65% to 87% RAM reduction compared to prior work across a variety of MCUs.\\\nd. Clarity and Rigor: The paper clearly explains the problem definition, graph modeling, and algorithm design, supported by mathematical formulations and complexity analysis.\n\nMinor Weaknesses\n\na. Limited Network Scope: The method focuses solely on CNNs, overlooking emerging architectures such as transformers. This limitation is acknowledged by the authors.\\\nb. Venue Fit: While technically solid, the paper may be more suited for machine learning systems venues, given its strong emphasis on system-level optimization rather than novel learning algorithms or architectural innovations.\n\nQUESTIONS:\nHave the authors explored adapting msf-CNN to non-CNN architectures such as transformers? If so, what challenges arise (e.g., handling attention layers), and how might the approach be adapted to these architectures? A positive response demonstrating feasibility would significantly strengthen the work's impact and generality.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes an algorithmic optimization for fitting CNNs on microprocessors. It introduces the tight resourcing constraints that motivate the problem. It covers the limitations of related work as being model or architecture specific. It presents the basic idea, which is based on an inverted dataflow graph in which ops are nodes and stored weights are edges, and the optimization is cast as a graph-based algorithm. Edges are annotated with RAM usage if unfused and compute cost if fused; optimization then seeks to find a path that minimizes one cost potentially under a constraint on the other. The authors validate their work first with an analytical model, then with actual experiments. The authors show their algorithm is very flexible, and outperforms competitive algorithms in some or all cases. They also cover related work more broadly.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-- Real problem solved with a useful technique, with a clean algotihmic formulation \n-- Good use of both analytical and experimental proofs to validate\n-- Thorough treatment of related work\n-- Good structure: sets up problem, solves problem, shows value\n-- Clearly written\n\nWeaknesses:\n-- Somewhat incremental and narrow in impact (MCUs)\n-- The graphs are hard to read, especially in grayscale\n-- The analytical result table uses random number of digits for results\n\nSection by section:\n-- Introduction: section would benefit from a sentence or two describing layer fusion up front, given the importance of the idea to the rest of the explanation.\n-- Introduction: these is are tradeoffs that limit the use of fusion (why not fuse everything) which could be more clearly introduced\n-- Background: suggest a more broad strokes intro to the work on fusion, which is more or less a loop transform -- it's worth establishing that this work is novel in both the AI domain and from a basic compiler perspective\n-- High-level idea: this section is a strength of the paper and something more papers should include. Suggest introducing the RAM and ops optimization criteria in the main text rather than through the caption though.\n-- Sections 4-5: no issues\n-- Section 6: The structure in this section could made clearer: there is a jump from the idea of dual problems to two corresponding optimizers, but this jump is not well supported in the text. Suggest conveying the structure of the section in the first couple of paragraphs.\n-- Section 7: This section would benefit from an introductory sentence or two. \n-- Section 8: The experimental setup clear enough, but not well connected to the structure of section 6 -- the reader must figure out which optimization strategy is being used when from context. Also, the tables here are should have a fixed numerical format with right hand justification for ease of digestion.\n-- Section 9: related work and background field duplicative IMO\n-- Section 10: conclusion is crisp and effective\n\nQUESTIONS:\nMy general question on this paper is around significance. It's good work, but limited to applying a specific optimization for a specific hardware class. Acceptance should depend on significance relative to the field of papers submitted. \n\nDetailed suggestions in the strengths/weaknesses analysis will improve clarity of the paper, but significance alteration would take substantial work. \n\nTwo ideas for leveraging the strength of the approach in this paper to a broader problem area would be:\n(1) Is there an opportunity to apply this technique outside MCUs? \n(2) Is there an opportunity to say something significant in a larger context around the shape of RAM -- Compute curves relative to model size for common model architectures?\n\nAddressing either one well would raise my score a level but would require substantial effort.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}