Title: MobileODE: An Extra Lightweight Network
Venue: NeurIPS 2025
Accepted: yes
Keywords: depthwise-separable convolution

lightweight models

nmODE
Abstract: Depthwise-separable convolution has emerged as a significant milestone in the lightweight development of Convolutional Neural Networks (CNNs) over the past decade. This technique consists of two key components: depthwise convolution, which captures spatial information, and pointwise convolution, which enhances channel interactions. In this paper, we propose a novel method to lightweight CNNs through the discretization of Ordinary Differential Equations (ODEs). Specifically, we optimize depthwise-separable convolution by replacing the pointwise convolution with a discrete ODE module, termed the \emph{\textbf{C}hannelwise \textbf{O}DE \textbf{S}olver (COS)}. The COS module is constructed by a simple yet efficient direct differentiation Euler algorithm, using learnable increment parameters. This replacement reduces parameters by over $98.36$\% compared to conventional pointwise convolution. By integrating COS into MobileNet, we develop a new extra lightweight network called MobileODE. With carefully designed basic and inverse residual blocks, the resulting MobileODEV1 and MobileODEV2 reduce channel interaction parameters by $71.0$\% and $69.2$\%, respectively, compared to MobileNetV1, while achieving higher accuracy across various tasks, including image classification, object detection, and semantic segmentation. The code is available at {\url{https://github.com/cashily/MobileODE}}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MobileODE, a novel lightweight CNN architecture that replaces pointwise convolution in depthwise-separable convolution with a Channelwise ODE Solver (COS) module based on discrete neural memory ODEs (nmODEs). By modeling channel interactions via a learnable time-increment Euler discretization (DDE), MobileODE achieves 98.36% parameter reduction compared to conventional pointwise convolution. Two variants, MobileODEV1 (0.97M parameters) and MobileODEV2 (1.03M), outperform MobileNetV1/V2 and other lightweight networks (e.g., GhostNet, ShuffleNet) on image classification, object detection, and segmentation tasks while maintaining low computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ Novel architecture: the Discrete Derivative Estimator (DDE) with learnable time-increments yields more stable gradients and higher accuracy than fixed-step ODE solvers, while the Channel-wise ODE Solver (COS) replaces 1 × 1 point-wise convolutions, dramatically cutting channel-interaction parameters relative to MobileNet V1/V2.\n+ Consistently strong empirical results across image classification, object detection, and segmentation.\n+ The ODE-based variants match or surpass MobileNet latency despite using far fewer parameters.\n\nWeaknesses\n- Performance gains taper off at large expansion factors (e ≥ 9 in Table 6), hinting that the bilinear interpolation used for channel reshaping may bottleneck scalability.\n- COS updates channels sequentially (Algorithm 1), which limits parallelism; for long ODE trajectories (large L) this can increase inference latency.\n- The “extra-lightweight” claim is overstated: ShuffleNet V1 (1.1 M parameters) is still smaller than MobileODE-V1 (1.14 M) as shown in Table 3.\n\nQUESTIONS:\n- What the performance difference between DDE and higher-order solvers like Runge-Kutta? Table 7 only shows FLOPs and latency.\n- It is possible to apply COS modules to NLP or speech?\n- Why omit MobileNetV3/V4, MobileOne? How does COS compare to dynamic convolution or attention-based channel mixers (e.g., in EfficientFormerV2)?\n- COS reduces parameters but increases FLOPs (Tab 7). Is this favorable on edge devices? Are there any energy consumption support on edge devices?\n- Why $\\sqrt{C}$ factorization for $\\Phi$? Ablate against linear (C×k) or low-rank (C×r×r) alternatives.\n- Why not test on full ImageNet-1K? Accuracy on IN-tiny (64×64) may not reflect real-world performance.\n\nI give a borderline score currently, may change my rating once the authors answered my concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for making MobileNet CNNs more lightweight by using Ordinary Differential Equations as a replacement of the pointwise convolution in depthwise seperable convolutions. The authors apply their technique to MobileNetV1 and MobileNetV2 and provide experiments for classification tasks, semantic segmentation, and objects detection mainly using #Params besides Accuracy/mIOU/mAP to back up their claim.\n\nSTRENGTHS AND WEAKNESSES:\nThe substantial experiments indicate that the proposed COS module significantly reduces model parameters while even outperforming the AI performance achieved by the respective baselines (MobileNetV1 and V2), as well as more recent MobileNet variants (e.g. MobileNetV3, V4 and ShuffleNet). The high quality of the method, the explanation of its mathematical background, the hyperparameter ablation study of $L$, $\\delta_t$ and $e$, and the provided source code make this a significant and novel contribution.\n\nThe part of the evaluation that focuses on computing efficiency is rather brief, especially since, as the authors also state, ODE methods have a high computational complexity. At the very least, the authors could specify the type of hardware on which the latency numbers were achieved and whether they represent averaged execution times. If so, they should also provide the standard deviation and clarify whether the measurements were taken per sample or per batch.\n\nFigure 2: The visual difference between (a) and (b) is very subtle. This could be improved by introducing a colour to guide the reader to the different delta t values.\n\nMinor inconsistency: in subfigure c, \"Accuracy (%)\" is defined between 0.0 and 1.0, but this should be between 0 and 100%, as in the rest of the paper.\n\nQUESTIONS:\n- Why did the authors provide ODEV variants for MobileNetV1 and V2, but not for the more recent versions, V3 and V4? Instead, they only used these for comparison in the experiments.\n\n- Minor question regarding an inconsistency between the paper and the provided source code: In lines 144 ff. of the paper, the authors explicitely state that \"These increments are then passed through a ReLU6 activation function to ensure they remain non-negative and to limit their output range.\" However, in the provided source code (cvnets/modules/mobilenetv2.py, line 259, CustomActivation class), something slightly different seems to be done (torch.maximum(torch.tensor(self.epsilon, device=x.device), x)), with the ReLU6 part being commented out above (line 258). While this implementation is basically a slightly modified version of ReLU and should still ensure that the values in x remain non-negative (given that the default value of epsilon is set to 1/num_layers), it does not enforce an upper limit on x, as ReLU6 does, but instead leaves the values in x unbounded.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MobileODE, a lightweight network architecture designed to improve the efficiency of depthwise-separable convolutions. The core idea is to replace the inefficient pointwise convolution component with a novel, parameter-efficient module named the Channelwise ODE Solver (COS). The COS is based on a simple yet efficient direct differentiation Euler algorithm. The paper introduces two variants of COS-based blocks (COS-base and COS-inv), and evaluates their performance on various tasks. The results show a significant reduction in parameters for channel interaction while achieving competitive or even improved accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper addresses an important problem.\n- The proposed method is evaluated comprehensively across different tasks and datasets, which helps demonstrate its generality.\n- Experimental results are promising in terms of both model size reduction and accuracy. \n\n**Weaknesses:**\n- The experimental details for the latency benchmarks are insufficient. Since the proposed method increases FLOPs but is reported to be faster, it is important to clarify the evaluation setup (e.g., CPU/GPU model, memory) and understand if the reported latency improvements are specific to one type of hardware or if they generalize to other common accelerators like GPUs, NPUs, and TPUs, which are highly optimized for standard convolutions. \n- The motivation could be made stronger. For example, providing a breakdown of the computational cost or latency contribution for depthwise and pointwise convolutions would better highlight the inefficiency of pointwise convolution and justify the design choice. Similarly, showing the latency before and after replacing pointwise convolution would help quantify the benefit of the proposed method more clearly.\n\nQUESTIONS:\nPlease solve the questions in weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MobileODE, a novel and ultra-lightweight convolutional neural network (CNN) architecture designed to significantly reduce model size and computational cost while maintaining or improving accuracy. At its core, MobileODE revisits the standard depthwise-separable convolution architecture popularized by MobileNet, and replaces the pointwise (1×1) convolution — traditionally used for channel mixing — with a lightweight, learnable Channelwise ODE Solver (COS).\n\nThe COS module is built using a custom numerical solver called the Direct Differentiation Euler (DDE) algorithm. Unlike traditional discretization schemes that rely on fixed time steps, DDE introduces learnable time increments (∆t) that are optimized during training. This adaptive mechanism allows the model to fine-tune the temporal evolution of the feature transformations, enabling stronger inter-channel interactions with far fewer parameters. The result is a principled and trainable alternative to pointwise convolutions that operates at linear complexity with respect to the number of channels.\n\nTo showcase the versatility of the approach, the authors propose two variants:\n\nMobileODEV1: Replaces pointwise convolutions in MobileNetV1-style blocks with the COS module, preserving the basic block structure.\n\nMobileODEV2: Introduces the COS-inv block, inspired by MobileNetV2's inverted residual bottlenecks, and strategically applies COS before and after depthwise convolutions to enhance non-linear capacity and channel expressiveness.\n\nThe proposed models are evaluated across a diverse set of computer vision tasks, including image classification, semantic segmentation, and object detection, using datasets such as CIFAR-10/100, ImageNet-R, Tiny ImageNet, ADE20K, and PASCAL VOC. Experimental results show that MobileODEV1 and MobileODEV2 consistently outperform or match their MobileNet counterparts while achieving a ~70% reduction in parameters for channel interactions. Moreover, when paired with lightweight self-attention blocks (e.g., MobileViT), the models achieve state-of-the-art accuracy under strict efficiency constraints, all with lower inference latency and memory usage.\n\nSTRENGTHS AND WEAKNESSES:\n** Strengths\n* Novel Architectural Contribution\nThe paper proposes a new module, Channelwise ODE Solver (COS), which replaces traditional pointwise convolutions in depthwise-separable CNNs. COS is implemented via a novel Direct Differentiation Euler (DDE) algorithm, enabling learnable and adaptive time steps (∆t) during training. This introduces a principled, lightweight mechanism for inter-channel communication with strong theoretical and empirical justification.\n\n* Significant Parameter Reduction\nFor MobileODEV1 and MobileODEV2, the method reduces total channelwise parameters by 71.0% and 69.2%, respectively — a notable compression rate that doesn’t sacrifice accuracy. \n\n* Strong and Diverse Empirical Results\nImage classification (CIFAR-10/100, ImageNet-R, Tiny ImageNet)\nSemantic segmentation (PASCAL VOC 2012, ADE20K)\nObject detection (BUSI, FFE)\n\nResults hold across both general-purpose datasets and challenging domain-specific benchmarks, reinforcing the generalizability of the approach.\n\n\n** Weaknesses\n* No Evaluation on Full ImageNet-1k. The model is tested on ImageNet-R and Tiny ImageNet, but not on standard ImageNet-1k, which is the primary benchmark for mobile models. This makes it difficult to judge scalability and robustness on high-resolution, real-world tasks.\n* Limited Comparison to Competing ODE Frameworks. The authors briefly contrast COS with standard ODE solvers (e.g., Runge-Kutta) in latency/FLOP terms, but do not compare to existing ODE-inspired architectures like Neural ODEs, NODE-ResNets, or recent state-space models.\n* Scalability Constraints of COS. The paper notes that COS involves sequential computation across L discrete layers, which limits parallelism and may hinder scalability.\n\nQUESTIONS:\n** Typos & Formatting Issues\n- Algorithm 1 has a typo 'Reture' instead of Return. \n- small and overly compressed Figure 3.\n\n Q1: Why was standard ImageNet-1k training omitted from the experiments?\n\nWhile the proposed MobileODE variants show promising results on CIFAR, ImageNet-R, and Tiny ImageNet, it is unclear how well the method scales to full-resolution, large-scale datasets. Given that ImageNet-1k remains the standard benchmark for evaluating lightweight architectures (e.g., MobileNetV1–V4, MobileViT), could the authors clarify why ImageNet-1k training was not included? Are there challenges with scalability, convergence, or runtime that prevented this evaluation?\n\nQ2: Could the authors provide a detailed analysis of how increasing the number of DDE steps (L) affects inference latency, not just accuracy?\nWhile the ablation study in Section 4.4 explores the effect of L on accuracy, it would be helpful to see how this impacts latency and runtime. Since COS involves sequential updates, increasing L may introduce bottlenecks in real-time deployment.\n\nQ3: The use of ImageNet-R partially addresses robustness to distribution shift. Have the authors considered evaluating on additional robustness benchmarks such as ImageNet-C (corruptions), ImageNet-A (natural adversarial examples)?\nGiven the attractor-based design of COS and the dynamic evolution of features over discrete layers, it would be valuable to know whether the proposed method improves stability under noisy or corrupted inputs beyond stylized domains like ImageNet-R.\n\nQ4: While the paper introduces COS as a multi-step, attractor-based channel evolution module, it is not entirely clear how feature representations evolve across the L steps. Since the COS module is inspired by discrete ODE solvers and functions similarly to a recurrent update, it would be highly valuable to include visualizations or analyses that help interpret this process. For instance: \n- How different are the intermediate representations at each step, especially for small vs. large L?\n- Are later steps primarily refining the signal, or do earlier steps contribute more heavily to the final output?\n- How does the internal state y_l change over time across the L steps?\n\nSuch analysis could take the form of:\n- Activation similarity matrices across COS steps\n- t-SNE or PCA visualizations of intermediate outputs\n\nLIMITATIONS:\nWhile the paper briefly acknowledges some limitations in the conclusion (such as the sequential nature of the COS module limiting parallelism when L is large) and appendix, there is no dedicated Limitations section, and several important points are under-discussed. \n\n- The authors could explicitly discuss trade-offs between the number of discrete steps (L) and inference latency, memory usage, and deployment feasibility, particularly since the COS design introduces a recurrence-like structure (see Q2).\n\n- The method assumes that the number of channels C can be reshaped into a square matrix (i.e., C = k × k). This imposes structural constraints that may limit generalizability to arbitrary architectures or channel sizes (it would be specially difficult to combine with NAS where the channel size is learned and not fixed).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-22"}