Title: ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal deep learning

dynamic neural networks

compute limited systems
Abstract: Multimodal deep learning systems are deployed in dynamic scenarios due to the robustness afforded by multiple sensing modalities. Nevertheless, they struggle with varying compute resource availability (due to multi-tenancy, device heterogeneity, etc.) and fluctuating quality of inputs (from sensor feed corruption, environmental noise, etc.). Statically provisioned multimodal systems cannot adapt when compute resources change over time, while existing dynamic networks struggle with strict compute budgets. Additionally, both systems often neglect the impact of variations in modality quality. Consequently, modalities suffering substantial corruption may needlessly consume resources better allocated towards other modalities. We propose ADMN, a layer-wise Adaptive Depth Multimodal Network capable of tackling both challenges - it adjusts the total number of active layers across all modalities to meet compute resource constraints, and continually reallocates layers across input modalities according to their modality quality. Our evaluations showcase ADMN can match the accuracy of state-of-the-art networks while reducing up to 75% of their floating-point operations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a layer wise adaptive multimodal network that has two innovations: 1. the proposed method is able to work within the strict bound of varying compute budgets, where most existing dynamic methods on optimize for average case efficiency. 2. the proposed method is able to adaptively assign compute resource to different input modality, based on their quality of information (QoI), such that the input modality with less corruption is assigned more resources, to achieve optimal performance. The experimental results show that the proposed method indeed achieve better computation consumption / performance tradeoff comparing to a series of non-adaptive baselines, positively supporting the claimed strengths.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well motivated. As in resource limited platforms, optimizing the model within the bound of strict computation budget is more important than optimizing for average case efficiency; Also, modality-quality aware compute allocation is also important to ensure the information among input modalities are reliably utilized, and is a less explored research direction in dynamic network design domain.\n\n2. The method proposed in the paper makes intuitive sense. Firstly, the method finetune the model with layerdrop regularization, ensuring that the network performance will not be affected by layer skipping operation; Further, in the controller training stage, to make sure that the control module is aware of the QoI of the input modality when making compute allocation decision, the paper proposes to add corruption aware prediction loss in supervised setting, or auto-encoding like reconstruction loss in unsupervised setting. And the ablation results show that the added losses indeed facilitate the controlled to make more QoI informed layer allocation predictions.\n\n3. The experimental results show that under the same computation budget, the proposed method achieve better performance than non-adaptive baselines and baselines that are trained from scratch, it also shows that the added computation load of the controller still guarantees high throughput and negligible FLOPS increase. Together the results show that the proposed controller can distribute the compute resource among modalities optimally with acceptable increased model complexity.\n\n4.  The limitations of the current version of the proposed method is clearly discussed.\n\nWeaknesses:\n1. The experiment only tests the proposed method when the feature backbones have 12 layers, so it's not clear whether the controller can still work well with increased backbone depth, or how much additional capacity are needed in the controller in order to learn budget allocation with deeper backbones.\n\nQUESTIONS:\n1. Will the proposed controller show the same competitive performance against the baselines when the depth of feature backbones are increased?  since more complex backbones require more sophisticated decision mechanism, I assume this means that the capacity of the controller will need to be increased as well in order to make well informed decisions. is it possible to show, e.g., when the feature backbones are 16 or 20 layers, how many more layers in controller are needed for optimal performance, and whether the increased controller load is still acceptable throughput / FLOP wise.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes a way to automatically adapt computation by selecting layers  automatically depending on input quality and computational budget, in a multimodal setting. The main contribution is the use a controller that selects, for each modality, which layers to use. The distribution of layers to choose is automatically learned end-to-end by minimizing the target task loss, and depends on the input data and how much informative is the input, for instance an input which is distorted (e.g. with gaussian blur) will use fewer computation than an an input which is clean.

STRENGTHS AND WEAKNESSES:
The paper is well written and well motivated, dynamic computation in the presence of inputs of varying  quality within strict and constrained resources is an important topic to study. The controller and thus layer selection is trained end-to-end to minimize task loss, which is quite elegant.

Table 5 show that removing the corruption supervision make the results worse. Since the layer selection is trained end-to-end to minimize  the task loss, why can’t the controller figure out automatically to not leverage computation on e.g. severely distorted samples ? could it be related simply the controller network capacity?

The choice of baseline of models is reasonable. However, the Naive Scratch and MNS baselines
are trained from scratch, while ADMN uses pre-trained networks. Could it be that the difference in performance come from the use of pre-trained models ? 

Although the controllers are lightweight, it seems redundant to train a controller for each single layer budget. Would it be feasible to have a single controller that generalizes across budgets ?

QUESTIONS:
Please refer to Strengths And Weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper introduces ADMN, a layer-wise Adaptive Depth Multimodal Network designed for dynamic environments with fluctuating compute resources and input quality. ADMN adjusts the number of active layers across modalities based on resource constraints and reallocates computation based on modality quality. It achieves similar accuracy to state-of-the-art models while reducing computational cost by up to 75%.

STRENGTHS AND WEAKNESSES:
- The exact novel contribution is not clear. LayerDrop finetuing is based on the existing LayerDrop method, while more clarity on autoencoder-based initialization in QoI-aware training would be helpful. 
- Novelty is rather limited. 
- Acknowledging this existing work would benefit the paper: Cai, Han, et al. "Once-for-all: Train one network and specialize it for efficient deployment." arXiv preprint arXiv:1908.09791 (2019).
- The topic of the paper is quite relevant from the point of view of practical deployment on edge devices.
- Limitations have been discussed well.

QUESTIONS:
- Including the main conceptual differences from Mixture-of-Experts approach would be helpful.
- Paper would benefit from backing the motivation behind autoencoder-based initialization (lines 211 - 213) with some evidence.
- In stage 2 of controller training, why is the backbone frozen?
- Minor: Typo in line 2? ‘Afforded’ -> ‘offered’?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03