ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources
Authors: Jason Wu
Yuyang Yuan
Kang Yang
Lance M. Kaplan
Mani Srivastava
Venue: NeurIPS 2025 poster ()
Keywords: multimodal deep learning
dynamic neural networks
compute limited systems
Primary area: applications
TL;DR: ADMN enables layer-wise allocation of resources among modalities at inference time according to modality quality for a fixed resource budget.

Abstract: Multimodal deep learning systems are deployed in dynamic scenarios due to the robustness afforded by multiple sensing modalities. Nevertheless, they struggle with varying compute resource availability (due to multi-tenancy, device heterogeneity, etc.) and fluctuating quality of inputs (from sensor feed corruption, environmental noise, etc.). Statically provisioned multimodal systems cannot adapt when compute resources change over time, while existing dynamic networks struggle with strict compute budgets. Additionally, both systems often neglect the impact of variations in modality quality. Consequently, modalities suffering substantial corruption may needlessly consume resources better allocated towards other modalities. We propose ADMN, a layer-wise Adaptive Depth Multimodal Network capable of tackling both challenges - it adjusts the total number of active layers across all modalities to meet compute resource constraints, and continually reallocates layers across input modalities according to their modality quality. Our evaluations showcase ADMN can match the accuracy of state-of-the-art networks while reducing up to 75% of their floating-point operations.

ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources
Jason Wu Electrical and Computer Engineering University of California, Los Angeles jaysunwugg. ucla.eclle
Yuyang Yuan Electrical and Computer Engineering University of California, Los Angeles
yuanyuyangāg wwwa.lllaeeeoooo.
Kang Yang
Lance Kaplan
Electrical and Computer Engineering University of California, Los Angeles kyang738g. ucla.edi
DEVCOM Army Research Laboratory
US Army
lance m...e.m.oommo kaplan. ci civearmy mil
Mani Srivastava"
Electrical and Computer Engineering
University of California, Los Angeles mbsčucla, edu
Abstract
Multimodal deep learning systems deployed in dynamic scenarios due to the bustness afforded by multiple sensing modalities. Nevertheless, they struggle with varying compute resource availability (due to multi-tenancy, device heterogeneity, etc.) and fluctuating quality of inputs (from sensor feed corruption, environmental noise, etc.). Statically provisioned multimodal systems cannot adapt when compute resources change over time, while existing dynamic networks struggle with strict compute budgets. Additionally, both systems often neglect the impact of variations in modality quality. Consequently, modalities suffering substantial corruption may needlessly consume resources better allocated towards other modalities. We pro- pose ADMN, layer-wise Adaptive Depth Multimodal Network capable of tackling both challenges it adjusts the total number of active layers across all modalities meet strict compute resource constraints, and continually reallocates layers across input modalities according to their modality quality. Our evaluations showcase ADMN match the accuracy of state-of-the art networks while reducing up 75% their floating-poie operations.
Introduction
Background: Multimodal deep learning systems fusing sensory data from various modalities are the standard for accurate, robust sensing [ Accordingly, these multimodal systems are invaluable (Qol) can vary drastically across samples. Low Qol data is corrupted, noisy, or otherwise degraded manner reducing its informativeness. Fluctuations in modality's 's Qol occur slowly (e.g., lighting conditions over the day) or rapidly (e.g., battlefield settings or unstable sensor feeds).
in highly dynamic environments, where given input modality's sufultt--o-informaate
Mani Srivastava holds concurrent appointments as a Professor of ECE and CS (joint) at the University of California, Los Angeles, and as an Amazon Scholar at Amazon. This paper describes work performed at UCLA and is not associated with Amazon.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Although multimodal deep learning systems are generally robust to variable Qol, key challenge surrounds the computational efficiency. Most state-of-the-arr multimodal networks employ static provisioning in which multimodal inputs are processed by fixed architecture 3 4 regardless of their individual utility. Consequently, valuable resources may wasted on low-Qol modalities. Recent work has explored dynamic networks that train policy networks to reduce computation for easy samples [5] 6 gl. Unfortunately, explicit consideration of highly dynamic Qol variations among modalities has been largely overlooked. We hypothesize that flexibly allocating computational resources among modalities in accordance with each modality's 's Qol on a per-sample basis can substantially improve model performance in compute -limited settings.
Challenges:
Additionally, current works also neglect the challenge of dynamic compute resource availability. The environments where multimodal systems are most relevant often impose temporally variable but strictly bounded compute budgets. The maximum budget varies with time according to factors such as thermal throttling, energy fluctuations, or multi-tenancy, and cannot be exceeded at any given moment. Neither statically provisioned models nor dynamic networks are equipped to function under such constraints. Most dynamic networks optimize for average-cass efficiency, without mechanisms to constrain the worst-case compute beyond the total cost of the full network. The only partially
compatible works are those performing model selection with gafing networks [6 g 22, which can be
adapted to varying compute resource availability by training a set of models for each budget. Aside from requiring unreasonable amount of training resources, it also impedes the standard practice of loading pretrained weights prior to finetuning [10]. as there do not exist publicly available pretrained weights for every possible compute budget. We hypothesize that single network initialized with pretrained weights, which can dynamically adjust its resource usage, offers an effective solution to the challenge of fluctuating compute resources.
We propose ADMN, an Adaptive Depth Multimodal Network jointly tackling the challenges of adaptation to both dynamic compute resources and variable Qol in- puts. While these challenges are agnostic to the multimodal fusion method (c.g., data-level [[]] cmbedding-level [12], and late [T3), we focus specifically on embedding-level fusion due to
Proposed Solution:
Clean image
INol
Noisy Death
Fuee
Clean Depth
applicability and ease of implementation. Figure provides high-level depiction of ADMN. Fol- lowing the standard for embedding-level fusion, each modality is processed with an independent backbone before undergoing fusion with a trans- former encoder.
Computee budget: Layers
Compute budget: Layers
Figure I: Overview of ADMN. Variable depth back- bones adapt to both changing compute resources and input noise characteristics
First, ADMN addresses the challenge of dynamic
compute resources through adaptive backbones containing adjustable layer configurations. With the same set of model weights, ADMN activates a subset of backbone layers according to the available compute resources. We ensure compatibility with pretrained weight initializations by injecting the LayerDrop technique [4] into both the backbone pretraining and multimodal network finetuning processes. This requires the adaptation of the unimodal, text-only LayerDrop technique to not only Vision Transformers, but also multimodal networks. Such a strategy produces a novel multimodal network whose backbones are resilient to missing layers at test-time, allowing for the usage of fewer layers during resource scarcity.
Second, given established layer budget, ADMN addresses the challenge of dynamic Qol by adapting the choice of selected backbone layers according to each modality's 's Qol. ADMN Icarns a multimodal controller top of the adaptive backbones to perform layer allocation. Unlike existing works, we explicitly structure the controller around dynamic Qol by introducing into the training loss. However, as it is reliant on the presence of labeled corruption information during training, we propose an alternative autoencoder-bel based initialization (ADMN _MM to emphasize the Qol without corruption labels. The controller is trained end-to-end for particular layer budget through Gumbel-Softmax Sampling [5] and the straight-through estimator (16).
peruptiionawaar supervision
We benchmark ADMN against several basclines to demonstrate its ability to preserve accuracy while simultancously minimizing energy and latency costs. ADMN is tested on both multimodal localization
2

---

Stage Training
Stage 2 Training
Stage VIT
Pretrained Image
m1
TE Fusion
Dutoct Head
Ourpt
Controller
TE Fuslon
Dulput
m2
m2
Pretrained Depth VIT
Stage
Depth
VIT
Figure 2: ADMN architecture. [Gray box]: dropped layer, [Blue box]: frozen layer, [Red box]: tunable
layer. TE: Transformer Encoder.
and classification tasks with realistic sensor corruptions representing dynamic Qol, reinforcing its generality and applicability. ADMN can match the performance of larger state-of-the-arr models while reducing FLOPS by up to 75% and latency by up to 60%. We release our code at https Our contributions summarized below:
ceitthub.com/esll/ ADMM
We present the first layer-wise adaptive multimodal network where resource allocation among modality backbones is dictated by Qol characteristics and temporally variable but strictly bounded compute budgets at inference time for every sample. We evaluate ADMN with realistic sensor corruptions modeling dynamic Qol.
We create general framework for training layer-adaptive multimodal networks by injecting the LayerDrop technique into both unimodal backbone pretraining and multimodal finetuning, and systematically evaluate it across various modality combinations and datasets.
We design multimodal perceptual controller that explicitly attends to modality Qol through either corruption-awen aaaero supervision when training-time corruption labels are provided, or autoencoder- based initialization when they not.
We introduce low-complexity method training a multimodal controller that meets strictly bounded compute budgets instead of reducing average-case computation.
2 Related Work
Early Exiting in Unimodal Networks. Early Exiting has been explored extensively in unimodal
networks to improve the average-cuse inference efficiency [1]]8]
18 5]. Methods like DeeBERT [17]
and PABEE [8] use confidence thresholds to halt computation for simpler inputs. However, extending such techniques to dynamic multimodal Qol and strictly bounded, variable compute budgets nontrivial. Specifically, they neglect to consider the impact of low-Qol samples setting, while ADMN addresses the complex interplay dynamic multimodal Qol. Morcover, Early Exiting is unable to constrain the worst-case inference computation beyond the total network cost, clashing with the need to accommodate strietly bounded compute budgets.
in the unimodal
Unimodal Sub-Networks. Instead of performing input-aware network adaptation, other works prune single large network runtime accommodate varying computational resources [9 Once-For-All [19 proposed algorithm pruning large network across depth, width, kernel size, and resolution, showcasing competitive performance against state-of-the-a neural architectural search baselines. LayerDrop [14] introduced layer-wise dropout rate during pretraining for on-demand layer reduction at test time. We leverage LayerDrop as the foundation for ADMN, but emphasize that ADMN's contributions can easily be applied to other inference-ting pruning techniques.
Dynamic Inference for Multimodal Systems. To address the inefficiency of static multimodal networks, dynamic networks {8
6 20 21) leverage input-dependent forward paths. DynMM g and DynaFuse train set of expert networks with different modalities and perform network selection at inference time. AdaMML [ and Listen to Look (20) leverage multimodal information to eliminate temporal redundancy in videos. ACF dynamically replaces certain modules with lightweight networks according the input for greater efficiency. Despite these advances, existing methods (1) overlook significant QoI variations arising from corruption, (2) utilize or discard entire modalities without fine-grae -rainnd control, and (3) fail to consider fixed resource budgets. ADMN addresses

---

these gups by introducing two methods of performing layer allocation with an explicit focus on relative modality Qol. while also accounting for strictly bounded compute budget.
3 Methodology
3.1 Problem Description
In real-world multimodal systems, sensor data suffers from dynamic Qol variations. Environmental factors such as inclement weather and lowlight can corrupt the utility of a modality, while sensor- specific factors such camera ISO audio gain also play factor. Additionally, environmental Qol corruptions may unequally affect modalities. For example, fog may affect vision (heavily) and depth (partially), but not audio. Furthermore, also impose strictly bounded, time-varying compute constraint to emulate factors such as thermal throttling or variable available energy. Thus, the goal of ADMN is to perform correct modality resource allocation according to the diverse sensor Qol variations, all while ensuring that the total allocated resources adhere to a strict budget.
A typical embedding-lewee fusion multimodal network is illustrated on the left side of Figure It extracts unimodal embeddings with moodality-speciffe backbones, condenses them into one joint embedding via self-attention with Transformer encoder, and obtains output from the joint embedding. ADMN builds upon this architecture by proposing two novel advancements. First, we introduce layer-wise adaptive maltimodal network comprised of dynamic modality backbones, thus enabling inference-time adaptation to any layer budget. Second, ADMN proposes a Qol-aware controller that dynamically allocates the layer budget optimally among modality backbones (i.e., transformer layers), allowing it to greatly outperform static models with the same layer budget.
3.2 ADMN Architecture
Figure shows the task-agnostic architecture of ADMN, which involves a two-stage training process.
Stage 1: LayerDrop Finetuning. We first initialize each unimodal backbone with a set of general weights (e.g., ImageNet weights) pretrained with LayerDrop [4] Subsequently, the multimodal network is finetuned with LayerDrop on the desired task while freezing the earlier backbone layers (shown in blue) to prevent overfitting. Stage I's objective to adapt the pretrained weights the specific target task, while also training the later Fusion and Output layers to accommodate diverse embeddings arising from various backbone layer configurations.
Stage 2: Controller Training. We freeze all the Stage 1 network weights and train a controller with either corruption-aware supervision autoencoder initialization to allocate fixed budget of L layers in accordance to each sample's relative modality Qol. From the multimodal inputs, the controller outputs discrete sequence summing L representing the selection of backbone layers.
3.3 Stage LayerDrop Finetuning
We adapt LayerDrop [4) originally designed for unimodal text transformers, support arbitrary multimodal transformers through integration into unimodal pretraining and multimodal finetuning.
3.3.1 Vanilla LayerDrop
LayerDrop I4 enabled on-demand test-time depth reduction of textual transformers by training with layer-wise dropout rate. By stochastically dropping out the transformer layers, the network is trained to function with only a subset of layers. For an inference-time layer budget L, they introduced an "every-other" dropout strategy where layers are dropped in an alternating fashion starting from the middle. We refer readers to the original work for greater detail.
3.3.2 Multimodal LayerDrop
ADMN extends LayerDrop to Vision Transformer-style networks (ViT) 22 by first integrating them into the unimodal backbone pretraining process. We pretrain 12-Layer visual and audio backbones on the ImageNet-1k (23) and AudioSet (4 datasets, respectively. Rather than performing supervised training, which suffers from convergence issues, we pretrain both backbones with the Masked

---

26] We enforce a LayerDrop rate of 0.2 in each layer of the ViT encoder while fixing the decoder. By employing LayerDrop MAE pretraining. the model learns to reason in the presence of missing layers.
Autoencoder (MAE) approach [25
Subsequently, the MAE pretrained weights are loaded into the appropriate backbones of multimodal (e.g., vision-depth) neural network to be finetuned on specific corrupted dataset. As shown in Figure the majority of the early backbone layers are frozen during finetuning, with later layers adjusting to the new task. A LayerDrop rate of 0.2 is maintained during the finetuning process in all the backbones, allowing the remaining learnable layers (Fusion and Output) to adapt to the countless combinations of missing layers. This process ultimately creates a task-specific multimodal network supporting adaptive backbone layer configurations inference time.
bonee LayerDrop. One unique challenge with multimodal I LayerDrop is the need to subject individual backbones extreme dropout conditions. While dropping all layers in unimodal network is rare, one may frequently drop all layers a heavily corrupted modality in a multimodal network. Unfortunately, the typical LayerDrop training ratio of 0.2 standard 12 layer ViT is unlikely to expose the network to such a case during training. To remedy this, we employ full-backbone dropout during training time, establishing a 10% chance that all layers of given modality's 's backbone will be dropped out independently of the 0.2 LayerDrop rate.
3.4 Stage 2: Controller Training
The controller selectively activates backbone layers in the frozen Stage network according the relative modality Qol and total layer budget I. on a per-sample basis to minimize task loss, as shown in Figure 2 Ideally, the controller should also remain lightweight in operations, latency, and size.
3.4.1 Controller Architecture
In contrast to previous works 21 g which train controllers to reduce computation in the average case with regard variable Qol or fixed resource budgets, ADMN's controller is explicitly con- structed around both factors. Figure reveels the structure of the controller. First, we downsample the inputs and pass them through modality specific lightweight convolutional networks. These convolutions aim to produce embeddings containing information solely regarding each modality's Qol, which is sufficient from low-resolution data. With M modalities, the convolutional networks produce M total noise embeddings, which are fused by a transformer encoder into one embedding Ccore representing the Qol of every input modality. From Ccorr. we can obtain set of raw logits
MLP, & (ecorr) where RC C=
Σ bii
(I)
where |b/] is the total number of backbone layers for modality my In essence, represents an allocation of L available layers among C' total backbone layers, with values dependent on the Qol characteristics of every input sample.
3.4.2 Qol-Aware Training
Ideally, the convolutional layers will automatically focus on each modality's 's Qol to provide correct layer allocations from the task-specific loss Lmodel Nevertheless, we later show in Section [4.4] supervision with purely the task loss is insufficient, where the controller fails to properly attend to Qol and outputs unsuitable layer allocations (Section[A.7) Thus, we introduce two methods enabling Qol-awareness: corruption-awan supervision when the training dataset contains Qol labels, and autoencoder--aase initialization (termed as ADMN ADMN_AE) when they do not.
Corruption-Aware Supervision. We supervise the controller with an additional corruption loss Lcorr. If the corruption is quantifiable (c.g., camera ISO, Gaussian Noise), we predict corruption vector G MLP corr (Crore) containing the corruption values for all modalities. The corruption loss is the MSE loss Lcorr MM I0m, Jm, I. where represents the ground truth corruption value. If the corruption is difficult to quantify but can be grouped into K categories (e.g., weather annotations), we obtain a set of logits f E R where f = MLP cor (err(Ecrr)) and set Leorr as the cross-entropy loss between f and the category labels. We optimize over the joint loss Liotal Lmmade + Loorrs which explicitly instructs the model to attend to the modality Qol.
5

---

3 Layer Conv
ecorr
ADMN Corruption
MLP
m1
2.123
TE Fusion
Um
0.5
m2
ADMN AE Pretraining
Layer Deconv
1.32
[0.01
0.05
1.02
0.21
Loss
0.08
n.67
0.17
Layer MLP
0.80
Gumbel-
0.05
Softmax
Top-L
Recon. Loss
Sampling
Discretization
Figure 3: Detailed depiction of the ADMN controller.
Autoencoder-Basee Initialization. In many scenarios, there is lack of any ground truth Qol infor- mation. The key intuition of the cutoencoder-basse initialization approach that the compression and reconstruction objectives of autoencoders result in well-organized latent space that group similar samples together (27 (28). Since the Qol corruptions are significant enough to impact downstream accuracy, they are vital to the reconstruction objective. Consequently,
autoencoder trained variable Qol data is likely to structure the latent space in a manner that reflects each sample's Qol properties. Figure fillustrates the structure of ADMN ADMN_AE's autoencoder pretraining. The encoder is comprised of the controller's perceptual components (i.e., convolution layers and fusion transformer), while the decoder performs reconstruction from Ceorr with modality specific deconvolution layers. With autoencoder pretraining, the controller's perceptual layers learn to attend to input modality Qol without the need for any Qol labels. We then load the encoder weights into the controller, freeze them to retain Qol understanding, and learn the layer MLP allocations with the singular loss Lmodel-
3.4.3 Differentiable Layer Selection
Although the logits ı provide information on which layers to select, activating given layer is binary decision. Such categorical decisions interrupt the flow of gradients and prevent end-to-end training. Traditionally, unimodal carly-exit networks [5] employ Gumbel-Softmay Sampling [[5] to approximate categorical distribution while retaining differentiability. These networks employ decision networks at every layer of the model and leverage whether to execute that particular layer. Coupled with resource- aware loss, these techniques can train dynamic networks that reduce computation in the average case easy samples. However, ADMN substantially differs from previous dynamic networks in that ADMN predicts (1) the entire allocation of layers at the beginning of model execution that (2) sum to hard layer budget L. Simply applying Sampling to ADMN is insufficient, as the approach cannot select total layers among all modality backbones.
Sampling to decide
We devise method to select L total layers in a differentiable manner. Initially, we perform standard Gumbel-Softman sampling with a temperature of to obtain This allows multiple high-value logits be represented in the resulting probability distribution, better accommodating the selection of layers. This is accomplished at the expense of the highly desirable near-discn dear-discrete behavior of the low-temperature Gumbel-Softrax We introduce subsequent top-L. discretization on y to activate only I. layers, and maintain gradient flow through the straight-throughh estimator [16]. The downstream layers receive the discretized value, while the gradients received are copied over to the continuous logits allowing for gradient flow. We provide additional justification in Appendix A.2

---

Table 1: GDTM Localization Error (em) (L) with 6-16 Layers of Budget
GDTM Dataset
Gaussian
Lowlight
Blur
Method
6
12
16
6
12
16
6
8
12
16
Upper Bound
29.6
18.8
9.4
ADMN
51.4
39.0
33.1
30.3
49.5
23.9
18.0
17.3
11.8
11.2
9.8
9.9
ADMN_A
53.6
38.4
33.5
29.4
35.3
22.8
18.7
17.6
14.0
11.4
9.8
9.6
MNS
39.3
32.1
57.5
73.0
117.5
83.9
117.6
116.8
109.5
114.2
117.7
74.0
Naive Alloc.
112.5
97.6
46.9
31.0
90.3
67.3
27.1
17.7
81.4
50.6
12.5
Naive Scratch
39.1
36.7
40.4
37.3
117.5
84.6
117.6
117.5
117.8
116.2
117.7
93.6
Image Only
67.5
53.2
53.7
56.8
45.6
46.2
15.4
11.2
10.3
Depth Only
85.0
61.7
58.6
72.1
64.9
63.4
25.7
22.6
22.3
Table 2: MMFI Classification Accuracy (1)
Table 3: AVE Classification Accuracy (t)
MMFI (Gaussian)
AVE (Rain Background Noise)
Method
8
12
16
Method
6
8
12
16
Upper Bound
44.44%
Upper Bound
71.19%
ADMN
35.03%
39.25 %
41.92%
43.31%
ADMN
57.07
62.95%
67.48%
66.60%
ADMN_AE
36.16%
38.43%
42.18%
44.40%
ADMN_AE
52.95%
62.30%
66.77%
66.67%
MNS
3.70%
3.70%
3.70%
3.70%
MNS
25.81%
26.56%
17.21%
13.34%
Naive Alloc. 5.56%
12.96%
29.01%
42.90%
Naive Alloc.
36.16%
46.89%
65.71%
67.95% %
Naive Scratch
3.70%
3.70%
3.70%
3.70%
Naive Scratch
23.94%
26.56%
18.70%
13.22%
Image Only 18.83%
18.52%
23.15%
Image Only
47.76%
54.11%
57.85%
Depth Only
17.59%
30.86%
32.72%
Audio Only
36.41%
40.52%
42.64%
Evaluations
4.1
Experimental Setup
4.1.1 Datasets and Corruptions
The GDTM localization dataset [12 involves distributed nodes each containing modalities RGB, depth, maveve radar, and multichannel audio. The target centoee-ontrolleed car on an indoor track. The MM-Fi (29] human activity recognition dataset contains 40 subjects, 27 total activities, and modalities RGB, depth, mm Wave, and WiFi. We focus on the visual modalities (RGB, depth) for these two datasets. Finally, we evaluate on the audiovisual AVE 30 dataset with 4143 videos covering 28 classes, from which we use both RGB and audio modalities. These datasets do not naturally contain varying modality Qol, so we synthetically corrupt the modalities on per-sample basis (Section A.7 Section A.3) to simulate realistic sensor corruptions as follows:
We add N(O, aiy) to each modality i's input data. Each modality defines a set of N; standard deviations {o;,, 022: ...oix,] from which oi drawn for each sample. This setting can represent systems with unstable links injecting different levels of noise, sensors with different settings such as camera ISO levels. We apply this to the RGB and depth modalities of the GDTM and MM-Fi datasets with N. = 4.
Gaussian Noise:
Rain: For the outdoor RGB samples of the AVE Dataset, we also explore simulating rainfall and haze corruption, following the technique in (31]
Lowlight: We mimic lowlight corruption through a combination of gamma correction, color shift, and additive noise [32]. We create moderately and severely impaired lowlight RGB samples from GDTM while leaving the IR depth unchanged. However, under normal lighting (RGB unchanged), we emulate a sight-saturated IR depth sensor by utilizing frame of sunlight saturated IR depth. We also add lowlight corruption to both rainy outdoor and clean indoor RGB samples of the AVE Dataset.
Blur: We employ Gaussian Kernel blurring with two kernel sizes to emulate different severity of blur on RGB images from the GDTM.
Background Noise: We mix in wind audio for outdoor events and sound from a standing fan for indoor events to corrupt the AVE dataset's 's audio modality.
7

---

4.1.2 Baselines
Upper Bound: The full 12 layers are allocated to each backbone (no dropout), representing the maximum layer budget.
Naive Allocation: Given layer budget L and M modalities, we naively allocate backbone following "every-other" allocation.
L. layers to each M
Image/Depth/Audio Only: All L layers are allocated to one modality, valid only for L S 12
Naive Scratch: We train network from scratch without LayerDrop for every layer budget L on L
the downstream task with each backbone containing M layers.
Modality Network Selection (MNS): Most existing dynamic networks ignore fixed resource budgets and cannot be directly compared to ADMN. We adapt the network selection technique from 6 9 which train a set of "expert models" and use gating network to select among them to create the MNS baseline. This technique is also frequently referred to as "mixture-of-experts" MNS trains an individual unimodal network for each modality and one multimodal network where each model is trained from scratch to conform to a specific layer budget. In contrast, ADMN requires only the training of one single backbone network with LayerDrop. scaling much more effectively with the number of budgets, while also reaping the advantage of pretrained weight initializations.
Essentially, Upper Bound, Naive Allocation, and Image/Depth/Audio Only are allocation baselines operating on the same finetuned Stage backbone, while Naive Scratch and MNS train new networks.
4.2
Main Results
GDTM Localization Error. Table highlights that both ADMN (corruption supervised) and ADMN_AE (autoencoder pretrained) drastically outperform all the allocation baselines regardless of the corrup- tion, especially at small layer budgets. Under Lowlight with layers of budget, ADMN and ADMN_AS localize within 5 cm of the upper bound, while the next best baseline incurs almost 27 cm of localiza- tion error. The performance difference is the smallest under Blur, which we attribute to the strength of the visual backbone. While the heavily blurred RGB images are uninformative to the human eye (Section (A.7), the visual backbone can still produce accurate localizations, and the controller correctly prioritizes image for layer allocation despite heavy blur. This exemplifies strength of ADMN - learning allocations from the task loss prevents errors from preconceived human bias. In comparison to Naive Scratch and MNS, ADMN handily outperforms them with the exception of 6 and 8 layer Gaussian corruption. Due to the need to train networks from scratch (cannot leverage pretrained weights for arbitrary layer budgets), these two basclines struggle to converge with deeper networks and more complex corruptions. ADMN_AES great performance also demonstrates that knowledge of the ground truth corruption is unnecessary.
Classification Accuracy. Tables and depict the classification accuracy for the MM-Fi and AVE Datasets, respectively, from 6 to 16 Layers. We observe similar trends as the GDTM dataset, but notice weaker results for the Naive Scratch and MNS baselines. These classification tasks with changing environments and temporal dependencies are considerably more complex than the single-car GDTM localization task. Consequently, the accuracy of networks trained from scratch suffers. This serves to illustrate the importance of our LayerDrop training process it enables the adaptation of pretrained networks to any arbitrary layer budget.
We show the meaningful reduction in latency (ms) and Giga-Floating Point Operations (GFLOPs) of ADMN in Figure Although the controller accounts for a significant proportion of the latency at smaller layer budgets (~ 20%), the high throughput at these latencies (> 150 fps for GDTM) surpasses most sensor sampling rates. Moreover, ADMN's 's controller utilizes negligible amount of FLOPs. The controller constitutes about 1% and 0.2% of the model's total operations for GDTM and MM-Fi, respectively, at the fewest allocation of 6 layers. When viewing Tables 1 2 and f in context of these metrics, we can observe the significance of ADMN. For instance, under Blur corruption GDTM, ADMN localizes within cm of the Upper Bound while reducing latency by ~60% and FLOPs by 75%..
Latency and FLOPs.
8

---

Base ADMN
100
ADMN
Latency (ms))
10
GFLOPs
75
Controller
Latency (ms)
30
ADMN
Controller
50
20
200
24
12
24
16
12
24
16
12
24
16
12
6
Layers
Layers
Layers
Layers
Figure Latency (ms) and GFLOPs vs Layers for GDTM (left) and MM-Fi (Right)
4.3 Extended Evaluations
Three Modalities. To ensure that ADMN generalizes to more than two modalities, we perform localization on GDTM with RGB, depth, and mm Wave radar modalities. Figure shows CDF of the error on a dataset where each modality has 50% likelihood of suffering extreme Gaussian Noise. ADMN continues perform correct allocations with three heterogeneous modalities.
Unequal Backbone Computation. For each AVE dataset sample, we process eight image frames and one large audio spectrogram. Thus, the visual backbone consumes approximately three times the FLOPs of the audio backbone. We neglected this Table 3 by assuming that backbones were equally resource intensive. In Table we present results in terms of Audio Layer Budgets, where acti- vating one layer in the image backbone costs three audio layers. In comparison to Naive Alloc., which activates the same number of real layers per modality (e.g.. Naive Alloc. 12 layers is three image and three audio layers), ADMN performs superior allocations accounting for unequal computation.
Table 5: Impact of supervision AVE Classifi- cation Accuracy (t)
Table 4: Unequal AVE Classification Accuracy
(t), budget in Audio Layers
AVE (Rain Background Noise)
AVE (Rain + Background Noise)
Method
6
8
12
16
Method
12
16
24
32
Upper Bound
71.19%
Upper Bound
71.19%
ADMN
57.07%
62.95%
67.48%
66.60%
ADMN
51.83%
64.34%
66.46%
67.68%
ADMN _OM
52.95%
62.30%
66.77%
66.67%
Naive Alloc.
36.16%
46.89%
65.71%
67.95%
Task Loss ADMN
46.20%
57.13%
64.53%
67.5 67.50%
4.4 Ablation Studies
Efficacy of LayerDrop. LayerDrop integrated into two stages initial MAE pretraining ImageNet and Stage Finetuming. Figure 6 showcases the localization error on the GDTM dataset for separate unimodal depth and image networks. We observe that adding Layer- Drop during finetuning (LD FT) has the greatest impact, but benefit is observed when added into both stages (LD Both). The ability drop many layers without significant degradation confirms that Lay- erDrop is compatible with the VIT architecture, MAE pretraining, and is also effective when finetuned on another task. We present additional results in Appendix A.4
t-SNE of GTDM Blur Dataset
Comuptions
Pedmat Mh
Impact of Controller Supervision. In Table 5 we omit both the autoencoder pretraining and the corruption supervision get variant of ADMN supervised purely by the Task Loss only. The inferior results show the importance of utilizing either corruption supervision or autoencoder pretraining.
Figure 5: t-SNE of the autoen- coder for different levels of RGB Blur on GDTM Blur
Autoencoder Latent Space. In Figure 5
we visualize how ADMN_AE's autoencoder performs corruption-awart clustering in the latent space through a t-SNE plot. We note that this behavior learned purely through data driven manner, and no information on the corruptions were provided during autonncderr training. The obvious clustering of the various corruptions reveals how ADMN AE an effective replacement for explicit corruption supervision.

---

LayerDrop in Image
LayerDrop in Depth
Image (LD Bot-)
Septt (LD Both))
1.0
140
Image
Localization Eroor (cm)
150
Jepth
Image (No L.)
Septh (No LD)
0.8
Localization Error
100
100
0.6
ADMN
0.4
Usper
Depth Only
40
86
0.2
Image Only
Only
Naive Allocation
0.0
125
Number of Layers Dropped
Number of Layers Dropped
Localization Error (cml
Figure 6: Effect of LayerDrop on 12-layer unimodal image and depth localization networks, evaluated on GDTM.
Figure 7: Localization Error for GDTM with Three Modalities
Limitations and Discussions
One drawback is that the ADMN controller is trained only for one particular layer budget L. Consequently, we must train and load multiple controllers for dynamic test-time compute requirements induced by factors such as thermal throttling. Nevertheless, the training overhead for the controllers is low (Section [A.I), enabling a controller be trained for every layer budget. The controller also constitutes only 2.3% and 3.2% of the total network parameters in the GDTM and MM-Fi tasks, respectively, allowing for easy storage on disk. Furthermore, we present some preliminary results a universal controller in Appendix A.I
Dynamic Compute Constraints:
Batched Inference: Since ADMN obtains the layer allocation prior to backbone execution, future work can exploit this for efficient batched inference. Classic adaptive models are incompatible with batched inference due to per-layer execution decisions for each sample. However, with ADMN, we can run an initial profiling stage and group samples into sub-batches based on similar layer allocation. For instance, samples with high depth noise and low image noise activate similar layers and can be grouped together.
Fusion with Early Exit: While ADMN and unimodal Early-Exit methods tackle fundamentally different problems, the two techniques can be combined for further efficiency gains. ADMN always allocates L layers across all the modalities. However, on simple inputs, all L layers may not be necessary, allowing for Early-Exit techniques to improve performance.
Conclusion
This paper proposes ADMN, a multimodal network capable of dynamically adjusting the number of active Transformer layers across modalities according to the quality of each sample's input modalities. Through this continuous reallocation, ADMN can match the accuracy far larger networks while utilizing a fraction of their operations. Additionally, the dynamic backbones of ADMN are also well suited for scenarios with adaptive compute, ranging from heterogeneous deployment devices to fluctuating energy availability. We demonstrate the superiority of ADMN compared to other baselines across both classification and localization tasks.
7 Acknowledgments
The research reported in this paper was sponsored in part by the DEVCOM Army Research Laboratory (award # W911NF1720196 ), the Air Force Office of Scientific Research (awards # FA95502210193 and (A95502310559) and the National Institutes of Health (award 1P41EB028242), The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, the funding agencies. Jason Wu was supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate (NDSEG) Fellowship Program. Mani Srivastava was also partially supported by the Mukund Padmanabhan Term Chair UCLA.
10

---

References
[1] Yi-Ting Chen, Jinghao Shi, Zelin Ye, Christoph Mertz, Deva Ramanan, and Shu Kong. Multi- modal object detection via probabilistic ensembling. In European Conference Computer Vision, pages 139-158. Springer, 2022.
[2] Andreas Eitel, Jost Tobias Springenberg, Luciano Spinello, Martin Riedmiller, and Wolfram Burgard. Multimodal deep learning for robust rgb-d object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 68 681-687,2000
[3] Mengmeng Wang. Jiazheng Xing, Boyuan Jiang, Jun Chen, Jianbiao Mei. Xingxing Zuo, Guang Dai, Jingdong Wang, and Yong Liu, A multimodal, multi-task adapting framework for video action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5517-5525, 2024.
[4] Junbo Yin, Jianbing Shen, Runnan Chen, Wei Li, Ruigang Yang. Pascal Frossard, and Wenguan Wang. Is-fusion: Instance-scene collaborative fusion for multimodal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14905-14915, 2024.
[5] Lingchen Meng. Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser- Nam Lim. Adavit: Adaptive vision transformers for efficient image recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309-12318 2022.
[6] Rameswar Panda, Chun-Fu Richard Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris. Adamml: Adaptive multi-modal learning for efficient video recognition, In Proceedings of the IEEE/CVF international conference on computer vision, pages 7576-7585, 2021.
[7] Qixuan Cai, Xiulong Liu, Kaixuan Zhang, Xin Xie, Xinyu Tong, and Keqiu Li. Acf: An adaptive compression framework for multimodal network in embedded devices. IEEE Transactions on Mobile Computing, 23(5):5195-5211 2024.
[8] Zihui Xue and Radu Marculescu. Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2575-2584, 2023.
[9] Hamidreza Alikhani, Anil Kanduri, Pasi Liljeberg, Amir M Rahmani, and Nikil Dutt. Dynafuse: dynamic fusion for resource efficient multimodal machine learning inference. IEEE Embedded Systems Letters, 15(4):222- 2-222,222,220 2023.
[10] Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, and Shangsong Liang. Multimodality representation learning: A survey on evolution, pretraining and its applications. ACM Trans. Multimedia Comput. Commun. Appl., 20(3), October 2023.
[11] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution region supervision. In International conference on machine learning, pages 5583-5594. PMLR, 2021.
[12] Ho Lyun Jeong, Ziqi Wang, Colin Samplawski, Jason Wu, Shiwei Fang, Lance M Kaplan, Deepak Ganesan, Benjamin Marlin, and Mani Srivastava. Gdtm: An indoor geospatial tracking dataset with distributed multimodal sensors. arXiv preprint arXiv:2402. 2024.
[13] Colin Samplawski, Shiwei Fang, Ziqi Wang, Deepak Ganesan, Mani Srivastava, and Ben- jamin M. Marlin. Heteroskedastic geospatial tracking with distributed camera networks. In Robin J. Evans and llya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Un- certainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 1805-1814 14. PMLR, 31 Jul-04 Aug 2023.
[14] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: 1156, 2019.
[15] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00711 2016.
11

---

[16] Yoshua Bengio, Nicholas Léonard. and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rXivv1308.344 2013.
[17] Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. arXiv preprint arXiv:2004. 10:::20004..299,,, 2020.
[18] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. BERT Loses Patience: Fast and Robust Inference with Early Exit. In Advances in Neural Information Processing Systems, volume 33, pages 13330-18341, 2020.
[19] Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all. Train one network and specialize it for efficient deployment. arXiv preprint arXiv: 09791, 2019.
[20] Ruohan Gao. Tae- Hay-H Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10457-10467, 2020.
[21] Ravi Teja Mullapudi, William R Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Specialized dynamic architectures for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8080-8089, 2018.
[22] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
[23] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 1522, 2015.
[24] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[26] Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In NeurlPS, 2022.
[27] Troy Luhman and Eric Luhman. High fidelity image synthesis with deep vaes in latent space. arXiv preprint arXiv:2303. / 13714, 2023.
[28] David Zimmerer, Simon AA Kohl, Jens Petersen, Fabian Isensee, and Klaus H Maier-Hein. Context-encoding variational autoencoder for unsupervised anomaly detection. arXiv preprint arX1v:1812.005941, 2018.
[29] Jianfei Yang, He Huang, Yunjiao Zhou, Xinyan Chen, Yuecong Xu, Shenghai Yuan, Han Zou, Chris Xiaoxuan Lu, and Lihua Xie. Mm-fi: Multi-modal non-intrusive 4d human dataset for versatile wireless sensing. Advances in Neural Information Processing Systems, 36, 2024.
[30] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In ECCV, 2018.
[31] Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1633-1642, 2019.
[32] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. Llnet: A deep autoencoder approach natural low-light image enhancement. Pattern Recognition, 31:650-662, 2017.
[33] Sang Michael Xie and Stefano Ermon. Reparameterizable subset sampling via continuous relaxations. arXiv preprint arXiv: 1901 2019.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the methodology and results, we showcase how ADMN and ADMN_AE can perform Qol aware allocations while being subjugated to strict layer budgets. the case of 6 layer GDTM Blur, obtain near upper bound accuracy while reducing the number of flops by 75%.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the "Limitations and Discussions" section, we discuss how we still have to train unique controllers for each layer budget rather than using universal controller for all budgets. Additionally, we also discuss the possibility of batched inference that the current version of the work does not yet support.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work is primarily empirical evaluation into Qol-aware resource allocation. We do not present any new theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the code in the supplementary. explain architecture in the main paper, and outline our training process the appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
14

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access all our code, which will be released on Github with extensive documentation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wttpps.ccpublli/guiiieee guwde// CodeeSumiia issiiissiio onlolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Not only does the released code contain the information, in Section A.6 describe the training configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide the standard deviation across six seeds in Section A.5
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide an explanation on the compute resources Section A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://nerrip:: http://s..../culll/hhheseseeeeee
Answer:
[Yes]
Justification:
We adhere to all the ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: ADMN proposes method to make multimodal networks more efficient. It does not have direct ties to broader societal impacts.
16

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: ADMN poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Existing datasets and code are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Paper does not exist new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: ADMN does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: ADMN does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: ADMN does not involve LLMs in any meaningful capacity.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:e///eeerrssscccc///ooeeeeee for what should or should not be described.
19

---

A Appendix
A.1 Controller Details
Training Details and Overhead. We found it sufficient to train the controller for 10 and 15 epochs on the localization and classification tasks, respectively. We use learning rate of 1 x 10-3 and linear decay scheduler. For the corruption supervised ADMN controller, we train only on the corruption loss for the first epoch to ensure that the model could accurately differentiate between modality Qol before learning optimal layer allocations. After the first epoch, we add the task loss and optimize the controller over the joint loss. Training an ADMN controller on the GDTM Gaussian Noise dataset took only 27 minutes on an Nvidia RTX 4090, For ADMN_ AE, we divide the training process into two stages. First, we train the autoencoder through reconstruction loss on the target dataset. This is a one-time process, and the set of trained encoder weights can then be used to train controllers of all layer budgets. Subsequently, we load the encoder weights into the controller, freeze them to maintain the Qol information, and train only the layer MIP of the controller on the task loss.
Controller Type
Seed
Layers
8 Layers
12 Layers
16 Layers
100
51.96
36.35
32.24
29.38
ADMN
200
47.29
39.12
33.01
30.95
300
56.09
37.02
32.11
31.27
100
118.14
90,36
73.87
32.15
Straight- Through
200
94.43
58,66
50.76
51.32
300
103.21
82.20
51.84
29.81
Table 6: ADMN GDTM Gaussian Noise Localization Error (cm) + comparing ADMN and the Straight- Through Estimator, with three seeds each
Universal Controller. On the GDTM dataset with Gaussian Noise corruption, we train a single controller for all four layer allocations (6, 8, 12, and 16 layers). We introduce 4 learnable context tokens that correspond to each layer budget, allowing the controller to understand the current layer budget. During training. we randomly select a layer budget for each batch, and concatenate the relevant context token to the input of the TE Fusion block within the controller.
Table depicts the performance of the universal controller (single set of weights) in comparison to training a new controller for each layer allocation on the Gaussian Noise corrupted GDTM dataset. The universal controller successfully accommodates different layer budgets, even outperforming the baseline ADMN on small layer allocations, at the cost of slightly worse performance on larger layer allocations. Future work can further explore the design of the controller for elevated performance.
End-to-end Controller Training. We provide insights into why corruption-aware supervision or autoencoder initialization is necessary during controller training. We hypothesize that the complexity of the training process hinders the controller from learning the corruption distribution without any corruption supervision (explicit with metadata through autoencoder). Since the selection of layer is not a differentiable operation, we model it with Gumbel-Softmax Sampling, followed by discretization and straight through estimator. The gradients received by the controller are thus only an estimation of how that particular layer impacts the downstream loss, with additional complexity arising from the dependence on other layers that were selected alongside it. Consequently, despite training the layer selection mechanism end-to-end, it is very difficult for the earlier perceptual components of the controller to learn to attend to the input modality Qol from the noisy layer gradient information, usually requiring assistance in the form of our corruption-awan supervision or autoencoder initialization.
A.2 Justification of Gradient Propagation Technique in the Controller
Directly Employing the Straight-Throo Estimator, ADMN utilizes the combination of standard temperature Gumbel-Softtax sampling and the through estimator to propagate gradients over the discretization to the continuous logits. One natural question whether Gumbel-Softmax Sampling is necessary, as one could theoretically simply discretize the raw logits and propagate gradients with the straight-through estimator. In Table 6 we present the localization results on the GDTM Gaussian Noise dataset across different layer configurations, with three seeds for each
20

---

Method
6 Layers
8 Layers
12 Layers
16 Layers
ADMN
51.4
39.0
33.1 36.69
30.3
ADMN Universal
43.77
36.78
34.94
Table 7: Localization error on the GDTM Gaussian Noise. Comparison between base ADMN and ADMN with single universal controller
Original Image
Rainy Image
Lowlight + Rainy Image
Original Image
Rainy Image
Lowlight Rainy Image
Figure 8: Visual Depiction of AVE image corruption for outdoor classes
experiment. The results highlight that Gumbel-Softma sampling plays an important role in model training.
This behavior can be attributed to several reasons. First, by applying the softmax function the logits, we convert them into probability values where one logit's 's high probabilities come at the expense of the others. As a result, the softmax function encourages performing layers for some value of noise and minimize the probability of the remaining layers. Additionally, utilizing the Gumbel distribution also introduces stochasticity into the sampling process. Instead of always selecting the top-L logits the active layers, the stochasticity intuitively serves to encourage exploration of different layer configurations.
controller to select only the best
Progressive Top-L Gumbel Softmax Sampling, Instead of employing the straight-through estimator, one can also utilize repeated Gumbel-Softmax Sampling to emulate discrete top-L sampling. Xie et. al. softmax function T. times while adjusting the logits each iteration. However, these methods may cause issues when applied to ADMN. First, methods utilizing Gumbel-Softmax Sampling to emulate discrete distributions typically have to undergo temperature annealing (15), where the temperature is slowly decreased until the distribution is approximately categorical. Utilizing annealing can lead to longer and more complicated training process for the controller. Additionally, the lack explicit discretization during training may also result in distribution shift at inference time, where the controller may learn to over-rely upon partially activated layers during training.
33 proposed method to emulate discrete top-I. sampling by repeatedly applying the
A.3 Additional Dataset Information
We provide further information on the construction of the datasets used in our main evaluations. Visual examples of samples from these datasets, along with the corresponding controller layer allocations, are shown Section A.7
GDTM Gaussian Noise. We define four standard deviations for each modality. The image modality defines a set of standard deviations (0, 1, 1,3,3 while the depth modality defines own set standard deviations [0, 0. 0.25, 0.5, 0.7. We construct the dataset by randomly selecting a standard deviation from each set to corrupt the corresponding modality with. Note that since the GDTM dataset distributed with three nodes, we apply the same corruptions across the three nodes. Thus, the same
21

---

noise that is applied to RGB image in Node 1 will be applied to RGB image in Nodes 2 and 3. We do not consider QoI variations across the distributed nodes.
GDTM Lowlight. We define three levels of lowlight - ordinary lighting. medium lowlight, and severe lowlight. Under ordinary lowlight, we do not modify the RGB modality, but introduce saturation corruption into the depth modality. IR depth cameras frequently saturate under direct lighting conditions, and we wished to emulate this phenomenon in the ordinary lighting case. We obtained a sample of corrupted depth data (sensor is entirely saturated and contains no useful information) by bringing the sensor into direct sunlight. Under medium lowlight, the RGB image is darkened while the depth frame is left unchanged. Finally, for severe lowlight, the RGB image is substantially darkened, and the depth frame left unchanged. The dataset is equally split between these three conditions. Similarly to the GDTM Gaussian Noise dataset, there are no Qol variations across distributed nodes- a given sensor type across all nodes will suffer from the same lighting condition.
GDTM Blur. We simulate the effects of blur on only the RGB camera modality. We define three levels of blur no blur, medium blur, and severe blur through the use of Gaussian Kernel blurring. We split the dataset equally among these three combinations with depth left unchanged. The blur corruption does not change across the distributed nodes.
MM-Fi Gaussian Noise. MM-Fi Gaussian Noise leverages a similar principle to the GDTM Gaussian Noise dataset. The image modality defines set of standard deviations [0, 0.75, 1.5, 2] while the depth modality defines set of standard deviations [0, 2,3, 4]. The MM-Fi dataset contains data across the temporal axis with several frames spanning given human activity. When draw one standard deviation for a modality, we apply noise with that one standard deviation across all frames of that given modality. Within a particular sample, we do not consider Qol variations across frames.
AVE sain/Background Noise. The audiovisual AVE dataset contains 28 classes that contain both indoor and outdoor samples. To ensure that we apply realistic corruptions, we corrupt indoor and outoor samples differently. For outdoor samples, we choose to induce rain and lowlight corruption in the RGB image modality, and add background wind noise to corrupt the audio modality. For indoor samples, the RGB modality is corrupted with lowlight, and audio is corrupted with background noise of a standing fan. We follow the techniques in [3] for adding rain, and mix in the background audio noise through the PyDub library. Fach clean outdoor/indoor sample is transformed into one with corrupted image/clean audio, and another with clean image/corrupted audio. We show the roin+lowlight corruption of outdoor samples in Figure g to demonstrate why we had to add an additional lowlight corruption. After adding rain, the main subject of the image is still quite visible, necessitating the addition of lowlight to ensure that the model cannot rely solely on visual information perform classification.
A.4 Additional LayerDrop Results
Removed Layer Indices
Normal Pre Normal FT
LayerDrop Pre Normal FT
Normal Pre LayerDrop FT
LayerDrop Pre LayerDrop
None 6
81.16%
80.36%
79.91%
78.92%
79.993,
79.35%
79.61%
78.59%
6,8
77.20%
76.77%
78.68%
77.72%
6,
72.97%
73.94%
77.64%
76.84%
2,
70.01%
71.48%
76.90%
76.37%
2,4, 6, 10
64.27%
65.07%
75.44%
74.70%
2,4, 7, 10
33.91%
38.76%
69.83%
69.28%
1, 7, 10
13.74%
23.60%
66.67%
66.94%
Table 8: ImageNet-IKK IK performance with different layer indices removed. Normal refers to LayerDrop rate of O, while LayerDrop refers to utilizing a LayerDrop rate of 0.2. Pre indicates the MAE pretraining stage, while FT refers to supervised finetuning ImageNet-IK.
ImageNet-1K Results. The visual backbones are first pretrained on the ImageNet dataset with Masked Autoencoder pretraining, in which we add LayerDrop. To understand its performance on the ImageNet- dataset, we perform a subsequent stage of supervised learning on the ImageNet dataset to obtain the validation accuracy. Table 8 reveals the validation accuracy on the ImageNet- 1K dataset with various dropped layers, and with LayerDrop integrated into different stages of training. When comparing the model trained without any usage of LayerDrop to the one in which
22

---

Removed Layer Indices
Normal Pre
LayerDrop Pre
-ayerDropp
None
33.77%
32.01%
31.31%
31.67%
6.
27. 27.43%
31.17%
4,6,
2,4,
15.64%
30.08%
6,
9.82%
28.79 18.79%
2,4,6 8,
4.42%
24.64%
1. 6, 7,
2.43%
22.77%
5.
1, 3. 5,
7.
1.11%
3.89%
1, 3,5, 7,
Table 9: AudioSet performance with different layer indices removed. Normal refers to LayerDrop rate of O, while LryerDrop refers to utilizing a LayerDrop rate of 0.2. Pre indicates the MAE pretraining stage, while FT refers to supervised finetuning AudioSet.
LayerDrop employed in both stages, when 7 layers are dropped during inference time. Curiously, given that LayerDrop is added during supervised finetuning, applying MAE pretraining with LayerDrop does not appear to be necessary in ImageNet-1K K. However, the results in Figure Gsowwesee that it has an impact on downstream tasks.
observe accuracy improvement of over 50%
AudioSet-2M Results. The audio backbones are pretrained on the AudioSet dataset with AudioMAR pretraining. We present the AudioSet test split results in Table finetuning on a balanced subset referred to AudioSet-20k. We compare the impact of adding LayerDrop into both stages of the process, and find that introducing LayerDrop is key for graceful degradation as layers are removed. With four layers removed, the model pretrained and finetuned without LayerDrop has its accuracy cut half, while the model with LayerDrop loses less than 10% accuracy.
Random Seed
Avg.
Std.
Noise Type
Layer
100
200
300
400
500
600
52.01 53.3
153.7
56.1154.0
52.3 153.6
55.2153.5
45.6 53.4
$1.4153.6
4.21
36.41
39.1 137.6
37.0137.5
41.4137.9
41.0141.7
38.8 38.1
39.0138.4
2.0
Gaussian
12
35.0
33.0132.6
32.1135.00
32.8
35.8132.1
32.5 33.5
13.1
1.41
16
29.41 29.7
31.01 30.2
31.3
29.3 128.7
31.61
29.31 28.6
30.3
1.1
6
56.81 33.7
34.0140.5
36.6133.4
14.0
57.8133,
57.5
33.5
49,5
11.1
Lowlight
19.5 19.4
27.2 127.0
29.21
12.4
27.5113.4
19.5 27.9
23.91/2..7
3.914.0
12
18.1 18.2
17.6118.2
18.1118.2
18.2120.
18.4
17.6 18.7
18.01 18.7
0.310.88
16
17.31 18.1
17.5
16.9117.
17.1 11.4
17.41
17.4 17.7
17.3
0.210.3
6
14.71 12.6
1252
11.2112.
11.2111.3
11.51114
11.21 11.3
11,8114.0
1.41555
11.01 10.6
10.6111.1
111111.
12.9110.5
11.1113.1
10.5 11.7
11.21114
0.9110
Blur
12
10.019.6
9.21
9.319.8
9.419.6
10.519.7
10.019.7
9.719.7
0.5
16
10.419.2
9.619.8
9.619.6
10.1
9.619.8
10.0
9.919.6
0.3
Table 10: ADMN (left) ADMN (right) Localization Error (cm)
GDTM Dataset
Random Seed
Avg.
Std.
Layer
100
200
300
400
500
600
38.0
36.1 137.7
34.6 36.7
35.2135
33.6
35.41
35.0136.2
8
38.9137.77 33.77
40.7 35.8
42.01 37.3
39.51 39.5
39.2
35.2
39.2 38.4
2.3
12
39.8 40.1
42.6143.
42.01 44.1
46.6144.
39.8 14.4
40.7
44.1
2.6 41.9142.2
16
13..8
42.6146..
42.01
46.01
43.5
43.2 143.2
43.31 44.4
1412.2
Table 11: ADMN (left) vs ADMN_AE (right) Classification Accuracy (%) on MM-Fi Dataset
A.5
Stability Analysis Under Noise and Seeds
To assess the stability of our results across random initializations, we train the ADMN and ADMN_AE AE controllers across six random seeds for each layer budgets and dataset. Tables TO 12 the result for each seed, along with the averaged result and standard deviation:
showcase
23

---

Random Seed
Avg.
Std.
Layer
100
200
300
400
500
600
49.6
51.9156.4
59.91
57.6155.1
58.5
57.1
61.5 165.2
165.2
57.0162.)
66.1 61.7
65.6 59.5
61.8 165.0
59.77
62.91 62.3
3.6 12.5
12
67.01 66.0
66.1 164.3
61.61 67.3
70.3
70.7
16
67.1 16.33
16.3
69.0168.
67.3 68.1
66.516663
66.8163.6
68.01
66.61 66.7
2.7
Table 12: ADMN (left) vs ADMN_AE (right) Classification Accuracy (%) on AVE Dataset
When comparing ADMN and ADMN_ AE, one may expect that ADMN should provide more consistent results, given that we explicitly leverage known ground truth labels as Qol supervision. However, in the case of seeds GDTM Lowlight, find that the ADMN AE approach provides a significantly lower standard deviation with better localization results. This can potentially be attributed towards the freezing of network weights employed by ADMN_AS where only learn the layer MLP during controller training. In contrast, ADMN's controller is fully unfrozen during the training process, resulting in greater susceptibility to poor seed initializations. In the rest of the experiments, the two methods have approximately similar standard deviations.
ADMN vs ADMN AE:
We observe consistent reduction of standard deviation with increasing layer budget, which suggests that larger layers budgets have more stable layer allocation policies. This aligns with the intuition that the result is more sensitive to layer allocation strategies when there is a smaller budget.
Layer Budget:
A.6
Training Details
Parameter
MM-Fi
GDTM
Lowlight
Blur
AVE
Epochs
400
400
100
200
200
Learning Rate
IE-4
5.00E-04
5.00E-04
5.00E-04
5.00E-04
Scheduler
LinearLR
Optimizer
Adam
Adam
Adam
Adam
Adam
LayerDrop
0.2
0.2
0.2
0.2
0.4
Fusion Layers
6
6
6
6
6
Fusion Dimension
64
256
256
256
256
Fusion Heads
4
4
4
4
4
Modality Dropout
0.1
0.1
0.1
0.1
0.1
Depth Corruption
Gaussian Noise
Gaussian Noise Gaussian Noise
Lowlight Saturation
Blur
Image Corruption
Gaussian Noise
Rain/Lowlight
Audio Corruption
Wind/Fan
Table 13: Finetuning Settings of All Datasets
Finetuning Details. We depict the finetuning configurations for each dataset in Table 13 We employ the embedding-level fusion architecture in all these datasets, where we extract unimodal features with modality specific backbones, merge them with transformer encoder, and perform the task with an output head. The training configurations are similar across the datasets, with small exceptions in MM-Fi and AVE. In MM-Fi, we found it advantageous to utilize fusion dimension of 64 instead of 256 due to the large number of image and depth frames that we process. In the AVE dataset, we employed more aggressive LayerDrop rate of 0.4 during the fine-tuning process, the standard rate of 0.2 led to significant and rapid degradation when dropping layers during inference time.
found that
Compute Requirements. Pretraining on ImageNet- Ik and AudioSet-2M with the MAE and Au- dioMAE techniques were performed on GPU server containing 4 Nvidia H100 GPUs over 5 days. Finetuning the models and performing controller training were performed on two machines one with Nvidia 4090 and the other with two Nvidia 3090 GPUs.
24

---

M
Image Noise Standard Deviation: 0
Depth Noise Standard Deviation: 0.75
Selected Layers:
Selected Layers: [1 0100000000001
110001011000
Image Noise Standard Deviation: 1
Depth Noise Standard Deviation: 0.25
Selected Layers: 1100011100000
Selected Layers: [1000000000 00000001000000000000000000000000 0]
A.7 Qualitative Results
we visually showcase examples of the corrupted data samples and appropriate controller al locations for the GDTM Gaussian Noise, GDTM Lowlight, GDTM Blur, AVE Corruption datasets, and MM-Fi Gaussian Noise datasets, respectively. With the exception of the AVE Corruption dataset, show the normalized images that are input into the model. The controller always activates the first backbone layer each modality for stability reasons. We observe that the controller makes intelligent allocation decisions that align with our expectations from the modality Qol. One interesting phenomenon occurs in the GDTM blur dataset, where although the vehicle is not visible in the "Heavy Blur" image, the controller still allocates all the resources to image, as the model has superior perception compared to human vision and can localize the car in spite of the heavy blur.
In Figures
Additionally, we also compare the ADMN controller to the Task Loss controller (Figure 13) that is supervised by neither the corruption nor the autoencoder pretraining. On the AVE dataset, we can
25

---

Image Noise Standard Deviation: 3
Depth Noise Standard Deviation: 0.25
Selected Layers:
Selected Layers: [1 1100001100001
[1 0100010000011
Image Noise Standard Deviation: 3
Depth Noise Standard Deviation: 0
Selected Layers:
Selected Layers:
[1 0
0
[1 0 1 0 0 1]
Figure 9: Visual Results on the GDTM Dataset highlighting the impact of noise and featuring the controller layer allocation
that the controller allocations do not account for the changing modality Qol, which explains the inferior results in T
26

---

Clean Image
Corrupted Depth
Selected Layers:
Selected Layers:
[100001110001]
[1 0 1100000000011 o]
Medium Lowlight
Clean Depth
Selected Layers:
Selected Layers: 1 110001110001
[1 1100000000001
27

---

Severe Lowlight
Clean Depth
Selected Layers:
Selected Layers:
[1
0
[1 0 1 1 0 0 1]
Figure 10: Visual Results on the GDTM Dataset highlighting the impact of lowlight and featuring the controller layer allocation
g9
Clean Image
Clean Depth
Selected Layers: [100111110001]
Selected Layers: [1000000000000 0]
28

---

Medium Blur Image
Clean Depth
Selected Layers: [ 1101111100011
Selected Layers:
[1 010000000000000
Heavy Blur Image
Clean Depth
Selected Layers: [1 0 1 1 1 1 1000011
Selected Layers:
[1 00000000000000 0]
Figure 11: Visual Results on the GDTM Dataset highlighting the impact of blur and featuring the controller layer allocation
29

---

Corrupted Audio & Clean Image
Selected Layers: [1 1100000100001]
Selected Layers: [1 10001110001
Clean Audio & Corrupted Image
Selected Layers: [1 [101000110011
Selected Layers: [1 000000000001
30

---

Clean Audio & Corrupted Image
Selected Layers: 1110101100011
Selected Layers: [1 000000000011
Figure 12: Visual Results on the AVE Dataset highlighting the rain/lowlight visual corruptions and background noise audio corruptions. We also showcase the ADMN controller layer allocation.
31

---

Corrupted Audio & Clean Image
Task Lost Controller Selected Layers: [1 [100100000001]
Task Loss Controller Selected Layers: 100001100011
Clean Audio & Corrupted Image
Task Lost Controller Selected Layers:
[1 [10000001001]
Task Lost Controller
Selected Layers:
1100011100111
Figure 13: Visual Results on the AVE Dataset highlighting the rain/lowlight visual corruptions and background noise audio corruptions. We showcase the Task Loss controller layer allocation.
32

---

Image Noise Standard Deviation: 0.0
Depth Noise Standard Deviation: 4.0
Selected Layers: 1100000111011
Selected Layers: [1 11000100000001 0]
Image Noise Standard Deviation: 0.75
Depth Noise Standard Deviation: 2.0
Selected Layers: [100010111001]
Selected Layers: [10001000000 0]
33

---

Image Noise Standard Deviation: 2.0
Depth Noise Standard Deviation: 0.0
Selected Layers: [1 0100000000000
Selected Layers: [110001101101]
Image Noise Standard Deviation: 2.0
Depth Noise Standard Deviation: 3.0
Selected Layers: [1 0100000000000 [110001100 10001110
Selected Layers:
1]
Figure 14: Visual Results on the MM-Fi Dataset highlighting the impact of Gaussian Noise. We
showcase the ADMN controller layer allocation
34


Reviews:
[{"review_id": "LBTIQnHzbW", "reviewer": "Reviewer_bi2r", "summary": "This paper introduces ADMN, a layer-wise Adaptive Depth Multimodal Network designed for dynamic environments with fluctuating compute resources and input quality. ADMN adjusts the number of active layers across modalities based on resource constraints and reallocates computation based on modality quality. It achieves similar accuracy to state-of-the-art models while reducing computational cost by up to 75%.", "questions": "- Including the main conceptual differences from Mixture-of-Experts approach would be helpful.\n- Paper would benefit from backing the motivation behind autoencoder-based initialization (lines 211 - 213) with some evidence.\n- In stage 2 of controller training, why is the backbone frozen?\n- Minor: Typo in line 2? ‘Afforded’ -> ‘offered’?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- The exact novel contribution is not clear. LayerDrop finetuing is based on the existing LayerDrop method, while more clarity on autoencoder-based initialization in QoI-aware training would be helpful. \n- Novelty is rather limited. \n- Acknowledging this existing work would benefit the paper: Cai, Han, et al. \"Once-for-all: Train one network and specialize it for efficient deployment.\" arXiv preprint arXiv:1908.09791 (2019).\n- The topic of the paper is quite relevant from the point of view of practical deployment on edge devices.\n- Limitations have been discussed well.", "was_revised": true, "final_justification": "After reading the rebuttal and other reviewers’ comments, I am revising my rating of the paper.\n\nThe authors have provided clarifications that helped better understand the novelty of the work, particularly the focus on modality-aware, fine-grained resource allocation under strictly bounded compute budgets, which sets it apart from prior dynamic methods. The explanation around the controller design (including both supervised and unsupervised approaches) and its grounding in modality quality adds further depth to the contribution.\n\nOverall, the authors' clarifications have addressed my major concerns, and I now find the paper's contribution more substantial than initially perceived. I am therefore increasing my rating.", "rebuttal": "Dear Reviewer bi2r,\n\nThank you so much for your constructive comments, and we look forward to improving our work with your ideas. We address some key points and questions below: \n\n**Point 1: The exact novel contribution is not clear/Novelty is rather limited**\n\nWhile we acknowledge that ADMN leverages previous techniques such as LayerDrop to enable dynamic layer selection or Gumbel-Softmax Sampling for controller training, ADMN provides several novel contributions both in the novelty of the problem explored and the techniques utilized. \n\n**First, ADMN is the first work to address the impact of relative modality QoI on resource allocation across modalities in multimodal networks, demonstrating that proper allocation can achieve competitive performance with a fraction of the compute.** Previous dynamic techniques such as Early-Exiting reduce unimodal network computation by saving resources on simple inputs and indexing heavily into complex samples. While effective, it is nontrivial to extend these techniques to multimodal networks, especially in realistic scenarios where relative QoIs among modalities vary across time. Additionally, another important factor overlooked by existing dynamic networks is the need to accommodate strictly bounded compute resource budgets. The majority of dynamic networks optimize to lower the *average* resource usage, where difficult samples still fully propagate through the whole network. In contrast, ADMN explores the novel problem of fine-grained resource allocation across modalities while considering a temporally variable but strictly bounded budget. \n\n**Next, ADMN introduces a unique design explicitly structured around resource allocation with respect to variable modality QoI in the presence of fixed resource budgets, proposing two novel methods of grounding a controller network to modality QoI.** Our corruption-aware supervision leverages corruption metadata to train the perceptual components of the controller to attend to modality QoI. In situations where corruption metadata is unavailable, ADMN leverages an unsupervised autoencoder approach encoding modality QoI characteristics into a structured latent space. Previous works have not considered training controllers in the presence of corrupted multimodal samples, much less when the corruption distribution is unknown. Moreover, in Section 3.4.3, ADMN also provides original insights on how to easily extend Gumbel-Softmax Sampling to accommodate selection of a fixed number of layers. \n\n**Finally, while ADMN utilizes the LayerDrop method, we also introduce several innovations on the original work.** The scope of LayerDrop involved only unimodal textual transformers, and ADMN showcases its feasibility on Vision and Audio Transformers through integration with Masked Autoencoder pretraining. Furthermore, ADMN also performs LayerDrop finetuning on multimodal embedding-level fusion networks, and introduces full-backbone dropout to simulate scenarios in which a modality provides little utility. \n\nIn summary, ADMN offers novelty in the problem space by exploring the new challenges of resource allocation among corrupted multimodal samples, while simultaneously offering novel technical contributions in QoI-aware controller training and accommodation to fixed resource budgets. \n\n\n**Point 2: Acknowledge existing work “Once-for-all”**\n\nThank you for sharing this work! We will definitely include it in the related work of our paper. “Once-for-all” provides insights on how an existing large network can be easily pruned down to fit various hardware and energy constraints, offering an interesting alternative to the LayerDrop technique. However, we note that “Once-for-all” does not address the allocation of resources among modalities, which is the primary contribution of ADMN. \n\n\n**Point 3: Differentiating ADMN from Mixture of Experts (MoE)**\n\nWe will provide additional clarification in the paper about ADMN’s advantages over MoE techniques. The key differences between ADMN and MoE are (1) MoE's need to train a new model for each compute budget, and (2) its inability to utilize pretrained weights. First, MoE adapts to new compute budgets by training several backbones of various sizes, which incurs a high cost during training. In order to perform fine-grained allocation with MoE, we must train a new model for each allocation. To implement the MNS baseline in Tables 1, 2, 3, we had to train 4 backbones (one for each layer budget) per modality, which was very costly in terms of training resources. In contrast, ADMN requires only the training of one single backbone network with LayerDrop, scaling much more effectively with the number of budgets.\n\nSecond, the inability to utilize pretrained weights degrades the MoE accuracy. Typically, loading pretrained weights prior to fine-tuning greatly improves network performance by leveraging strong priors learned from pretraining. Unfortunately, since MoE trains static networks of different sizes, it is likely that pretrained weights do not exist for that particular network size, resulting in lower accuracy when training from scratch, or additional compute-heavy techniques such as knowledge distillation. Conversely, ADMN trains one large network from pretrained weights and dynamically adapts it to runtime conditions.\n\n\n**Point 4: Provide Additional Evidence for Autoencoder-Based Method**\n\nAutoencoders have been known to cluster similar samples close together in the latent space, enabling interpretability in VAEs [1]  or usage in anomaly detection [2]. Based on this insight, we postulate that the autoencoder can learn to encode the corruption information solely through the reconstruction objective without any corruption metadata. This is substantiated by the competitive performance of ADMN_AE, which illustrates that the unsupervised autoencoder training has learned to capture noise characteristics. We will add this additional justification to the paper, along with TSNE plots to visually demonstrate clustering of similar samples in the autoencoder’s latent space. We are unfortunately unable to provide these plots during the rebuttal, as we are limited to text-only responses. \n\n\n**Point 5: Backbone Frozen during Stage 2 Training**\n\nWe freeze the backbone during Stage 2 Training for two reasons. First, during Stage 1 Training, the backbone is already exposed to countless configurations of backbone layers as a consequence of employing LayerDrop, and has learned to perform the downstream multimodal task in the presence of missing layers. Thus, unfreezing the backbone in Stage 2 Training will result in the learning of minimal new information. Second, since we may repeat Stage 2 Training for several controllers of different layer budgets, we wish to reduce the number of learnable parameters in the Stage 2 network to expedite training time. \n\n\n**Point 6: “Afforded” vs “Offered”**:\n\nWhile we used the word “afforded” to indicate “to provide or supply”, we acknowledge that “offered” is a better choice of words. \n\n[1] Luhman, T., & Luhman, E. (2023). High fidelity image synthesis with deep vaes in latent space. arXiv preprint arXiv:2303.13714.\n\n[2] Zimmerer, D., Kohl, S. A., Petersen, J., Isensee, F., & Maier-Hein, K. H. (2018). Context-encoding variational autoencoder for unsupervised anomaly detection. arXiv preprint arXiv:1812.05941."}, {"review_id": "zkdqZE54Ea", "reviewer": "Reviewer_dare", "summary": "The paper proposes a way to automatically adapt computation by selecting layers  automatically depending on input quality and computational budget, in a multimodal setting. The main contribution is the use a controller that selects, for each modality, which layers to use. The distribution of layers to choose is automatically learned end-to-end by minimizing the target task loss, and depends on the input data and how much informative is the input, for instance an input which is distorted (e.g. with gaussian blur) will use fewer computation than an an input which is clean.", "questions": "Please refer to Strengths And Weaknesses", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is well written and well motivated, dynamic computation in the presence of inputs of varying  quality within strict and constrained resources is an important topic to study. The controller and thus layer selection is trained end-to-end to minimize task loss, which is quite elegant.\n\nTable 5 show that removing the corruption supervision make the results worse. Since the layer selection is trained end-to-end to minimize  the task loss, why can’t the controller figure out automatically to not leverage computation on e.g. severely distorted samples ? could it be related simply the controller network capacity?\n\nThe choice of baseline of models is reasonable. However, the Naive Scratch and MNS baselines\nare trained from scratch, while ADMN uses pre-trained networks. Could it be that the difference in performance come from the use of pre-trained models ? \n\nAlthough the controllers are lightweight, it seems redundant to train a controller for each single layer budget. Would it be feasible to have a single controller that generalizes across budgets ?", "was_revised": true, "final_justification": "Explanations given  about attempts to train a controller in a fully end-to-end manner and cause failures were convincing as differentiable selection is a hard problem by itself, and their approach based on autoencoders makes it possible to avoid explicit supervision of corruption level.", "rebuttal": "Dear Reviewer dare,\n\nThank you so much for your constructive comments, and we look forward to improving our work with your ideas. We are glad to hear that you found the work to be well motivated and important to study. We address some key points and questions below: \n\n**Point 1: Inability of the controller to automatically learn the corruption information**:\n\nThe obstacle hindering the controller from learning the corruption distribution without any corruption supervision (explicit with metadata or through autoencoder) is the complexity of the training process. Since the selection of a layer is not a differentiable operation, we model it with Gumbel-Softmax Sampling, followed by discretization and a straight through estimator. The gradients received by the controller are thus only an estimation of how that particular layer impacts the downstream loss, with additional complexity arising from the dependence on other layers that were selected alongside it. Consequently, despite training the layer selection mechanism end-to-end, it is very difficult for the earlier perceptual components of the controller to learn to attend to the input modality QoI from the noisy layer gradient information, usually requiring assistance in the form of our corruption-aware supervision or autoencoder initialization. \n\n**Point 2: MNS and Naive Scratch baselines do not utilize pretrained weights**: \n\nThe ability to use pretrained weights is a direct advantage of our design choices of ADMN. By specifically using the LayerDrop technique, we can initialize our model with pretrained weights and selectively drop certain layers according to the input modality QoI and the current resource budget. In contrast, the MNS and Naive Scratch methods are not compatible with pretrained weight initialization since pretrained weights only exist for a few configurations of model size and thus must be trained from scratch. We acknowledge that part of ADMN’s performance can be attributed to the usage of pretrained weights, but the drastic benefits over the Naive Alloc baseline, which uses the same weights as ADMN, highlights the importance of intelligent resource allocation. Additionally, the MNS and Naive Scratch baselines have the advantage of training the network for that particular resource budget, while ADMN is forced to use one set of backbone weights for all budgets, which can slightly offset ADMN’s advantage with pretrained weights. \n\n\n**Point 3: Training a Universal Controller**\n\nThank you for the suggestion! This is a good idea to further extend the generalizability of ADMN by training a universal controller for all layer budgets. To achieve this, we can add an additional budget token representing the number of available layers to the input, or condition the network weights with a hypernetwork on the number of available layers. However, it is possible that the added complexity may result in unstable controller training. We will explore adding this in the camera-ready version of the paper."}, {"review_id": "JgVEI0gzvS", "reviewer": "Reviewer_1X5m", "summary": "This paper proposes a layer wise adaptive multimodal network that has two innovations: 1. the proposed method is able to work within the strict bound of varying compute budgets, where most existing dynamic methods on optimize for average case efficiency. 2. the proposed method is able to adaptively assign compute resource to different input modality, based on their quality of information (QoI), such that the input modality with less corruption is assigned more resources, to achieve optimal performance. The experimental results show that the proposed method indeed achieve better computation consumption / performance tradeoff comparing to a series of non-adaptive baselines, positively supporting the claimed strengths.", "questions": "1. Will the proposed controller show the same competitive performance against the baselines when the depth of feature backbones are increased?  since more complex backbones require more sophisticated decision mechanism, I assume this means that the capacity of the controller will need to be increased as well in order to make well informed decisions. is it possible to show, e.g., when the feature backbones are 16 or 20 layers, how many more layers in controller are needed for optimal performance, and whether the increased controller load is still acceptable throughput / FLOP wise.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well motivated. As in resource limited platforms, optimizing the model within the bound of strict computation budget is more important than optimizing for average case efficiency; Also, modality-quality aware compute allocation is also important to ensure the information among input modalities are reliably utilized, and is a less explored research direction in dynamic network design domain.\n\n2. The method proposed in the paper makes intuitive sense. Firstly, the method finetune the model with layerdrop regularization, ensuring that the network performance will not be affected by layer skipping operation; Further, in the controller training stage, to make sure that the control module is aware of the QoI of the input modality when making compute allocation decision, the paper proposes to add corruption aware prediction loss in supervised setting, or auto-encoding like reconstruction loss in unsupervised setting. And the ablation results show that the added losses indeed facilitate the controlled to make more QoI informed layer allocation predictions.\n\n3. The experimental results show that under the same computation budget, the proposed method achieve better performance than non-adaptive baselines and baselines that are trained from scratch, it also shows that the added computation load of the controller still guarantees high throughput and negligible FLOPS increase. Together the results show that the proposed controller can distribute the compute resource among modalities optimally with acceptable increased model complexity.\n\n4.  The limitations of the current version of the proposed method is clearly discussed.\n\nWeaknesses:\n1. The experiment only tests the proposed method when the feature backbones have 12 layers, so it's not clear whether the controller can still work well with increased backbone depth, or how much additional capacity are needed in the controller in order to learn budget allocation with deeper backbones.", "was_revised": true, "final_justification": "The author's results regarding the scalability of the proposed controller across multiple modalities is equivalent to increasing the neural network's depth, thus addressing my question. So I will maintain my current rating as accept.", "rebuttal": "Dear Reviewer 1x5m,\n\nThank you so much for your constructive comments, and we look forward to improving our work with your ideas. We are glad to hear that you found the work to be well motivated and the method to be intuitive. We address some key points and questions below: \n\n\n**Point 1: Scaling ADMN to deeper backbones**:\n\nWhile ADMN’s method is certainly compatible with backbones of any size, we are unable to provide evaluations on larger backbones in the short timespan of the rebuttal due to the training time required. Nevertheless, we are confident that the controller is able to easily scale to larger backbones. The bulk of the controller parameters are devoted to perceiving the modality QoI (e.g., the convolution layers and transformer fusion encoder). The required capacity of these components are related to the difficulty of perceiving the corruption information and are agnostic to the size of the backbone. The only component dependent on the backbone depth is the neural component tasked with translating the corruption embedding into an allocation of layers among backbones. This takes the form of a lightweight, two-layer MLP that projects the corruption embedding into the number of total backbone layers. \n\nIn Section 4.3, we showcase ADMN’s ability to handle three modalities, where the controller is capable of accurately allocating resources among 36 layers (12 layers in each of the 3 transformer backbones). We do not alter the controller's architecture, other than changing the output dimension of the MLP from 24 to 36. Thus, while we are unable to directly experiment on larger backbones, this instills confidence that the controller can support significantly more complex backbones (e.g., two backbones each with 18 layers) while maintaining its accuracy and lightweight nature."}]

Author rebuttal:

Meta-review: The paper received unanimous acceptance recommendations from the reviewers. The pros include relevance for deployment on edge devices, good writing and elegant design, reasonable choice of baselines, and strong results. Additionally, many reviewers have mentioned that limitations are very well-characterised, which I have validated and agree with - I am mentioning this explicitly, because so many papers pretend to do that accurately but in fact are very one-sided and focus on highlighting only the benefits of their work - the submitted paper is an exception in that regard, and deserves additional praise in my opinion.

At first, the reviewers were not sure about: 1) the novelty of the proposed method, 2) its generalisability, and 3) if the comparison to the baselines is fair. All of those shortcomings were addressed by the authors during the rebuttal, together with providing additional insights into their method. Overall, I find the state of the paper and the associated discussion sufficient for acceptance.
Decision: Accept (poster)