Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models
Authors: Siqi Lu
Wanying XU
Yongbin Zheng
Wenting Luan
Peng Sun
Jianhang Yao
Venue: ICLR 2026 Poster (2026)
Keywords: Modality Missing
Plug-and-play module
multimodal imbalanced learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Missing modalities present a fundamental challenge in multimodal models, often causing catastrophic performance degradation. Our observations suggest that this fragility stems from an imbalanced learning process, where the model develops an implicit preference for certain modalities, leading to the under-optimization of others. We propose a simple yet efficiency method to address this challenge. The central insight of our work is that the dominance relationship between modalities can be effectively discerned and quantified in the frequency domain. To leverage this principle, we first introduce a **F**requency **R**atio **M**etric (FRM) to quantify modality preference by analyzing features in the frequency domain. Guided by FRM, we then propose a **M**ultimodal **W**eight **A**llocation **M**odule, a plug-and-play component that dynamically re-balances the contribution of each branch during training, promoting a more holistic learning paradigm. Extensive experiments demonstrate that MWAM can be seamlessly integrated into diverse architectural backbones, such as those based on CNNs and ViTs. Furthermore, MWAM delivers consistent performance gains across a wide range of tasks and modality combinations. This advancement extends beyond merely optimizing the performance of the base model; it also manifests as further performance improvements to state-of-the-art methods addressing the missing modality problem.

Published as a conference paper at ICLR 2026
PLUG, PLAY, AND FORTIFY: A Low-CoST MODULE FOR ROBUST MULTIMODAL IMAGE UNDERSTANDING MODELS
Siqi Lu, Wanying Xu, Yongbin Zheng', Wenting Luan, Peng Sun, Jianhang Yao
National University of Defense Technology
[lusiqi9803, wanying xu, zybnudt 2ybnudt]eendtt Jenudt edu. on
(luanwent ingnudt,,, sunpeng, yso yaojianhang23 @nndd.. edu. cn
ABSTRACT
Missing modalities present fundamental challenge in multimodal models, often causing catastrophic performance degradation. Our observations suggest that this fragility stems from an imbalanced learning process. where the model develops an implicit preference for certain modalities, leading to the under-optimization of others. We propose simple yet efficient method to address this challenge. The central insight of our work is that the dominance relationship between modalities can be effectively discerned and quantified in the frequency domain. To lever- age this principle, we first introduce a Frequency Ratio Metric (FRM) to quantify modality preference by analyzing features in the frequency domain. Guided by FRM, we then propose a Multimodal Weight Allocation Module, a plug-and-plaa component that dynamically re-balances the contribution of each branch during training, promoting a more holistic learning paradigm. Extensive experiments demonstrate that MWAM can be seamlessly integrated into diverse architectural backbones, such as those based on CNNs and ViTs. Furthermore, MWAM de- livers consistent performance gains across a wide range of tasks and modality combinations. This advancement extends beyond merely optimizing the perfor- mance of the base model; it also manifests as further performance improvements to state-of-the-art methods addressing the missing modality problem.
INTRODUCTION
Multimodal vision understanding models leverage complementary data streams like visible, infrared, and depth images to achieve more robust and accurate inferences. While the efficacy multimodal techniques is well-established across tasks such as segmentation, detection, and classification, critical limitation in the existing work the underlying assumption of complete modal availability during inference. This assumption often proves unrealistic in practice, as real-world applications are frequently compromised by challenges such as sensor failure, adverse environmental conditions, or privacy constraints, resulting in scenarios with incomplete modalities (Zhang al. (024a)).
Addressing the challenge of missing modalities crucial for enhancing the robustness of multi- modal models. To this end, researchers have proposed several key strategies. One prominent ap- proach involves feature imputation, where the missing modality's s features are reconstructed from the available ones restore complete multimodal representation and thus mitigate performance degra- dation (Tran et al.] (2017); 7): Lin et al.] (2023). Another distinct strategy bypasses reconstruction and instead projects all modalities into unified, modality-agnostic feature space (Reza al. (2024); Lee et al. )2023). By minimizing inter-modality discrepancies, this method leverages consistent cross-modal information to ensure reliable and stable inference, even when some data is absent.
However, our analysis reveals critical vulnerability in unified models and standard learning meth- ods: their performance is highly sensitive to the identity of the absent modality. In Table [ the absence of Depth causes the most severe performance drop, even falling below that of unimodal model trained solely on the remaining data. Conversely, the absence of RGB has the least impact.
"Equal Contribution. 'Corresponding Author, Code: https: thub. com 61 01001122 02111/MWMM /MWAM

---

Published as a conference paper at ICLR 2026
Table 1: Analysis of model performance under various incomplete modality conditions the CASIA-SURF dataset. Performance is quantified using Accuracy (Acc) and Performance Collapse
Rate (PCR). Multi-modal refers to models trained on complete data but evaluated with missing modalities during inference. Uni-modal indicates models trained and evaluated using only a single
modality. FRM the Frequency Ratio Metric detailed in Section 4.2
Modal
Molti-modal
Uni-modal
RFNet
FRM (T)
Rgb
Depth
Ir
Acc(T)
PCR(L)
Acc
PCR
Acc
PCR
X
X
80.10 (-10.96)
18.37 (+11.17)
91.06
7.20
87.57
11.38
1.81 x 10
X
x
95.21 (-2.08)
2.97 (+2.12)
97.29
0.85
95.83
3.03
2.33 x
X
x
v
90,94 (-1.52)
7.32 (+1.55)
92.46
5.77
85.31
13.67
1.02x 109
V
x
96.20 (-0.83)
1.96 (+0.85)
97.03
1.11
97.77
1.06
2.34x
V
X
v
92.24 (-2.44)
5.99 (42.48)
94.68
3.51
95.73
3.13
1.20x 109
X
V
V
97.79 (+0.33)
0.34 (-0.33)
97.46
0.67
96.78
2.06
x.43 1010
V
98.12 (-)
98.12
98.82
x
Average
92.94 (-2.50)
6.16 (+2.98)
95.44
3.18
93.98
5.72
We posit that this phenomenon stems from an implicit bias within multimodal image understanding models (hereafter "multimodal models"), which tend to favor certain "preferred" modalities during training (Peng et al. (2022b)). This biased optimization allows dominant modalities to dispropor- tionately influence the gradient updates, leading their features being well-optimized while those of other modalities are neglected. Such imbalance results in disparate levels of feature learning across the encoders, causing significant performance fluctuations when the model faced with different combinations of inputs at inference time. This leads to the first pivotal question of our research: How can we identify and quantify the modality preferences inherent in a multimodal model?
While existing work has attempted to improve robustness by balancing modality contributions (Wei al.] (2023): Yang et al. (2024c)). they typically operate in the spatial domain and, as our exper- ments show, have not yet reached their performance ceiling. We argue that critical, yet overlooked, source of information lies in the frequency domain. The distinct textural and structural features of different modalities manifest unique signatures in the frequency spectrum, offering more robust basis for comparison. Experimental analysis reveals a crucial insight: models are predominantly re- liant low-frequency information for recision-making (Fig. D.. Therefore, diverging from existing works (Peng et al. (2022b); Yang et al. (2024d), we introduce the Frequency Ratio Metric (FRM), a novel indicator to evaluate modality preference. As shown in Table findings confirm strong correlation: modalities with a higher FRM tend to be the ones that dominate the training process.
Building upon the FRM, introduce the Multimodal Weight Allocation Module (MWAM), which ald--and module designed to actively mitigate modality bias By assigning weights in- versely proportional the FRM, MWAM actively re-balances the model's focus in gradient or loss space, steering model's optimization trajectory towards a more equitable and robust representation.
The principal contributions of this paper as follows: Firstly, we establish through experimental and theoretical validation that dominance relationship between modalities can be effectively dis- cerned and quantified in the frequency domain. Based on this finding, we introduce new metric for this purpose, termed FRM. Secondly, we develop novel huug-and-play module based on the FRM, termed MWAM, to dynamically allocate modal weights within each mini-batch. MWAM fosters more robust feature learning and, in contrast to existing gradient balancing methods, is inherently scalable to multiple modalities and maintains a simple backpropagation path. Finally, we conducted experiments across diverse modality combinations, various visual understanding tasks, and different backbone architectures to demonstrate the method's effectiveness and generality.
2 RELATED WORK
2.1 INCOMPLETE MULTIMODAL LEARNING
Enhancing the robustness of multimodal models in the presence of missing modalities has emerged as a critical research direction. Contemporary efforts in this domain can be broadly categorized into two dominant paradigms: imputation-based and imputation-free methods (Zhang et al. (2024))).
2

---

Published as a conference paper at ICLR 2026
Imputation-based methods attempt reconstruct the missing data. These strategies range from transforming raw data into kernel or graph representations for imputation (Liu et al. (2017 2020);
Liu ) Wen et al. (2021) employing model-based approaches that recover modality infor- mation from the original data TTan et al. (2017); : Lin et al. (2023)) However, these methods often necessitate auxiliary recovery modules, introducing significant computational overhead that renders
them unviable for deployment on resource-constraing devices.
In contrast, imputation-free methods operate exclusively on available modalities, forgoing recon- struction entirely (Zhang et al. (2024a) Reza el (2024): Lee et al. (2023)). While these approaches are computationally efficient and have fewer parameters, they often suffer from performance degra- dation, as they cannot compensate for the information lost from the absent modality.
Leee
more promising direction, which has recently gained traction, explores the latent relationships between modalities to dynamically assess their contributions (Li et al. (2023b); Du et al. (2023); Wei et al.] (2023): ZZhang et al.] (2024b)). Building upon this principle. we leverage frequency- domain analysis to probe the latent contributions of different modalities. This allows us to introduce the FRM that measures model's intrinsic preference for each modality. Furthermore, the pro- posed MWAM serves as potent balancing mechanism, dynamically modulating the contributions of different model components to provide more balanced attention to different modalities.
2.2 FREQUENCY DOMAIN DEEP LEARNING
Convolution in the frequency domain offers distinct advantage over its spatial counterpart by pre- serving global feature integrity during iterative extraction and mitigating information loss. This in- herent efficiency and representational power have spurred the increasing prominence of deep learn- ing architectures that operate directly in the frequency domain.
In image processing, frequency information characterizes the rate of change in pixel intensities, with low frequencies corresponding to smooth regions and high frequencies to edges and textures. Historically, many models treated the frequency-based module as an auxiliary branch, using it to im- pose priors or constraints on the primary spatial backbone network (Yang et al. (2023 (2024a); Guo ctt al. ). More recently, the trend has shifted towards a deeper integration where models are designed to directly leverage frequency-domain insights. For instance, (Xu et al. (2020)) developed a learning- based frequency selection method that prunes irrelevant frequency components without compromising accuracy, thereby boosting performance in image understanding tasks.
Building on the critical finding that models exhibit distinct preferences for low- and sigh-freuuency channels (Xu et al., (2020); WWang et al. (2023); He et al. (2024)), we investigate the approach to mitigate performance degradation caused by missing modalities. We posit that these frequency preferences are key indicator of a model's 's intrinsic bias toward specific modalities, and that this understanding can be leveraged to enhance the robustness of multimodal models.
FREQUENCY-DOMAIN MANIFESTATIONS OF MODALITY PREFERENCE
3.1 EXPLORATORY ANALYSIS OF FREQUENCY COMPONENTS
Our core insight is that the preference of multimodal models for specific modalities can be observed and quantified in the frequency domain. To this end, we first conduct an exploratory analysis vestigate how different frequency components impact multimodal learning. Specifically, we employ the Fast Fourier Transform (FFT) to decompose images into their frequency representations. We then apply a series of low-pass and high-pass filters with varying cutoff frequencies (i.e., window sizes) to generate frequency-filen filtered datasets for model training. The resulting learning dynamics, illustrated by the training loss and accuracy curves in Fig. | reveal three key insights.
Firstly, low-frequency components are crucial for mitigating training bias. As shown Fig. Da configurations that preserve low frequencies exhibit lower training losses. Secondly, low- uewffreqynncy information facilitates sustained optimization. Models trained exclusively on fighffreuency con- tent exhibit early loss saturation (around 30 epochs), whereas models exposed to low- fow-frequency bands continue to learn, as indicated by their dynamic loss curves (Fig. b.). Finally. while low-frequency components are dominant in determining final performance, high-frequency infor-
3

---

Published as a conference paper at ICLR 2026
0.20
Lowe-passt
0.30
1.0
Low poss620)
Hig Hightpeel! 15)
High-pas|50)
dacasee
0.20
0.15
0.8
0.15
passs
0055(30)
0.55
High-aaer11
Han-poss
0.6
His poss)
*ggt--eee (5)
Raw
Raw dataser
0.00
0.00
0.5
23
40
60
80
20
40
60
83
100
20
60
80
100
Epoch
Epoch (c)
(8)
(b)
Figure I: Impact of different frequency components on multimodal model performance. (a): Train- ing loss curves. (b): Validation loss curves. (c): Validation accuracy curves. The numbers in the legend represent the window sizes of the filters, and "Raw dataset" denotes the original dataset without any filtering applied. More experimental details can be found in Appendix A.5
mation provides non-negligible contribution. Fig. e demonstrates that incorporating more high- frequency content consistently boosts model accuracy.
These findings establish principle for our motivation: effective modality preference metric must prioritize the foundational signal from low frequencies while strategically incorporating the fine-grained details provided by high frequencies.
3.2 FREQUENCY-DOMAN PREFERENCES IN MULTIMODAL MODELS
While Section 3.1 empirically outlined the core design principles of FRM based on experimental observations, this section aims to uncover the underlying reasons driving the frequency-domain preferences of multimodal models. Our goal is to provide substantial theoretical grounding for our proposed methods.
Preliminaries. Consider training set D {aim 2ing y" J==1...,N for dual-modal K-way classification task. Let [(Em, Emg C) denote the complete classification network employing a concatenation-basee fusion strategy, where Em, (:; f,, and Emn (:; 0m,) represent the modality- specific encoders, and C(:; W,b) serves as the classifier. The model is optimized using the gradient descent algorithm with Cross- Cooss-Entropy loss.
Theorem 3.1. Let the training set, model architecture, loss function, and optimizer be defined as in the Preliminaries. The update rule for each modal branch is coupled through shared global error signal. A modality significantly superior at the start of training will dominate the back proo-prrpggated error signals due to effective loss reduction, thereby hampering the optimization of weaker modali- ties through gradient suppression.
The detailed proof of the theorem 3.1 can be found in the Appendix A.I.I This theorem confirms that the model does not attend to modalities equally but rather establishes a preference hierarchy during optimization. Building on this fact, we proceed to map this preference relationship into the frequency domain. This allows us to characterize why the model prefers certain modalities and observe the spectral signatures of this optimization imbalance.
Theorem 3.2. For neural network with sufficient width, the training dynamics along the eigenvec- tors the Neural Tangent Kernel (NTK) are decoupled. The convergence rate of the loss function along the i-th eigenvector direction is strictly determined by its corresponding eigenvalue A;:
Decay Rate oc - 777s);
(1)
where 77 denotes the learning rate of the model.
The detailed proof of the Theorem 3.2 can be found in the Appendix Appendix[A.112 1.2 This theorem elucidates an intrinsic property of neural network optimization: eigenvectors associated with larger eigenvalues exhibit faster convergence rates. Moreover, extensive prior research has substantiated that these eigenvectors correspond to low- low-frequency functions within the input data (Rahaman al. (201)); Wang et al. (2021): Basri et al.) (2019). Synthesizing these findings with Theorem 3.1, we arrive critical conclusion: during the optimization process, multimodal models exhibit preferential

---

Published as a conference paper at ICLR 2026
3al
Loo-urouc Agyegalion
Nat
Kat
High EroqucieN Appregation
Calculation
Widglt calcobatten
Essadox
Encada
FRM Bank
Loe-Frogurncy
INCT
FRM
K
erequueccy
Caloubtton
Apprepation
KWWX
FRM Bank
Fusdon
1-W
1-W
1-W
F4
Aux Hand
Hladd
A Head
Yat
y,
Your
1-w
x
Weightrd Caltelation
(c)
Head
Number opochs
Tstal
Optional Components
Muliioodal Weights
Number of iterations each epoch
W W eight historkcal FRM
Aux Bead
and F correspend equation (2)
0g axressbillity control factors
Figure 2: Architecture and application of our proposed MWAM. (a): Main structure of the MWAM. (b): FRM bank, designed handle modality exceptions. Its update mechanism is governed by Eq. (c): An illustration of the integration of MWAM into multimodal host model. The calculation rules of FRM follow Eq. which requires flipping and aligning the nigh-frequency components.
bias towards modalities rich in low-frequency information. Consequently, the persistence of this preference severely inhibits the optimization of the weaker modality branch.
To mitigate this reliance on frequency-specific preferences, we design the MWAM to first quantify the extent of the model's bias using FRM towards different modalities. Subsequently, by assigning adaptive weight coefficients to each modality, the MWAM effectively alleviates modality imbalance, thereby ensuring balanced optimization process.
PROPOSED METHOD
4.1 MULTIMODAL. WFIGHT ASSIGNMENT MODULE
We design the MWAM, a plug-and-play module that dynamically allocates guidance weights to each modal branch of the multimodal host model during training, thus providing more equitable optimization environment for the model, shown Fig. 2 As illustrated, MWAM is composed of two key components: (a) a core structure responsible for extracting frequency domaai features and computing our FRM. and (b) an FRM bank designed handle anomalous inputs. This section details the overall architecture of MWAM, while the design of the FRM is elaborated in Section 4.2
The workflow of MWAM begins by separating the input into grid of non-overlapping p p patches. The Discrete Cosine Transform (DCT) is then applied to each patch, converting it into the frequency domain. From each transformed patch, we extract the top-left q x q block as the low- low-frequency component and the bottom-right components are then spatially reassembled from their respective patch locations to construct two distinct feature maps representing the frequency profile of the entire input. We set the patch size p = and the frequency block size = 2 in this paper, unless mentioned otherwise. The resulting FRM for each modality is then processed by its corresponding FRM bank, which employs update
block as the high-frequency component. These frequency
5

---

Published as a conference paper at ICLR 2026
rule (detailed in Fig. b) to smooth the metric and derive the final guidance weights, formulated as:
F F =1 +1-
j= 0
j= 1,,,, nt 72
where F is the FRM of T/ Py, is the j-th mini- of modality m, F is the FRM output from the j iteration FRM bank, which integrates the past state and the current state, and w is the weight of historical FRM. We set w 0.5, unless mentioned otherwise.
4.2 MODALITY PREFERENCE METRIC
Identifying the dominant modality is a crucial step toward mitigating the robustness degradation caused by imbalanced model optimization. Prevailing strategies typically assess this imbalance in the spatial domain at the feature level (Peng et al. (2022b); Wei et al. (2024); Huang al. (2025); Wei et al. (2023)), thereby overlooking crucial frequency- domain characteristics that offer global and independent perspective. Motivated by the Frequency Principle of neural net- works (Xu et al. (2024) and our analysis in Section [3.] we hypothesize that modality preference is quantifiable in the frequency domain. To this end, we introduce novel metric, termed FRM, designed to diagnose this preference in real-time.
Motivation:
Frequency Ratio Metric: In Section 3.1 we present key insight: model predictions are predomi- nantly influenced by sow-frequency components. This induces a learning bias, leading the model to favor modalities rich in low- frequeeyy information, which typically encode fundamental structural and textural patterns. Based on this observation, an intuitive way to define modality preference is by the overall magnitude of the low-frequency components, quantified as their Ln-nom:::
10- 1h-1
M MP(mm.))
= Σ
Hlo (a, b)1,
(3)
a=0 6=0
where Mow (a. b) is the low-frequency components of the Imy Wa/p and h Hq/p the width and height of the I respectively. Here, W and H are the width and height of the Im.-
However, we further reveals that high-frequency components still provide valuable discriminative cues for the model. This finding suggests that a strategy of completely discarding these components, as implemented in Eq. B is inherently suboptimal. To address this limitation, we formally define the preference metric as the Ln-norm of the ratio between the low- and high- figh-frequency components:
Mi-1 A-11
I (a, b) Thigh (w a,
FRM(fm.)) =
(4)
b)
a=0 b=u
where Thigh b) is the high-frequency components of the Tm;, a scaling factor.
In the Eq. when Thigh (a, b) 0, will scale up the action of flo at the corresponding position. In addition, on modalities where low-frequency energies are similar, Eq. Feffectively amplifies the FRM differences between these modalities, thereby increasing the weight gap and enhancing the method's performance. To validate our design choice, we evaluate several alternative FRM design rules in Appendix A.8
4.3 HOW TO USE FRM FOR TRAINING INTERVENTION?
Altering model's optimization trajectory via methods like gradient editing and weighted loss func- tions is a well-establisher paradigm for controlling model behavior during training (Peng et al. 2022b); Yang al. (2024d); Huang al. (2025)). We leverage this principle by first employing FRM to quantify the model's reliance on each modality. Based on this measurement, we intervene in the training process through two distinct mechanisms: gradient editing and dynamic loss weight- ing (see Fig. 3). Crucially, our gradient editing approach is parameter-free, and the dynamic loss weighting method necessitates the introduction of lightweight auxiliary heads with negligible pa- rameter cost. To implement this, we define a dynamic weight allocation function, which assigns adaptive weights to each modality within every mini-batch. The function can be expressed as:
8
K3 (x) aa- m.
(5)
1 e X(

---

Published as a conference paper at ICLR 2026
where B, A, and are adjustable scaling factors. T is the proportion of FRM of x the mean FRM all modalities in that mini- denoted as:
FRM
T
M M E (RM +
Eq. 66 dynamically assesses the contribution of each modality within a mini-batch, informing the assignment of corresponding weights during gradient updates. A key characteristic of the Eq. is its inherent flexibility; it is designed to extend beyond dual-modality scenarios to ac- commodate an arbitrary number of modalities. In Appendix A.9 we conducted detailed ab- lation experiments on these scaling factors in order to show the sensitivity of these factors.
Integrating the MWAM into multimodal host model can significantly enhances its training fairness (as shown in Fig. e). with the cor- responding pseudocode provided Appendix A.2 For models that compute modality- specific losses, typically via auxiliary heads, the MWAM globally regulates the training pro- cess by weighting these losses, as formulated in the following equation:
W3d
(4) Gradient Editing Using Weights
X
E₁
Head
Yoi
Fusion Modille
Welght
Head
Lens
M
Ciorel = Σ 1=1
Km, L(y, ym,,) + L(y. Vo), (7)
Xau
Tux
Weighted Lanss UundlAr Hearksard Weighhss
where y. You and y/,, are the groundtruth la- bel, the total output of the model, and the out-
Figure 3: Schematic of training intervention
mechanisms. (a) is parameter-free, and (b) need extra lightweight auxiliary heads.
put of the auxiliary head, respectively. Km, can be calculated from Eq. and Eq. 6 Tt is worth noting that the auxiliary heads and weighted loss are optional extensions rather than required, allowing for flexible integration. In its base configuration, MWAM is entirely parameter-free, incurring only negligible computa- tional overhead terms of foating-poine operations during training.
5 EXPERIMENTS
To validate the efficacy and generalizability of our method, we conduct extensive experiments diverse range multimodal visual understanding tasks. this section, we highlight the principal re- sults, while comprehensive experimental details and further analyses are deferred to the Appendix.
EXPERIMENTS ON MULTIMODAL SEGMENTATION
We validate our method segmentation and semantic segmentation. For brain tumor segmentation, we utilize the BRATS2020 dataset (Menze al. (2015)). We split datasets into 219. 50, and 100 subjects for training, vali- dation, and testing, respectively. For semantic segmentation, we use the NYU-Depth V2 dataset (Nathan Silberman Fergus| (2012)). We split the dataset into 795 images for training and 654 images for testing, adhering to a standard 40-class label setting following official recommendations. Model performance is evaluated using the widely recognized Dice coefficient the whole tumor for brain tumor segmentation and Mean Intersection over Union (MIoU) for semantic segmentation.
Datasets & Metrics:
challenging segmentation tasks: brain tumor
Baselines: We select HeMIS Havaei et al. (2016)). Robust Segmentation (Chen et al. 2020
RFNet (Ding et al. (2021)), mmFormer (Zhang et al. (202)), A2FSeg (Wang & Hong 2023
M3AE (Liu et al. (2023)), GSS (Qiu et al. (2023),, , and LS3M (Zhang et al. (2025)) ) as comparison
models for the brain tumor segmentation, and ESANet-MD (Seichter al. (222)) and MMANet

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison of multimodal robust solutions on BRATS2020 dataset. i denotes the integration of our proposed MWAM with the corresponding base model. Bold and italics indi-
cate the best value and the second best value for each row in turn. Due to space constraints, we
present only a subset of the comparison results here. More results are provided in Table 15 which
includes several advanced methods, including HeMIS, Robust Seg, M3AE, LS3M, and A2FSeg.
Modal
ReNet
RFNet
mmFormer mmFormer*
GSS
GSS
F TI Tle T2
Dice(t) PCR())
Dice
PCR
Dice PCR
Dice PCR
Dice PCR
Dice
PCR
X
X
X
85.07
5.62
85.93 5.32
84.71 5.91 85.80 4.96
85.78
5,64
86.57 5.23
X
V
74.95
16.85
76.39 15.83
74.92 16.78
76.00
15.82
75.96
16.44
79.08
13.43
X
V
X
x
71.98
20.15
74.12
18.33
74.56
17.18
75.93
15.89
75.38
17.08
79.03
13.49
V
X
X
X
85.16
5.52
4.70 86.49
85.87
4.62
86.75
3.91
85.99
5.41
87.34
4.39
X
V
86.91
3.58
87.86
3.20
86.80
3.59
87.54
3.04
87.39
3.87
88,40
3.23
X
V
V
X
78.48
12.94
80.14
11.70
79.48
11.72
80.09
11.29
80.30
11.67
81.93
10.31
X
88.08
2.29
89.06
1.87
88.42
1.79
88.75
1.69
88.87
2.24
90.05
1.42
X
X
86.94
3.55
87.89
3.16
86.96
3.41
87.52
3.06
87.75
3.48
88.11
3.55
X X 88.70
1.60
89.27
1.64
88.61
1.58
89.31
1.07
89.02
2.08
90.07
1.40
X
V
X
88,46
1.86
89.49
1.40
88.62
1.57
89.27
1.12
89.88
1.13
89.91
1.58
V 89.36
0.87
89.84
1.01
89.15
0.98
89.45
0.92
90.20
0.78
90.82
0.58
X
89.74
0.44
90.46
0.33
89.55
0.53
89.93
0.39
90.23
0.75
90.80
0.60
V
89.44
0.78
89.99
0.85
89.54
0.54
89.98
0.33
90.36
0.60
91.02
0.36
V
87.74
2.66
88.50
2.49
87.51
2.80
88.03
2.49
88.14
3.05
88.97
2.61
V 90.14
90.76
7
90.03
7
90.28
7
90.97
7
91.35
7
Average
85.41
5.62
86.41
5.13
85.64
5.21
86.30
4.71
86.41
5.30
87.56
4.44
Wei et al. (2023) for the semantic segmentation. The ESANet is well-established real-time RGB- D segmentation method widely recognized in academia and industry. We adapt it with modality- access control factors to create ESANet-MD, which handles incomplete modality inputs.
We embed MWAM into RFNet, mmFormer, and GSS for brain tumor segmentation, and into ESANet-MD and MMANet for semantic segmentation. Due to the design of the baseline architectures, we do not introduce additional auxiliary heads. We utilize the official code provided by the bascline and complied with the experimental configuration of the respective underlying model. For both tasks, we set the scaling factors in Eq. to 1.5, 1, 6, and 0.7.
Implementation Details:
Results: Results shown in Tables and 3 In the Table the integration of MWAM consistently enhances the performance of all host methods, leading them to outperform their vanilla models. Sig- nificant improvements are observed in both Dice and PCR metrics. Notably, when embedded with MWAM, earlier methods such as RFNet and mmFormel achieve an average Dice score comparable the SOTA method LS3M, and even surpass it in terms of PCR. Furthermore, GSS, when equipped with MWAM, outperforms LS3M on both Dice and PCR. This suggests that our MWAM enables the model to identify a more optimal decision boundary. It is important to highlight that RFNet, mmFormer, and GSS are all SOTA methods designed for the multi-modal missing modality prob- lem. Our MWAM further elevates their performance ceiling. Furthermore, the fact that mmFormer is a ViT-based model underscores the adaptability of our method to diverse architectures.
To further demonstrate MWAM's compatibility with various fusion strategies, we conduct semantic segmentation experiments by integrating it into the vanilla ESANet-MD. ESANet-MD is known for performing feature fusion during the extraction stage, and our results confirm that MWAM seam- lessly adapts to this architecture. Both methods enhanced by MWAM showed distinct performance gains in MIoU and PCR. In summary, our MWAM not only exhibits strong compatibility with di- verse architectures and fusion strategies but also provides substantial performance boosts to existing robust methods, enabling them to push beyond their intrinsic performance limitations.
5.2 EXPERIMENTS ON MULTIMODAL CLASSIFICATION
We utilize CASIA-SURE Zhang al. (219 for conducting our experiments in the multimodal classification task. We adapt the intra-testing protocol recommended by the vanilla work, dividing it into train, validation, and test sets with 29k, 1 and 57k samples, respectively. We
Datasets & Metries:
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison of multimodal robust solutions NYU-DDephh V2 dataset
Modal
ESANet-MD
MMANet
ESANet-MD*
MMANet
Rgb
Depth
MI)U(1))
PCR(L)
MIoU
PCR
MIoU
PCR
MIoU
PCR
41.30
13.07
42.24
12.47
44.27
9.28
45.88
8.77
38.79
18.35
39.93
17.26
40.34
17.34
41.64
16.55
47.51
48.26
/
48.80
/
49.90
Average 42.53
15.71
43.47
14.87
44.47
13.31
45.81
12.66
Table 4: Performance comparison of multimodal robust solutions SURF dataset. Due to space constraints, we present only subset of comparison results here. Comprehensive results are
provided in Appendix A.11
Modal
SF-MD
MMFormer
CRMT-JT
MMANet SF-MD" Ace
MMANet
Rgh Depth
Acc(t) PCR())
Acc
PCR
Acc
PCR
PCR Acc
PCR
Ace
PCR
X
x
83.92
13.83
88.85
9.39
90.13 8.66
91.43 7.77 92.13 6.87
93.49 5.32
V
x
93.65
3.84
95.33
2.78
96.26 2.45
97.73 1.41 96.55 2.41
98.31
0.44
X
v
89.60
8.00
86.01
12.29
89.75
9.05
89.96
9.25 93.76 5.23
94.18
4.62
V
x
95.43
2.01
98.07
-0.01
95.68
3.04
98.39
0.75 98.25 0.69
98.64
0.10
v
93.26
4.24
95.23
2.89
91.81
6.96
96.99
2.16 96.62 2.33
96.60
2.17
x
V
v
96.74
0.67
96.90
1.18
-0.06 98.74
98.82
0.31 98.63 0.30
99.27 -0.54
V 97.39
98.06
98.68
99.13
/ 98.93 /
98.74
/
Average
92.85
5.43
94.07
4.75
94.44
5.02
96.06
3.61 96.41
2.97
97.03
2.02
use Acc to quantify models' classification performance, and employ PCR to measure the extent of performance degradation when model inference using incomplete inputs.
Baselines: We compare our method to a list of SOTA methods, including SF-MD, HeMIS (Havaei
et al. (2016)), RFNet (Ding al. (2021)). mmFormer (Zhang al. (2022)). CRMT (Yang et al. (2024c)). COM (Qian & Wang (2023), DCP (Hu et al. (2024)), and MMANet (Wei et al.][[2200)) (2023) ) SF (Zhang et al. (2019) the official network for the SURF, while SF-MD is variant specifically designed to simulate missing modality scenarios by employing modality access control factors.
We integrate MWAM into SF-MD and MMANet to demonstrate the per- formance gains that MWAM can bring to the host model. In SF-MD, we additionally introduce aux- iliary heads, whereas we do not in MMANet. We follow the experimental settings recommended by MMANet, including epochs, mini-batch size, etc. Additionally, we set the scaling factors in Eq. 5 1.5, 6, and 0.7, respectively. All experiments were conducted on an NVIDIA RTX 3090 GPU.
Implementation Details:
The results are shown in Table After incorporating MWAM, the foundational SF-MD model demonstrate significant boost in both average accuracy and PCR. Notably, the accuracy for the RGB modality, the weakest in single-modality scenarios, surged by 8.21%. This not only indicates that the vanilla SF-MD possesses untapped performance potential but also proves that MWAM can empower simple, original methods to overcome performance bottlenecks and enhance their robustness. Furthermore, when embed to MMANet, an already high-performing contemporary method for missing modalities, MWAM push its capabilities even further. Although it yields sub- optimal results on certain modality combinations, the MWAM-enhanced MMANet achieves the highest average score, which can be attributed to the model's balanced attention across different modalities during training. Remarkably, the performance of SF-MD, when augmented by MWAM, surpasses that of recent SOTA methods such mmFormer and CRMT-JT. In summary, these results collectively demonstrate that MWAM not only delivers substantial performance gains to base models but also enables existing robust models to break through their own inherent performance ceilings. More comparison results can be found in Appendix.
Results:
5.3 ABLATION STUDIES
We conduct comprehensive suite of ablation studies and parameter sensitivity analyses. present analysis here on the performance impact of our intervention mechanism due to space con- straints. The complete experimental results detailed in the Appendix t. A.10

---

Published as a conference paper at ICLR 2026
20
dush
2.
144
3-
2.2
00
SF-MD (Ad0 Intervvnntion)
SI-MDD (w/ Loss Intervention))
SF-MD (w/ GG/Gradient Intervention)
SF-MDD (wr HHbrrid Intervention)
Figure 4: Training losses for different interventions
Table 5: Impact of constraints at different levels on model performance metrics. We use SF-MD as
the baseline framework and conduct experiments on the SURF dataset.
Modal
w/o MWAM
Loss
Gradient
Hybrid
Loss+Grad
Rgb
Depth
Acc
PCR
Acc
PCR
Acc
PCR
Acc
PCR
83.92
13.83
89.42
9.37
89.36
9.67
92.13
6.87
X
x
93.65
3.84
95.12
3.60
95.27
3.70
96.55
2.41
89.60
8.00
91.25
7.52
94.26
4.72
93.76
5.23
x
95,43
2.01
97.18
1.51
97.06
1.89
98.25
0.69
V
93.26
4.24
95.72
2.99
96.61
2.35
96.62
2.33
X
96.74
0.67
98.05
0.63
98.64
0.29
98.63
0.30
97.39
/
98.67
98.93
/
98.93
Average
92.85
5.43
95.06
4.27
95.73
3.77
96.41
2.97
The quantitative results in Table that all intervention strategies improve performance over the baseline. Gradient-level intervention yields superior gains compared loss-level intervention. We hypothesize this is because manipulating losses alone can disrupt global training dynamics and introduce subtle optimization trade-offs in the fusion block. In contrast, a hybrid approach proves most effective, achieving a synergistic balance between global and local modality attention.
Fig. provides visualization of these training dynamics. The baseline model exhibits entangled loss curves for each modality, indicating a competitive and unbalanced optimization process. In contrast, embedding MWAM stratifics the modality losses: the Depth modality, having the highest FRM, consistently maintains the highest loss, while the other modalities cluster together at lower loss values. This stratification shows that MWAM effectively guides the model to prioritize learning from modalities with lower FRM. Furthermore, the overall training process becomes evidenced by the reduced variance in the total loss curve. This suggests that MWAM fosters a more balanced and synergistic learning environment, enabling the model to harmonize contributions from all inputs, overcome performance bottlenecks, and ultimately achieve superior performance.
stable, as
CONCLUSION
We propose simple yet efficient method to enhance the robustness of multimodal models from frequency-domain perspective. Our approach is motivated by the common observation that models often develop a learning bias towards easier modalities. leading the under-optimization other modality-specific pathways. To counteract this imbalance, we introduce two key contributions: (1) the Frequency Ratio Metrie (FRM), a novel indicator to quantify this modality preference, and (2) the Multimodal Weight Allocation Module (MWAM), plug-and-pla to dynamically modulate the influence of each modality in the gradient or loss space. This training intervention mechanism enforces more balanced learning dynamic, enabling existing multimodal architectures to break through performance ceilings with minimal overhead, and to enhance their robustness. We conducted experiments across diverse modality combinations, various visual under- standing tasks, and different backbone architectures to demonstrate the method's effectiveness and generality. Additionally, discuss the limitations of our method in Appendix A.14
component that uses FRM
10

---

Published as a conference paper at ICLR 2026
LLM USAGE STATEMENT
In this work, we only employed Large Language Model (LLM) auxiliary tool to polish the manuscript. Specifically, its use was focused on enhancing the clarity and fluency of the writing to better articulate our core insights and findings.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, the hyperparameters and environmental configurations for our main ex- periments are detailed in the manuscript, specifically within Sections 4.1 and 4.2.
IMPACT STATEMENT
This paper presents work whose goal advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China under Grant 62273353.
REFERENCES
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies, 2019. URL /ttps://arxiv.orq/
abs/190s 00425
Cheng Chen, Qi Dou, Yueming Jin, Hao Chen, Jing Qin, and Pheng-Ann Heng. Robust multimodal brain tumor segmentation via feature disentanglement and gated fusion, 2020. URL https:
09708
Yuhang Ding, Xin Yu, and Yi Yang- Rfnet: Region-aware fusion network for incomplete multi- modal brain tumor segmentation. 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 3955-3964, 2021.
doi: 10. N0.1109/CCC48992.202 .00394..
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation s early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network, 2019. URL https: w/tpivvv///xxvv.../oo//v/ wwwp:xiii.orgggaaa..o 910. 01255
Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Tianyuan Yuan, Yue Wang, Yang Yuan, and Hang Zhao. On uni-modal feature learning in multi-modal learning, 2023. URL https://
scenreview.net.. wtw.::/wwtweeeeeeeeeee
7VM8 7VMB3DkyC
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably opti-
mizes over-parameterized neural networks, 2019. URL htt.ttss//aaavii.or////////1/11 02054
Shi Guo, Hongwei Yong, Xindong Zhang, Jianqi Ma, and Lei Zhang. Apatial-frequeng attention for image denoising. 2023. URL batttp::/aarxiv.or///aa
13598
Mohammad Havaei, Nicolas Guizard, Nicolas Chapados, and Yoshua Bengio. Hemis: Hetero-modal 05194
image segmentation, 2016. URL htt.:/ttpp..//raav.oor//aaaa////
Xilin He, Jingyu Hu, Qinliang Lin, Cheng Luo, Weicheng Xie. Siyang Song, Muhammad Haris Khan, and Linlin Shen. Towards combating frequency simplicity- biased learning for domain generalization, /tt.httppee////iii.oorr////////a
16146
16146

---

Published as a conference paper at ICLR 2026
Lianyu Hu, Tongkai Shi, Wei Feng, Fanhua Shang, and Liang Wan. Deep correlated prompting for visual recognition with missing modalities, 2024. htt.tttt////axii..rrrr////////// 06558
Chengxiang Huang, Yake Wei, Zequn Yang. and Di Hu. Adaptive unimodal regulation for balanced multimodal information acquisition, 2025. URL https: /ttp://////////....././////////.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen-
eralization in neural networks, 2020.
Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Multimodal prompting with missing modalities for visual recognition, 2023, URL
03369
Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou, Boosting multi-modal
model performance with adaptive gradient modulation, 2023a. URL https://axxi..ott abs/ /200/230...m 07686
Siting Li, Chenzhuang Du, Yu Huang, Longbo Huang, and Hang Zhao. Modality complementari-
ness: Towards understanding multi-modal robustness, 2023b. URL http:tttt////peeeeiiie net heet/Eorumiddgfffi
Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(4):4447-4461, 2023. doi: 10. N0.1100TPMM1220221197333
Hong Liu, Dong Wei, Donghuan Lu, Jinghan Sun, Liansheng Wang, and Yefeng Zheng. M3ae: Multimodal representation learning for brain tumor segmentation with missing modalities, 2023.
URL https::/axxxi..oooot www.:iiiirrrreooooooooo wwtp://p/....b...o.oooo/ 25300
Xinwang Liu. Incomplete multiple kernel alignment maximization for clustering. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence, 1613):1412-1422 2024. doi: 10.1109/
1194992001.33116488
Xinwang Liu, Miaomiao Li, Lei Wang, Yong Dou, Jianping Yin, and En Zhu. Multiple kernel k- means with incomplete kernels. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, AAAI'17, 17, pp. 2259-2263. AAAI Press, 2017.
Xinwang Liu, Xinzhong Zhu, Miaomiao Li, Lei Wang, En Zhu, Tongliang Liu, Marius Kloft, Ding- gang Shen, Jianping Yin, and Wen Gao. Multiple kernel kk-means with incomplete kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(5):1191-1204, 2020. doi:
10.1109//AAM..2019..8900
Bjoern H. Menze, Andras Jakab, Stefan Bauer, and et al. The multimodal brain tumor image seg- mentation benchmark (brats). IEEE Transactions Medical Imaging, 34(100:1993-2024, 2015. doi: 10.1 100.00/M..01..337/0
Pushmeet Kohli Nathan Silherman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV, 2012.
Samet Oymak and Mahdi Soltanolkotabi. takes the shortest path?, 2018. URL http://ss//axxii..o////////////s 10001100211111110
Overparameterizee nonlinear learning: Gradient descent
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on- gradient modulation, 2022a. URL http::ttt://te ax wwwpw/wwowg//b////03335555
Xiaokang Peng. Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8228-8237, 2022b. doi: 2w.11099CVPR52268.22222.0
Shuwei Qian and Chongjun Wang. Com: Contrastive
model for incom-
plete multimodal learning. Neural Networks, 162:443-455 455, 2023. ISSN 0893-6080. doi: http::/tbb.rrg//..00.......mmm cetiace//rrticll//////2222222220
URL //www.
httpl//ppp////o.oceeeeeeeeeeee scwencedirect........... com/
12

---

Published as a conference paper at ICLR 2026
Yansheng Qiu, Delin Chen, Hongdou Yao, Yongchao Xu, and Zheng Wang. Scratch each other's back: Incomplete multi-modal brain tumor segmentation via category aware group self-support learning. In 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 21260- 21269, 2023.
doi: 10. N0.1199/CCVV100022222..001
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks, 2019. URL
https://arxi:: wrg.or////ao /awgg/bb//ie www.a///./..13333
Md Kaykobad Reza, Ashley Prater-Bennette, and M. Salman Asif. Robust multimodal learning with missing modalities via parameter-efficient adaptation, 2024. URL. https:://rxxv..oo
abs/2310. 63986
Daniel Seichter, Mona Köhler, Benjamin Lewandowski, Tim Wengefeld, and Horst- Gross. Efficient rgb-d semantic segmentation for indoor scene analysis. 2021 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 13525-13531, 2021. doi: 10.1 N0.1109/CCR448006
2021 95616655
Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin. Missing modalities imputation via cascaded residual autoencoder. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4971-4980 2017. doi: 10.1109/VVR22207..225
Shunxin Wang, Raymond Veldhuis, Christoph Brune, and Nicola Strisciuglio. What do neural net- works learn in image classification? a frequency shortcut perspective. In 2023 IEEE/CVE Interna- tional Conference on Computer Vision (ICCV), pp. 1433-1442, 2023. doi: 10.1 10.11000000
2023.00138.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the cigenvector bias of fourier feature net- works: From regression to solving multi-scale pdes with physics-informed neural networks. Com- puter Methods in Applied Mechamics and Engineering, 384:113938, 2021. ISSN 0045-7825. doi: https://doi.org 10. 101 10t.1//...m..22211113339999
URL https: www. www.eeiindddrrett com/
Zirui Wang and Yi Hong. A2fseg: Adaptive multi-modal fusion network for medical image seg- mentation. In Medical Image Computing and Computer Assisted Intervention MICCAI 2023, pp. 673-681, 2023.
Shicai Wei, Yang Luo, and Chunbo Luo. Mmanet: Margin-aware distillation and modality-aware regularization for incomplete multimodal learning, 2023, URL
2304. 08028
Yake Wei, Siwei Li, Ruoxuan Feng. and Di Hu. Diagnosing and re-learning for balanced multimodal learning. In Computer Vision - ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXIV, pp. 71-86, Berlin, Heidelberg. 2024. Springer- Verlag. ISBN 978-3-031--3338822
doi: 10. 00077978-3-031-7300050 org/ 2og//0.1007//7888 3-119978-----113333
URL https:///p.. //doi.
Jie Wen, Ke Yan, Zheng Zhang, Yong Xu, Junqian Wang, Lunke Fei, and Bob Zhang, Adaptive graph completion based incomplete multi-view clustering. IEEE Transactions on Multimedia, 23:2493-2504, 2021. doi: 10.1 MM100/TMM..200001::440000
Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1737-1746, 2020. doi: 10. 10CCC//CV22220000000000001.
Zhi-Qin John Xu, Yaoyu Zhang, and Tao Luo. Overview frequency principle/spectral bias in deep learning, 2024. URL https: // /aaaiiiiiv. viv. wwwiivooorgggaae wwwp/baa///.////oooo 07395
Mengping Yang. Zhe Wang. Ziqiu Chi, and Yanbing Zhang. Fregan: exploiting frequency compo- nents for training gans under limited data. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024a. Curran As- sociates Inc. ISBN 9781713871088.
13

---

Published as a conference paper at ICLR 2026
Shuzhou Yang, Moxuan Ding, Yanmin Wu, Zihan Li, and Jian Zhang. Implicit neural representation for cooperative low-light image enhancement. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12872-12881, 2023.
doi: 10.11091CCCV100000023...1111
Yang Yang, Fengqiang Wan, Qing-Yuan Jiang, and Yi Xu. Facilitating multimodal classi- fication via dynamically learning modality gap. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan. U. Paquet, Tomezak, and C. Zhang (eds.). Advances Neural In- formation Processing Systems. volume 37. pp. 62108-62122 Curran Associates, Inc., 2024b. URL
ngs nwurinurrip... http.r/pppp.ccppppee.eeeee/e//ep pdf
Zequn Yang, Yake Wei, Ce Liang. and Di Hu. Quantifying and enhancing multi robustness with modality preference, 2024c. URL BtL_utpps//axxx..rrr///////00 06244
Zequn Yang. Yake Wei, Ce Liang, and Di Hu. Quantifying and enhancing multi-modal robustness with modality preference, 2024d. URL httpss//aaaxx...rr///////22222.
Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, and Changqing Zhang. Multimodal fusion on low-quality data: A comprehensive survey, 2024a. URLhttps::/ http:/:://r...v...ee.eeee /bb abs//4/55-00 18947
Shifeng Zhang, Xiaobo Wang. Ajian Liu, Chenxu Zhao, Jun Wan, Sergio Escalera, Hailin Shi, Zezheng Wang, and Stan Z. Li. A dataset and benchmark for large-scale multi-modal face anti- spoofing. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 919-928, 2019. doi: 10. 20.1009/.P..201.0000000
Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learn- ing by alternating unimodal adaptation, 2024b. URL htpp:::/aax.ivvoor/////////////
10707
Yao Zhang, Nanjun Hc, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang. Yang Zhang, Zhiqiang He, and Yefeng Zheng. mmformer: Multimodal medical transformer for incomplete multi- modal learning of brain tumor segmentation, 2022. /tl.attps://axxl.vor///////
02425
Zheyu Zhang, Yayuan Lu, Feipeng Ma. Yueyi Zhang, Huanjing Yue, and Xiaoyan Sun. Incomplete multi-modal brain tumor segmentation via learnable sorting state space model. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 25982-25992, 2025. doi: 10.119//CVPP327742222550222
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
We provide theoretical rationale for the design of MWAM and FRM, more detailed experimental implementation, and richer description of the experimental content in the supplementary materials to provide reviewers with additional reference information.
In summary, this appendix is organized as follows:
Section A.1 provides the theoretical derivations underpinning the design of our FRM and
MWAM.
Section
presents the pseudocode for our proposed algorithm.
Section
offers further details on the datasets and experimental implementation.
Section
details the evaluation metries used for each task.
provides supplementary details and analysis for the experiments presented in Section 3.1 of the main paper.
Section A.5
Section A.6
includes an analysis of the computational overhead introduced by our method.
Section A.7
presents a sensitivity analysis the size of the frequency sampling window
in MWAM.
Section A.8
investigates the impact of different design choices for the FRM model
performance.
Section A.9
reports the hyperparameter sensitivity analysis for the weighting function (Eq.
5
evaluates the performance of MWAM under different training schemes, namely small batch and online learning settings.
Section
A.10
showcases performance the action recognition task using video and optical flow as input modalities.
Section A.11
Section A.12
highlights the performance gains achieved by applying MWAM baseline
model for mu timodal object detection.
Section A.13
provides a comparative analysis of MWAM against state-of-the-art methods for multimodal imbalance optimization.
Section A.14
discuss the limitations of MWAM and potential avenues for future work.
presents an extended comparison with wider range of methods addressing missing modalities, which was omitted from the main paper due to space constraints.
Section A.15
Section A.16 demonstrates the effectiveness of MWAM on the task of ine-grained visual classification.
A.1 PROOF OF THE THEOREM
In this section, we provide detailed proofs for the two theorems in Section pleteness the manuscript.
3.2 to enhance the com-
We adopt the notations established in the Preliminaries (Section 3.2 Specifically, we address
K-way classification task involving two modalities, denoted as Im, and Fmng-
Definition 1. Let the training dataset be defined as a collection of N samples:
D {x) .v' }==1,2,..N.
(8)
where each input x- = [ord m my ma consists of data from two modalities, and y² E (1,2..... represents the corresponding ground-truth label.
Definition 2. We define the multimodal classification model as a composite function:
f f(Em, Emy, C):
(9)
where Em, (:;6m,) and Emy (-; Omg) denote the feature encoders for the respective modalities, pa- rameterized by 8m, and fmy. C(:; W. represents the classifier head parameterized by weights and bias b.
15

---

Published as a conference paper at ICLR 2026
Definition 3. The model is optimized by minimizing the empirical risk via the Cross-Entropy loss, formulated as:
N
1 N
L =
log
(10)
E edlw)
F=1
Assumption 1. To facilitate the analysis of gradient behavior, we consider a canonical fusion ap- proach based on feature concatenation. Under this setting, the network output f(x²) is expressed as:
f(x)) W [Emu (immiimm @ Emy (172g 0mz)] + Wm, Em ( mm0) W Wm, Emg (1mm 0mg) + b,
(11)
where + denotes the concatenation operation.
A.1.1 PROOF OF THEOREM 3.1
Under the assumptions outlined above, when optimized using the gradient descent algorithm, the optimization procedure proceeds as follows:
For the parameters specific modality my (with analogous steps applying to modality m2), the update rules at iteration t re formulated
N
ØL
W mi
W mi
V VW,, L(W We m1
11
N
af(x))
Fims (Imm
Omn);
(12)
j=1
N
OL
O(W) Fmmy my
0++1 0r V 0mp L(O ) O
(13)
N
0ʃ
D0
j=1
where 7/ denotes the learning rate. The gradient of the loss with respect the network output f(x²) is derived as:
OL e (Win Fin (oring Brry Wang Fing (s ) 0f(r)) Ege (Win Emn ( Smn ) ) Wing Emg (# 00gg bbb)
1 1e
(14)
Eq. 12 and Eq. 13 indicate that the multimodal branches are optimized independently, with the sole exception of the coupling term af(2)). This implies that the parameter updates of one modality are not directly interfered with by the other, aside from this shared error signal.
aL the contributions of different modalities are aggregated via dot products. Within the term J/227" Consequently, Eq. 14 can be reformulated in unified vector notation
aL af(w)
WW +5).
lcmyy'>
(15)
K >0 (W-EF+0)
where W = [Wm, Wmy). E = [Emy Emng] represent the concatenated weights and embeddings, respectively, and superscript T denotes the matrix transpose. The convergence behavior of the gra- dients can be analyzed via the cigendecomposittinn of E.
Eq. 15 reveals critical insight: if one modal branch performs significantly better, it will dominate the shared gradient signal during training. Consequently, the optimization process becomes biased, effectively establishing that branch the model's 's preferred modality.
A.1.2 PROOF OF THEOREM 3.2
Definition 4. To facilitate the theoretical analysis, without loss of generality, we define a simplified setting involving a single modality m, over the dataset {< neural network, where 0 R represents the learnable parameters and e R2 is the input. The optimization objective is defined as the minimization of the squared error loss function I(O)):
. ) Let f(x f) denote the
(( Σ
(f 2²
(16)
16

---

Published as a conference paper at ICLR 2026
where the parameter sequence {6,} is generated via gradient descent with learning rate 77. governed by the update rule:
Bct1 = 82 ().)).
(17)
Dynamics Analysis. Let Gr = (f(r)) JEINNI E IR be the network output vector at iteration f, and = (o') )EE) E IRN be the target vector. Following prior works on the linearization of wide neural networks (Dong et al. (2019); Du et al. (2019); Oymak & Soltanolkotabi (2018)) the evolution of the residual (prediction error) can be approximated by quasi-linear dynamics:
g++1 (I nHH,)(e v),
(18)
where He is the empirical Gram matrix defined as:
H. = (Vo 0f O2 + a(0+41 01)) de,,
Of 0.)
(19)
00
00
j
Crucially, as the network width - oo, it has been established that H, converges to a static matrix Hr known the Neural Tangent Kernel (NTK) (Jacot et al.) (2020); Oymak & Soltanolkotabi 2018 Specifically, ||H, - H| o(1). Thus, in the infinite-wide limit, we have H, H", where H is a positive semi defindefinite x N matrix.
Let \ 2 A2 2 An > 0 denote the eigenvalues of H and let u1, Ug, o. un be the corre- sponding orthonormal eigenvectors. The spectral decomposition H* is given by:
H = Σ A.aaT.
(20)
We now analyze the optimization trajectory by projecting the error signal onto the eigenvector basis. The dynamics in the direction of the i-th eigenvector u; are:
(+++1 y, uz) = ((I - n/H (g. v), us) (1 (a= y, us).
(21)
Eq. 21 demonstrates that the convergence rate along each cigen-direction is determined by the factor (1 n)))). Consequently, components corresponding to larger eigenvalues A decay significantly faster.
A.2 PSEUDO-CODE FOR THE ALGORITHMIC
We continue the assumption of Section A.I that D {a", y² >=1.2..... f(Em, Emy : O), et al. To make it easier to express, discard here the bias in C. Therefore, Algorithm [ demonstrates the training procedure for the underlying model + MWAM.
A.3 MORE DETAILED EXPERIMENTAL IMPLEMENTATION
We provide supplementary details on the implementation of each experiment according the quence outlined in the main text.
We conducted multimodal classification experiments on CASIA-SURE, CASIA-SURF is a large- scale benchmark dataset for face anti-spoofing, consisting of 87k sets of RGB, Depth. and IR im- ages. We integrated MWAM into the SF-MD and MMANet models, resulting in SF-MI SF-MD+WWAM and MMANet-WWAM. We followed the experimental settings recommended for MMANet, specif- ically: we set the mini-batch size 64 and trained the models for 100 epochs using the SGD opti- mizer. For the SGD optimizer, weight decay and momentum were set to 0.0005 and 0.9, respectively. The initial learning rate was set to le-3 with learning rate decay strategy: SF-MD+MWAM, the learning rate was divided by 10 every 30 epochs, while in MMANet+MWAM. it divided by 10 at epochs 16, 33, and 50. Additionally, the parameters (a. 5) in MMANet were set to (30, 0.5). All images were resized 112x 112 pixels before being fed into the network.
We conducted multimodal semantic segmentation experiments on NYU-Depth V2. NYU-Depth V2 is large-scale dataset for indoor scene semantic segmentation with 1,449 images containing
17

---

Published as a conference paper at ICLR 2026
Algorithm Training of underlying Model+MWAM.
Input: D = { Zny y² J iterations: T,, learning rate: and other input parameters in baseline.
1: for = 0, ..., T. 1 do
2: Sample mini-batch 8² {Imis Zmgy y}:
(high))))
3: Extract frequency components mi (122) using MWAM;
4: Calculate F (rM.) ) and F FRM
using Eq. 4
5:
Update FRMs using Eq. [ and calculate weights Kmy and Km, using Eq. 5 and Eq. 6 Feed-forward the batched data B to the model;
6:
7: Calculate the loss using Eq. and compute the gradients g( g(0m))) and g(8))) using
backpropagation:
8: Update the gradients according to the following rules:
g 0r Km, (1) g
o. 0g Kmg (1) g(0
OC + Of - g(6))
9: end for
RGB and Depth modalities. We integrated MWAM into the ESANet-MD and MMANet models, resulting in improved versions: ESANe--MD+MWAM and MMANet+MWAM, We followed the experimental settings recommended for MMANet: the mini-batch size was set to 8, and the models were trained for 300 epochs using the SGD optimizer. For the SGD optimizer, weight decay and momentum were set to le-4 and 0.9, respectively. The initial learning rate was set to 1e-2, and PyTorch's one-cycle scheduler to (4, 0.2). All images were resized to x80x640 pixels before being fed into the network.
used. Additionally, the parameters (a,, B) in MMANet were set
We conducted multimodal brain tumor segmentation experiments BRATS2020 which used for the MICCAI Brain Tumor Segmentation Challenge, includes 369 training subjects with modalities each. We integrated MWAM into the RFNet and mmFormer models, resulting in improved versions: RFNe++MWAM and munFormrr+WWMM For model training, we set the mini-batch size to 2 and used the Adam optimizer (31 0.9, 82 0.999), training for 300 epochs and 1000 epochs, respectively. The weight decay for the Adam optimizer was set to le-5 and le-4 for the two models, respectively, and both used the "poly" learning rate policy. All images were resized 80 pixels before being fed into the network.
A.4 EVALUATION METRICS
We use Accuracy (Acc) to quantify the model's performance multimodal binary classification tasks, Mean Intersection over Union (MIoU) to measure the model's performance on multimodal semantic segmentation tasks, and the Dice coefficient for the whole tumor evaluate the model's performance on multimodal brain tumor segmentation tasks. In addition, we use the Performance Collapse Rate (PCR) to assess the performance collapse for different incomplete modality combi- nations across various tasks.
We conducted a multimodal binary classification task in the body of our text; therefore, the accuracy discussed in this paper should be binary accuracy. Binary accuracy is mathematically represented as follows:
TP TN
Acc TN + +N
(22)
where TP, TN, FP, and FN represent True Positives, True Negatives, False Positives, and False Negatives, respectively.
Intersection over Union (IoU) is metric used to evaluate model performance in image segmentation tasks. MIoU is the average of loU values across categories, commonly used to assess multi-class segmentation tasks. Mathematically, the MIoU calculation formula as follows:
N
MloU
loU,
(23)
N
18

---

Published as a conference paper at ICLR 2026
where:
TP:
IoU; TP; + FP, FN;
(24)
where TP,, FP,, and FN; represent True Positives, False Positives, and False Negatives for class i, respectively. Higher MIoU values indicate hetter performance in multimodal semantic segmentation
The Dice coefficient (also known the Dice similarity coefficient) is metric used to measure the similarity between two sample sets and one of the most commonly used metrics for brain tumor image segmentation tasks. Its calculation formula is follows:
2. 11 vl1
Dice v)
(25)
11 + 11v112
where y and i represent ground truth and predictions, respectively. Larger Dice scores indicate that the predictions are more similar the ground truth, thus reflecting better segmentation accuracy.
Furthermore, we use the PCR to quantify model performance degradation under incomplete modal- ity imputs. PCR is defined as the rate of performance decline when using various combinations of incomplete modality imputs compared to complete modality inputs, mathematically represented as:
PCR ( 0)) - A(M (LMM(siise) (Fmiss: 0))
(26)
A(M (a 0))
where A(-) represents evaluation metrics for different multimodal tasks, (M,0-) a multimodal model parameterized by 0, full denotes the complete modality input, and Timis denotes different combinations of incomplete modality inputs. Generally, lower PCR value indicates that the impact of missing modality inputs model performance is smaller, meaning the performance is closer to that with complete modality inputs.
A.5 MORE SUPPLEMENTARY NOTES ON FREQUENCY DIFFERENCE EXPERIMENTS
In Section 3.1 in order to illustrate that different frequency bands in the frequency domain have different levels of influence in the multimodal visual comprehension model, used different types of filters with different window sizes to process the dataset post-training, and obtained the corre- sponding results and conclusions. However, due to space constraints, we present more detailed experiments here.
We first use FFT processing in the spatial domain for the three modalities RGB, Depth and IR, and move the DC component to the center of the image. After that, a filter of size x n is designed with the image center as the reference. Among them, the low-pass filter retains only n X n pixel points in the image center, and the other positions are set to O. The high-pass filter discards X pixel points the image center. Finally, the iFFT process is performed using the filtered spectrogram to obtain the filtered spatial domain image, and the results of each stage of the operation are shown in Figure 5
In Figure 5 images of different modalities present different visual effects in the spatial domain, c.g., the RGB image contains richer textures and details, and the Depth map contains more reflective of the structural hierarchy, despite the missing of these textures and details information. However, when these images are converted to the frequency domain, the differences are more obvious, e.g., the energy of the Depth map is more concentrated in some small number of frequency bands, while the energy of the RGB and IR images is more scattered. In addition, the processed images with low-pass and high-pass filters having different window sizes exhibit different characteristics. For example, the high-pass filter aims to preserve the tigh-frequency components of the image, and the processed image retains the edge information in the image. Compared to a high-pass filter of size 30 x 30, filter of size 15 x 15 retains more frequency components, and therefore it retains more edge information. Compared to a high-pass filter of size 30 x 30, a filter of size 15 x 15 retains more frequency components, and therefore it retains more edge information. The low-pass filter aims to retain the wwwffreueeccy components in the image, and the processed image retains the texture, detail features in the image. However, since the low frequency energy is more concentrated, the low pass filter size has a greater impact on the results. Low-pass filters using 15 x 15 retain most of the feature detail in the image, but they do not retain much detail the results of the 30 x 30
19

---

Published as a conference paper at ICLR 2026
RCB
Depatt
Domain
Spatial
Froquency
Dumain
Fillers
Resuls
IFFT
Fillers
Resulls
IFFT
Fillers
Resulls
IFFT
Fillers
Resulls
IFFT
Input Images
High-Pass
15x15 +]
High-Pass 30x30 +]
Low-Pass
15x15 M M
Low-Pass 30x30 H
Figure 5: Comparison of visualization of filter effects. In the horizontal axis, it can be divided into five parts, which are the spatial and frequency domain images of the input image, two high-pass filter results with different window sizes, and two low-pass filter results with different window sizes.
Table 6: Computational cost analysis MWAM. The Parameters and FLOPs statistics are reported for the entire model (with and without MWAM), while the Inference Time is measured for the MWAM module in isolation.
Semantic
Brain Tumor
Tasks
Classification
Segmentation
Segmentation
MMANet +MWAM
ESANet-MD
RFNet
Models
MMANet
ESANet-MD
RFNet
+MWAM
+MWAM
13.18
71.69
8.4
Params. (M)
13.18
71.69
8.4
(+0)
(40)
(+0)
60.6752
526.6627
204.5604
Flops (G)
60.63
526.60
204.56
(+0.0452)
(+0,0627)
(+0.0004)
Inference time
0.0385#0.005
0.0878+0.0040
0.0542+0.0139
size filter. It is also worth noting that we set the window sizes of the filters to 15 and 30 just to illustrate the general pattern found, and that there special values taken.
Therefore, the extent to which the model benefits from the different frequency bands of these images different. There reason to believe that multimodal preferences can be reasonably modeled from frequency bands.
A.6 COMPUTATIONAL OVERIIEAD OF MWAM
To assess the computational overhead of MWAM, we report the parameter count, FLOPs, and MWAM inference time for the three main experiments, both with and without the module. These results are detailed in Table
The experimental results are highly promising. The computational overhead of MWAM, primarily attributed to DCT and FRM computations, is negligible. This allows it to yield significant per- formance gains for existing baselines at minimal marginal cost, even for those that are already robust. Crucially, MWAM is engineered to refine the model's optimization trajectory exclusively during the training phase. The module is detached for inference, thereby incurring zero additional computational overhead at runtime.
A.7 IMPACT OF FREQUENCY SAMPLING WINDOW SIZE ON MODEL PERFORMANCE
In MWAM, sample the input frequency components using qxq window from patches of size pxp. In our work, we set p 8 to align with most baseline methods. The size of the frequency sampling window y directly affects the caleulation range of FRM. As q increases, more frequency
20

---

Published as a conference paper at ICLR 2026
Table 7: Impact of frequency selection window size model performance. We embed MWAM into MMANet and vary the sampling window size
In the table, "wo MWAM" is vanilla MMANet.
Modal
w/o MWAM
q=1
q=2
q=4
q=6
Rgb
Depth
Ir
Acc(t)
PCR())
Acc
PCR
Acc
PCR
Acc
PCR
Acc
PCR
x
x
91.43
7.77
93.07
5.92
93.49
5.32
92.43
6.54
89.93
9.22
X
v
x
97.73
1,41
98.62
0.31
98.31
0.44
98.61
0.29
98.51
0.56
X
X
v
89.96
9.25
90.62
8.40
94.18
4.62
91.90
7.08
91.18
7.95
X
98.39
0.75
98.86
0.07
98.64
0.10
98.68
0.22
98.00
1.07
X
v
96.99
2.16
96.52
2.44
96.60
2.17
96.19
2.74
95.71
3.38
98.82
0.31
99.18
-0.25
99.27
-0.54
98.65
0.25
99.14
-0.08
99.13
/
98.93
/
98.74
98.90
/
99.06
/
Average 96.06
3.61
96.54
2.82
97.03
2.02
96.48
2.85
95.93
3.68
Table 8: Comparison of the impact of FRM calculation rules model performance. In the table, "w/o MWAM" is vanilla MMANet, "w/o High-Freq."" is in the form of an Lı-Norm sum using the low-freoe wwwfrreuency components, as in Eq. 3 "w/ High-Freq." " is using Sum" is a directly summed -Nrmo shown in Eq. 27 and "Weighted Sum" is the L1-Norm weighted sum of the low-frequency and gegufrreyeency components, as shown in Eq.
FRM as in Eq 4 "Direct the low-frequency and high-frequence components, as
28
All experiments were conducted in the same
environment.
w/o High-Freq.
w/ High-Freq. Acc
Modal
w/o MWAM
Direct Sum
Weighted Sum
Rgb
Depth
Ir
Acc(t)
PCR(L)
Acc
PCR
PCR
Acc
PCR
Acc
PCR
X
x
91.43
7.77
90.79
7.54
93.49
5.32
92.28
6.53
91.97
7.29
X
V
x
97.73
1.41
98.38
-0.19
98.31
0.44
97.36
1.39
98.30
0.91
X
X
V
89.96
9.25
88.73
9.63
94.18
4.62
90.67
8.16
90.68
8.59
V
x
98.39
0.75
98.21
-0.02
98.64
0.10
98.58
0.15
98.87
0.33
96.99
2.16
92.28
6.02
96.60
2.17
95.18
3.60
95.80
3,43
X
98.82
0.31
98.42
-0.23
99.27
-0,54
98.98
-0.25
98.83
0.37
99.13
98.19
98.74
98.73
99.20
Average
96.06
3.61
95.00
3.79
97.03
2.02
95.96
3.26
96.23
3,49
points are included in the computation, impacting the output of MWAM. Therefore, we explore the effect of the sampling window size in MWAM.
Table 7 shows the performance differences of MMANet+MWAM when only the sampling window size q changed under the experimental configuration. As increases, more frequency com- ponents are included in the FRM calculation, leading to a noticeable performance improvement. However, this enhancement has an upper limit. When the patch size p fixed, increasing the win- dow size beyond a certain point leads to performance stagnation or even negative optimization. In our experiments, when q Z 4, the experimental results of MMANet+MWAM show degradation compared to MMANet. This degradation is not random. We believe this occurs because, at 4, irrelevant information from mid-frequency components interferes with the FRM calculation. At q 6, the sampled frequency components experience certain degree of aliasing, leading to emros in the FRM calculation. As a result, this causes a deviation in the training direction of the model when guided by MWAM. This also demonstrates the effectiveness of proposed method from another perspective. By intervening in the model's optimization path, can surpass its existing performance levels.
A.8 IMPACT OF FRM CALCULATION RULES ON MODULE PERFORMANCE
In section we introduced two methods for measuring multimodal preference, differing based on the inclusion of nigh-frequency components in the FRM calculation. To evaluate these methods, tested the multimodal classification performance MMANe++MWAM on the SURF dataset under identical experimental settings. The results presented in Table g
21

---

Published as a conference paper at ICLR 2026
The results are consistent with our hypothesis. Excluding high-frequency components from the pref- erence measurement leads to noticeable decline in experimental performance due to the loss of cru- cial information, resulting in suboptimal outcomes. Specifically, performance drops by 1.06% com- pared to the standard MMANet and by 2.03% compared to the FRMM-ased measurement method. This suggests that measuring multimodal model preferences using the FRM with high-frequency components (w/ High-Freq.) provides a more substantial performance advantage than employing the Li-Norm with low-frequency components (w/o High-Freq.). This finding supports our hypoth- esis outlined in Section 4.2 which posits that high- high-frequency components in multimodal image understanding tasks have measurable impact on model performance, although this impact less pronounced than the performance improvements achieved through low-frequency components.
Furthermore, we explored two additional methods for measuring multimodal preferences that con- sider both low-frequency and high-frequency components. Unlike FRM, these methods use a fre- quency component summation approach.
Firstly, we define the measurement method as the sum of the Li-Nomms of the low-frequency and high- figh-frequency components, which can expressed as:
w Σ
M MP(#m.)
(1 (a, b) [high (a, (a,b)) b)
(27)
where w and h represent the width and height of the frequency features, respectively, while m. (a, and Thigh (a, b) denote the low-frequency and high-frey components of Im,-
Tlo
Additionally, highlight the varying importance of different frequency bands for multimodal im- age understanding tasks, we use a weighted sum of the different frequency bands, which can be expressed as:
MP(mm.)
Σ
(w] Lay (a, b)] + (1 - w) (a, b]l)
(28)
hm
where w is a weight factor for different frequency bands. In our experiment, we set w 0.9.
The experimental results are shown in columns 5 (Direct Sum) and (Weighted Sum) of Table R First, it is undeniable that the performance of the experimental groups that fully considered both low-frequency and high-frequency components was better than that of the experimental groups that completely discarded high-frequency components. This further proves that our approach of retaining high-frequency components is valid. Second, directly summing the L1-Norms of the frequency components leads to a significant performance drop. This is because such a measurement scheme blurs the energy boundaries between frequencies, making the impacts of high-frequency and low- frequeney components similar, thus preventing the model from properly allocating weights based on low-frequency components. Furthermore, after applying weights to diffcrent frequency bands, the situation improves significantly. However, applying fixed weight factor is still suboptimal because the sporadic zeros in high frequencies can alter the preference measurement calculation (sum calculations aggregate all energy). Therefore, employs proportional factors to address the low-frequency components at positions corresponding zeros in high- frequeeyy features.
ratio-based approach, like FRM, which
Therefore, we choose FRM as the metric for measuring multimodal preferences and use it to opti- mize the model.
A.9 PARAMETER SENSITIVITY ANALYSIS FOR THE WEIGHT ALLOCATION FUNCTION
Fundamentally, Eq. represents variant of the sigmoid function that has been horizontally re- flected to create a decreasing curve. Its behavior is governed by four hyperparameters that enable independent scaling and translation along both the and y axes. We use this function to process the output of Eq. 66] which quantifies the relationship between a single modality's FRM and the mean FRM across all modalities, constraining the input value, T, the range [0. 2].
However, directly applying a standard or a simple reflected sigmoid for weight allocation presents two significant challenges. First, within the input range of [0, 2). the standard sigmoid function exhibits low sensitivity, meaning its output values are clustered within a narrow band. This results in minimal differentiation between the weights assigned to various modalities, thereby failing to
22

---

Published as a conference paper at ICLR 2026
Table 9: Parameter sensitivity analysis for the weight allocation function.
Parameter Combination
Average Accuracy Performance Collapse Rate
(a ß ty X)
(Acc)
(PCR)
6.0
96.48
3.34
1./ 0.7 6.0
96.42
3.09
1.5/1.30.7 /.06
96.04
3.45
96.53
2.32
L./660
96.73
2.57
L./
96.37
3.30
1.7
96.07
3.51
0.
95.87
3.26
2./1.5/1376.5
97.08
2.57
1.5/1.0/0.
97.03
2.02
effectively modulate the model's focus. Second, the sigmoid's output range of (0, 1) inherently suppresses the contribution of every modality during optimization. Although the degree of suppres- sion varies, this universal dampening effeec can lead model underfitting, scenario where overall performance sacrificed for the sake of enforcing training balance.
To overcome these limitations, we introduce the four aforementioned parameters to flexibly scale and translate the reflected sigmoid function. Recognizing that the disparity in FRM values across modalities varies significantly with different multimodal tasks and training paradigms, formulate these parameters as tunable hyperparameters. This design choice enhances the generalizability and adaptability of our MWAM.
We conduct an additional parameter sensitivity analysis for Eq. and Eq. 6 with the results pre- sented in Table 9
Experimental results demonstrate that the model's final performance is largely insensitive to these four adjustable scaling factors when they within reasonable range. However, excessively im- balanced settings-foo instance, assigning weights to different modalities that span several orders of magnitude- can misguide the training process and lead to a collapse in overall performance.
A.10 EVALUATING MWAM's PERFORMANCE UNDER DIVERSE TRAINING SCENARIOS
Since MWAM dynamically adjusts modality weights on batch-by- bytthhhy-atchh basis using spectral infor- mation, it inevitably required to operate under diverse, and sometimes extreme, training scenarios. While experiments in the main paper already cover spectrum of batch sizes-free large (64 for classification) to small (8 for semantic segmentation and for brain tumor segmentation)-ww conduct supplementary analysis here to further assess the robustness of MWAM. Specifically, evaluate its performance on the classification task across a wider range of settings, including medium and small batches, as well as extreme online learning scenario (i.e., batch size of 1). The results are presented in Table
As shown in the table, while the Acc and PCR metrics fluctuate with decreasing batch size, this volatility is primarily attributed to MMANet. In contrast, our MWAM maintains relatively stable performance. This stability stems from two key components: the scaling factor (Eq, 5 and the FRM Bank. The scaling factor allows for manual adjustments to adapt to diverse learning conditions, while the FRM Bank dynamically compensates abnormal FRM values. This dual mechanism particularly beneficial for small- batch learning. For instance, in the extreme cases of bs 4 and bs 1, we adjusted the hyperparameters
1.5, 1, 1, and 4, respectively.
A.11 PERFORMANCE MWAM ON ACTION RECOGNITION
To further demonstrate the adaptability of MWAM diverse modality combinations, conducted an additional experiment on the action recognition task. This task utilizes video and optical flow as input modalities the UCF-101 dataset. We benchmarked approach against several modality- balancing baselines (Peng et al. (2022a); Yang et al. (2024b)), with the results summarized in Table
23

---

Published as a conference paper at ICLR 2026
Table 10: Training models with different batch sizes. Bold indicates the best metric under the same batch size.
bs=64
bs=64
bs=16
bs=16
Modal
w/o MWAM
w/ MWAM
w/o MWAM
MWAM
Rgb
Depth
Ir
Acc(t)
PCR())
Acc
PCR
Acc
PCR
Acc
PCR
91.43
7.77
93.49
5.32
87.89
11.32
90.57
8.17
/
x
97.73
1.41
98.31
0.44
97.83
1.29
97.36
1.29
x
89.96
9.25
94.18
4.62
89.16
10.04
94.03
4.66
/
98.39
0.75
98.64
0.10
97.01
2.12
98.34
0.29
X
96.99
2.16
96.60
2.17
96.21
2.93
95.79
2.88
98.82
0.31
99.27
-0.54
98.65
0.46
98.29
0.34
99.13
98.74
/
99.11
98.63
/
Average
96.06
3.61
97.03
2.02
95.12
4.69
96.14
2.94
bs=4
bs=4
bs=1
bs=1
Modal
w/o MWAM
w/ MWAM
w/o MWAM
w/ MWAM
Rgb
Depth
Acc(t)
PCR())
Ace
PCR
Acc
PCR
Acc
PCR
87.10
11.80
89.94
9.31
76.66
11..74
72.02
4.69
97.84
0.92
98.52
0.66
88.70
-36.23
74.18
1.83
X
X
v
94.21
4.60
93.38
5.84
74.71
-14.74
80.98
-7.17
V
V
x
96.49
2.29
98.35
0.83
71.59
-9.95
74.32
1.64
95.63
3.16
96.51
2.68
75.16
-15.44
75.55
0.01
X
V
V
98.33
0.43
99.21
-0.04
70.06
-7.60
77.78
-2.94
98.75
99.17
65.11
75.56
Average
95.49
3.86
96.44
3.21
74.57
-6.955
75.77
-0.32
Table 11: Performance comparison of video-optical flow multimodal classification tasks.
Modal
Baseline
MWAM
OGM-GE
LFM
Vidio
O.F.
Acc(t)
PCR(.)
Acc
PCR
Acc
PCR
Acc
PCR
x
74.41
7.81
72.36
11.36
73.47
8.44
73.76
9.45
X
55.68
31.01
60.45
25.95
53.86
32.88
57.27
29.70
80.71
81.63
80.24
81.46
/
Average
70.27
19.41
71.48
18.65
69.19
20.66
70.83
19.57
Notably, we did not perform any task-specific hyperparameter tuning for this experiment, adhering to the default configuration reported in the main text. The results demonstrate that our MWAM enhances model performance the absence of the dominant modality (video), thereby boosting its robustness. Furthermore, our method outperforms two other leading fate--o-the-wee art approaches, demonstrating superior robustness comparison.
A.12 PERFORMANCE OF MWAM ON MULTIMODAL DETECTION
To further demonstrate the generalizability of MWAM to diverse multimodal tasks, we evaluate its performance on multimodal object detection using visible and infrared (RGB-IR) imagery.
To validate the effectiveness of MWAM, we integrated it into the Two-Stream YOLOv8n baseline and evaluated its detection performance on the Vehicle dataset. As detailed in Table 12, our module demonstrates significant improvement over the baseline model. The visualization results of the recognition task and their corresponding PR curves are presented in Figs. 6 and 7 offering
qualitative corroboration for the quantitative findings summarized in Table T2
should be noted that, a direct comparison with existing methods is not straightforward, as ap- proaches for handling missing modalities balancing multimodal inputs not tailored for the multimodal detection task. Therefore, this section presents the experimental results for MWAM exclusively.
24

---

Published as a conference paper at ICLR 2026
Table 12: Performance comparison of MWAM in the multi-modal detection task.
Modal
T-yolov8n
Lyolov8n+MMWAM
RGB
IR
mAP50(1)
PCR(L)
mAP50
PCR
x
0.215
72.78
0.628
21.89
X
0.669
15.32
0.738
8.21
0.790
0.804
Average
0.558
44.05
0.723
15.05
333
31
Tayelo
Complete
Miming infrared
Missing
(0)
Figure Qualitative results for object detection.
A.13 COMPARISON WITH MORE COMPARATIVE METHODS
In addition to the methods in the text that focus improving robustness missing modalities, we provide a performance comparison of MWAM with advanced multi-modal balanced training methods in recent years. These methods include: OGM- DGM-GE(Peng et al. (2022a)), LFM(Yang et al. (2024)), AGM(Li al. (2023a)), MSES and MSLR. We performed multi-modal classification tasks under the same experimental conditions. Due width constraints, into Tables and [4] Bold indicates the optimal metrics, and italic indicates the suboptimal ones. Our method achieves most optimal and some suboptimal results, showing it effectively enhances model robustness by balancing multi-mode modal preferences.
(2.2333))))))
split the comparison tables
A.14 LIMITATION
While proposed MWAM has demonstrated consistent performance gains, have identified several promising directions for future enhancement. Key directions for future research include mitigating the model's sensitivity its initial training state. This is motivated by our observation that the proposed method performs better when trained from scratch than when initialized with pre- trained weights. We attribute this the greater exploratory freedom in the gradient space afforded by random initialization. Conversely, pre-tre model operates within constrained, fine-tuning regime, which limits this exploration. Consequently, future work should focus on two areas: (1) enhancing the representational power of the pre-trained encoder, and (2) refining the formulation of quantitative evaluation metries.
Furthermore, we plan to refine specific components of the model. The FRM module currently mea- sures modality preference using frequency- weights, but does not explicitly disentangle the frequency contributions of each modality. A deeper analysis of these internal frequency dynamics could provide more granular control and potentially boost performance. Additionally, the fixed scal- ing factor in Eq. 5 represents potential constraint. We will explore adaptive learnable scaling strategies to enhance the model's dynamic range and overall efficacy. These research avenues will form the core of our subsequent work.
25

---

Published as a conference paper at ICLR 2026
T-yoks
T-yoko Gla +MWAMM
()) Complete modal
(b) Missing Infrared Modality
(c) Missing RGB Modality
Figure 7: Comparative analysis of PR curves.
Table 13: Comparison with different balanced multimodal training methods. To adapt to some
comparison methods, we trained our MWAM using pairwise combinations of three modalities. All
methods used the vanilla ResNet18 structure and concat- based fusion, and were tested on the SURF
dataset.
Modal
OGM-GE
LFM
AGM
MWAM
Rgb
Depth
Acc(t)
PCR(.)
Acc
PCR
Acc
PCR
Acc
PCR
x
79.87
17.43
69.29
29.67
88.81
9.86
93.99
4.96
x
87.49
9.55
98.81
-0.29
94.42
4.16
97.68
1.22
96.73
98.52
/
98.52
98.89
Average
88.03
13.49
88.78
14.69
93.92
7.01
96.85
3.09
Rgb
Ir
Acc(1)
PCR(J)
Acc
PCR
Acc
PCR
Acc
PCR
X
87.23
9.71
88.21
2.86
82.23
11.11
93.34
3.40
X
88.02
8.89
76.30
15.98
85,02
8.10
89.60
7.28
96.61
90.81
92.51
96.63
Average
90.62
9.30
85.11
9.42
86.59
9.60
93.19
5.34
Depth
Ir
Acc(1)
PCR())
Acc
PCR
Ace
PCR
Acc
PCR
X
86.79
12.40
97.28
1.34
96.84
0.83
98.12
1.00
X
70.17
29.18
67.85
31.19
65.52
32.90
91.70
7.48
99.08
98.60
97.65
99.11
Average
85.35
20.79
87.91
16.26
86.67
16.87
96.31
4.24
A.15 COMPARISON WITH MORE MULTIMODAL ROBUSTNESS METHODS
Due to space constraints in the main paper, we present extended performance comparisons hetween MWAM and other state-of-the-art methods here. These evaluations cover experiments on both brain tumor segmentation and multimodal classification. The detailed results for the brain tumor segmen- tation and multimodal classification tasks are provided in Table 15 and Tables 16 17 respectively.
A. PERFORMANCE ON FINE-GRAINED CLASSIFICTIIN TASKS
Given that the design of FRM prioritizes the role of low-frequency components across modalities, a natural question arises: Is the effectiveness of our MWAM limited to low-frequency-dominnd tasks, and does it underperform in trrquunccydoomnne we conduct experiments on fine-grained classification, a task inherently driven by figh-freeuency details, to demonstrate the broad applicability and robustness of MWAM.
scenarios? To investigate this concern,
26

---

Published as a conference paper at ICLR 2026
Table 14: Comparison with different balanced multimodal training methods. This table serves
continuation of Table [3
Modal
MSES
MSLR
MWAM
Rgb
Depth
Acc(t)
PCR())
Acc
PCR
Acc
PCR
X
85,86
8.57
70.72
25.96
93.99
4.96
84.95
9.54
79.79
16.47
97.68
1.22
93.91
/
95.52
98.89
/
Average
88,24
9.06
82.01
21.22
96.85
3.09
Rgb
Ir
Acc(f)
PCR(L
Acc
PCR
Acc
PCR
X
80.84
14.80
76.67
18.50
93.34
3.40
84.04
11.42
78.52
16.53
89.60
7.28
94.88
/
94.07
96.63
Average
86.59
13.11
83.09
17.51
93.19
5.34
Depth
Acc(T)
PCR(L)
Acc
PCR
Acc
PCR
X
82.52
15.36
79.79
16.13
98.12
1.00
V
79.67
18.28
69.74
26.69
91.70
7.48
97.49
95.13
99.11
Average
86.56
16.82
81.55
21.41
96.31
4.24
Table 15: Performance comparison of multimodal robust solutions BRATS2020 dataset. This table serves as a continuation of Table
Modal
HeMIS
Robust Seg
M3AE
LS3M
A2FSeg
F
TI
Tic
T2
Dice(1)
PCR())
Dice
PCR
Dice
PCR
Dice
PCR
Dice
PCR
X
X
X
V
79.85
6.27
82.20
8.13
85.17
4.99
86.34
6.26
85.36
6.52
X
X
V
64.58
24.19
71.39
20.21
77.65
13.38
79.36
13.84
77.86
14.73
X
V
X
X
63.01
26.04
71.41
20.19
74.88
16.47
79.35
13.85
75.86
16.92
V
X
X
52.29
38.62
82.87
7.38
85.86
4.22
88.14
4.30
86.46
5.31
X
X
84.45
0.87
85.97
3.91
86.51
3.49
88.01
4.45
87.74
3.91
72.50
14.90
76.84
14.12
78.18
12.78
82.73
10.18
80.59
11.74
X
65.29
23.36
88.10
1.53
87.98
1.85
88.93
3.45
88.85
2.69
82.31
3.38
85.53
4.40
86.34
3.68
87.30
5.22
87.87
3.77
81.56
4.26
88.09
1.54
89.20
0.49
90.48
1.76
89.20
2.31
69.37
18.57
87.33
2.39
88.36
1.43
90.43
1.82
88.36
3.23
73.31
13.95
88.87
0.67
88.46
1.32
90.29
1.98
89.32
2.18
X
83.03
2.54
89.24
0.26
89.54
0.11
90.79
1.43
89.96
1.48
84.64
0.65
88.68
0.88
89.54
0.11
90.11
2.17
90.00
1.43
X
85.19
0
86.63
3.17
86.94
3.01
88.27
4.16
87.64
4.02
85.19
7
89.47
89.64
92.11
91.31
1
Average
75.10
12.68
84.17
6.34
85.62
4.81
87.51
5.35
86.43
5.73
A.16.1
THE INCREASED RELIANCE OF PINE-GRAINED CLASSIFICATION ON -REEUEREQ COMPONENTS
We first conduct a series of experiments to demonstrate that fine-grained classification is high- frequency dominant task. As existing public datasets for fine- tineggrined classification do not meet our specific experimental requirements, we begin by constructing custom dataset through the fol- lowing procedure: We select the classic FGVC Aireraft dataset (30 classes) and then utilize the DepthAnything v2 model to generate a corresponding depth map for each RGB image, creating paired dataset.
For our experimental setup, we employ a vanilla ResNet18 as the backbone classifier. We evaluate performance based on classification Accuracy and the PCR across various modality combinations. A total of 14 experimental sets were conducted. In seven sets, the dataset was processed using low-pass filters, while in the remaining seven, high-pas, hass filters with identical kernel sizes applied. We investigate two ranges kernel sizes: small kernels (3 x3, 5x5, 7x7, and 9) and large kernels (17x x 17, 31 311 and 51 51x5)).
27

---

Published as a conference paper at ICLR 2026
Table 16: Performance comparison of multimodal robust solutions SURF dataset.
This table
serves as continuation of T
Modal
SF-MD
HeMIS
RFNet
MMANet
SF-MD'
MMANet
Rgb Depth Ir
Ac PCR())
Acc
PCR
Acc
PCR
Ace
PCR
Acc
PCR
Acc
PCR
X
83.92
13.83
85.64
12.64
87.57
11.38
91.43 7.77
92.13
6.87 93.49 5.32
X
V
x
93.65
3,84
95.30
2.78
95.83
3.03
97.73
1.41
96.55
2.41
98.31
0.44
v
89.60
8.00
83.79
14.53
85.31
13.67
89.96
9.25
93.76
5.23
94.18
4.62
V
x
95.43
2.01
96.77
1.29
97.77
1.06
98.39
0.75
98.25
0.69
98.64
0.10
V
93.26
4.24
93.73
4.39
95.73
3.13
96.99
2.16
96.62
2.33
96.60
2.17
X
96.74
0.67
96.32
1.74
96.78
2.06
0.31 98.82
98.63
0.30
99.27
-0.54
97.39
98.03
98.82
7
99.13
/
98.93
/ 98.74
/
Average
92.85
5.43
92.82
6.23
93.98
5.72
96.06
3.61
96.41
2.97 97.03
2.02
Table 17: Comparison with other methods for handling missing modalities. To ensure a fair com- parison, only report results the SURF dataset using the most dissimilar modalities: RGB and Depth.
modal
CRMT-JT
COM
DCP
MWAM
Rgb
Depth
Acc(t)
PCR(.)
Acc
PCR
Acc
PCR
Acc
PCR
x
93.08
5.42
92.79
5.28
93.01
5.74
93.99
4.96
V
96.74
1.70
96.82
1.16
98.15
0.53
97.68
1.22
98.41
97.96
98.67
/
98.89
Average
96.08
3.56
95.86
3.22
96.61
3.13
96.85
3.09
Visualizations of the filtered data are provided in Figure X The comparative experimental results are summarized in Tables 21 and the performance curves are plotted in Figure
The experimental results reveal several intriguing phenomena. First, for the low-pass filter group, performance exhibits a rise-then-fall pattern as the kernel size increases. We posit that with small kernels, the tow-pass filter effectively removes imperceptible high-frequese sian noise. Given the 518x518 image resolution, these small kernels are insufficient to smooth out task-relevant high-frequency details like edges. This noise removal enhances the robustness of the model's 's predictions, leading to slight performance gain. Subsequently, as the kernel size grows, the filter begins erode these critical high-fre gigh-frequency features. This detrimental effect becomes more pronounced with larger kernels, causing a sharp degradation in performance.
artifacts, such Gaus-
In stark contrast, the control group using high-pass filters demonstrates much more gradual decline in performance, even as the kernel size increases. Notably, it avoids the catastrophic collapse in classification accuracy observed in the low-pass scenario with large kernels.
From these observations, we can deduce that high-frequency components play a dominant role in fine-grained classification, far more than in general classification tasks. Furthermore, these critical signals are not only highly sensitive but are also intricately intertwined with complex noise, which constitutes a core challenge in fine- ane-grained recognition.
A.16.2 COMPARATIVE EXPERIMENTS ON FINE-GRAINED CLASSIFICATION
To verify the versatility of our approach, we conducted a fine-grained classification task. Specif- ically, we selected the Stanford Dogs dataset (120 classes) and the FGVC Aircraft dataset (30 classes), generating corresponding depth representations using DepthAnything v2. We chose the basic ResNet18 structure the bascline to verify the classification accuracy before and after adding MWAM. The experimental results are presented in Table 22
The results show that MWAM consistently boosts ResNet18's 8's performance in fine fine-grained classi- fication tasks. This indicates our method works well not only for low-frequency tasks but also for high-frequency ones.
Furthermore, Figure 10 compares the overall performance of the Baseline model against our Base- line+MWAM under various filtering conditions. The results highlight four key phenomena:
28

---

Published as a conference paper at ICLR 2026
Lavpos filtering
W
Elighpsss filttrigg
SPAP
Original Image
Kernel Size 3x3
Kernel Size 5x5
Kernel Size 7x7
Kernel Size 9x9
Kernel Size 17x17
Kernel Size 31x31
Kernel Size 51x51
Figure 8: Visualization of filtering effects on dataset samples. This figure showcases the comparative results of applying low-pass and high-pass filters with various kernel sizes to paired visible-light and depth images.
Table 18: Performance comparison of low-pass filters with varying small kernel sizes.
Modal
w/o filter
3x3
5x5
7x7
9x9
RGB Depth
Acc(t)
PCR(.)
Acc
PCR
Acc
PCR
Acc
PCR
Acc
PCR
X
61.60
9.57
66.95
6.53
67.22
7.03
62.77
11.03
56.73
15.98
X
39.81
41.56
39.54
44.80
40.02
44.65
39.78
43.61
38.58
42.86
V
68.12
1
71.63
/
72.30
/
70.55
67.52
Average
56.51
25.57
59.37
25.67
59.85
25.84
57.70
27.32
54.28
29.42
Augmenting the Baseline with MWAM yields a substantial and con- sistent performance improvement across nearly all filter settings (with the exception of the 51x51 low-pass filter). This primarily demonstrates the effectiveness of our proposed module.
Effectiveness of MWAM:
Both the Baseline and Baseline+MWAM exhibit similar per- formance trends under low-pass and high-pass filtering. This suggests the presence of subtle, high- frequency noise in the dataset that, while imperceptible to humans, adversely affects the model's training process.
Impact of High-Freeqency Noise:
Attention to High-Frequenee Information: When using large-kernel low-pass filters, the perfor- mance of Baseline+MMWAn MWAM significantly lower than its counterpart under high-pass filtering with the same kemel size. This indicates that for tasks dominated by high-frequency features, MWAM effectively utilizes the high-frequency information in the images.
Task Degradation under Extreme Smoothing: When the kernel size reaches 51x51, the excessive smoothing from the low-pass filter results comparable performance between the two models (with Baseline+MWAM even slightly underperforming). This implies that under such conditions, the model may degrade the fine-grained task into simpler one, such as color recognition, which relies heavily low-frequency components.
In summary, these findings demonstrate the versatility of MWAM: it not only excels at conventional www.frequency--ommiie image understanding tasks but is also highly effective for tasks that depend on high-frequency information.
29

---

Published as a conference paper at ICLR 2026
Table 19: Performance comparison of low-pass filters with varying large kernel sizes.
Modal
w/o filter
17x17
31x31
51x51
RGB
Depth
Acc(f)
PCR())
Acc
PCR
Acc
PCR
Acc
PCR
61.60
9.57
31.70
28.76
18.33
19.11
14.75
4.10
X
39.81
41.56
31.55
29.10
21.72
4.15
16.92
-00.01
/
68.12
44.50
22.66
15.38
Average
56.51
25.57
35.92
28.93
20.90
11.63
15.68
-2.96
Table 20: Performance comparison of high- pass filters with varying small kernel sizes.
Modal
w/o filter
3x3
5x5
7x7
9x9
RGB
Depth
Acc(t)
PCR(L)
Acc
PCR
Acc
PCR
Acc
PCR
Acc
PCR
61.60
9.57
56,49
12.35
54.51
13.32
52.22
14.10
50.21
14.57
39.81
41.56
39.69
38.42
39.66
36,94
39.60
34.86
39.42
32.92
68.12
64.45
/
62.89
60.79
58.77
Average
56.51
25.57
53.54
25.38
52.35
25.13
50.87
24.48
49.47
23.75
Table 21: Performance comparison of high-pass filters with varying large kernel sizes.
Modal
w/o filter
17x17
31x31
51x51
RGB
Depth
Acc(t)
PCR())
Acc
PCR
Acc
PCR
Acc
PCR
X
61.60
9.57
46.51
16.11
44.62
13.76
43.75
9.40
39.81
41.56
37.98
31.49 /
33.23
35.78
29.48
38.95
V
68.12
55.44
51.74
/
48.29
/
Average
56.51
25.57
46.64
23.80
43.20
24.77
40.51
24.18
3
L2v
51
Ste
Full Movialiles
Resulss
Figure 9: Effect of filter type and kernel size model performance.
Table 22: Validation the effectiveness of MWAM on fine-grained classification tasks. We selected the Stanford Dogs dataset and the FGVC Aireraft dataset, and used the DepthAnything v2 model to generate corresponding depth representations. The classification performance before and after deploying our MWAM is tested using the vanilla ResNet18 structure the base model.
Stanford Dogs Dataset
FGVC-Aireraft Dataset
Modal
w/o MWAM
w/ MWAM
w/o MWAM
w/ MWAM
Rgb
Depth
Acc(t)
PCR())
Acc.
PCR
Acc
PCR
Acc
PCR
X
32,91
38.73
43,91
28.84
61.60
9.57
74.85
6.21
/
27.90
48.05
36.20
41.34
39.81
41.56
56.22
29.56
53.71
61.71
68.12
79.81
Average
38.17
43.39
47.27
35.09
56.51
25.57
70.29
17.89
30

---

Published as a conference paper at ICLR 2026
90
80
80
70
Accuracy
60
70
60
40
30
20
Bascline
10
51
17
31
Filter Kemel Size
Filter Kernel Size
(a) Low-pass filtering
(b) High-pass filtering
Figure Performance comparison of methods under different filters.
31


Reviews:
[{"review_id": "RAdqjvX9no", "reviewer": "Reviewer_Lu1c", "summary": "This paper tackles the problem of modality bias and missing modality robustness in multimodal image understanding models. The authors observe that current multimodal systems tend to overfit or over-rely on certain “dominant” modalities (e.g., depth or RGB), leading to severe performance drops when one modality is missing at inference time. To address this, the paper introduces Frequency Ratio Metric and Multimodal Weight Allocation Module. Extensive experiments on segmentation (BRATS2020, NYU-Depth V2) and classification (CASIA-SURF) benchmarks demonstrate that MWAM consistently improves robustness and performance across CNN- and ViT-based architectures with minimal computational overhead.", "questions": "- How sensitive is the FRM metric to the choice of frequency cutoff (p × p patch size and q × q block size)? Could adaptive frequency selection further improve the metric?\n- How does MWAM compare to feature-level reweighting techniques like modality dropout, dynamic fusion gates, or attention-based balancing in terms of convergence stability?\n- Would FRM or MWAM also help in text–image or video–audio multimodal models, or is the current formulation specific to visual modalities only?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- he idea of diagnosing and correcting multimodal imbalance in the frequency domain is both intuitive and underexplored. The FRM formulation provides a new angle that complements existing spatial-domain balancing techniques.\n- MWAM is architecture-agnostic, parameter-light, and easy to integrate into existing backbones. This makes it particularly attractive for practitioners working on robustness in multimodal models.\n- The authors validate their method on diverse datasets and tasks (segmentation and classification) and show consistent improvements in both accuracy (Dice, MIoU, Acc) and robustness (PCR).\n- The ablation studies (gradient vs. loss vs. hybrid intervention) clearly show why hybrid balancing works best. The training loss visualizations (Fig. 4) effectively illustrate MWAM’s stabilizing effect.", "weaknesses": "- While the frequency-domain motivation is intuitive, the paper lacks a rigorous theoretical connection between FRM and gradient dynamics. A more formal justification for why FRM effectively measures modality dominance would strengthen the contribution.\n- All experiments are on moderate-sized datasets. It remains unclear whether FRM and MWAM scale effectively to modern multimodal foundation models.\n- Some figures and equations are dense and could be better formatted for readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JoFik73g9v", "reviewer": "Reviewer_gN31", "summary": "This paper addresses a common problem in multimodal learning: performance drops significantly when one of the data modalities is missing.\nThe authors argue this happens because the model learns to rely too much on one \"dominant\" modality during training, while neglecting the others. Their key idea is that this dominance can be measured in the frequency domain of the images.\nTo solve this, they propose:\nA Frequency Ratio Metric (FRM): A simple metric to quantify how much the model prefers each modality by comparing its low-frequency and high-frequency content.\nA Multimodal Weight Allocation Module (MWAM): A lightweight, plug-and-play module that uses the FRM to rebalance the training process.", "questions": "Consider this paper does not utilize LLMs to perform understanding tasks, so it more like image perception work.\n\n\"SF-MD, we additionally introduce auxiliary heads, whereas we do not in MMANet.\" Why introduce extra heads for SFMD only?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper analyzes the modality imbalance problem from the frequency domain and introduces an FRM to quantify how much the model prefers each modality. This seems to be promising. \n\nBased on FRM, this paper introduces MWAM to modulate the training process of multimodal models so that the performance can be balanced and enhanced.\n\nThe writting is easy to understand.", "weaknesses": "\"The overall training process becomes more stable, as evidenced by the reduced variance in the total loss curve\". But in Figure. 4 SF-MD (w/o Intervention)'s total loss is more stable than that of SF-MD (w / MD (w / Loss Intervention ntervention).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kv1rWrOemb", "reviewer": "Reviewer_bfT5", "summary": "This paper tackles robustness in multimodal image understanding, focusing on cases where some modalities are missing at inference. The authors show that models often become biased toward dominant modalities, causing severe performance drops when those are absent. To address this, they propose Frequency Ratio Metric (FRM) and Multimodal Weight Allocation Module (MWAM). Experiments across multiple tasks and architectures demonstrate that MWAM improves robustness and outperforms several state-of-the-art methods.", "questions": "1) Can MWAM and FRM be adapted for multimodal tasks involving non-image data (e.g., audio, text, tabular)? What are the challenges and potential solutions?\n2) How does MWAM handle cases where modalities are highly correlated or redundant? Is there a risk of over-balancing and degrading performance?\n3) Are there specific scenarios or datasets where MWAM fails to improve robustness or even degrades performance? If so, what are the underlying causes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) Using frequency domain analysis (FRM) to diagnose and mitigate modality bias sounds interesting, which goes beyond spatial domain balancing.\n2) The proposed MWAM is a lightweight and plug-and-play module with negligible computational overhead and no additional parameters during inference. This makes it attractive for real-world deployment.\n3) The method is validated on multiple tasks (classification, segmentation, detection) and datasets, showing improvements over baseline  methods.  Extensive ablation studies on FRM design, window size, parameter sensitivity, and training setting are ablated.", "weaknesses": "1. The approach is tailored to image-based modalities and frequency domain analysis. It is unclear how well MWAM would generalize to other multimodal settings (e.g., audio-text, video-language) where frequency analysis may not be directly applicable\n2. MWAM introduces several hyperparameters that require careful tuning for different tasks and batch sizes. While sensitivity analysis is provided and the authors claim the performance is not that sensistive to such hyper-parameters, considering the performance gain over baseline method is not that significant, such performance perturbations is not neglibile. However, there is no clear guidance about how to set such hyper-parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yD8RWXNhnu", "reviewer": "Reviewer_K6MD", "summary": "This paper presents a novel approach to multimodal learning that applies the Discrete Cosine Transform (DCT). \nThe purpose of using DCT is to extract the low- and high-frequency components of each modality in the frequency domain, thereby capturing the intrinsic characteristics of each modality. \nThis approach is employed in two key modules:\n\n- **Frequency Ratio Metric (FRM)**: Measures the dominance of each modality.\n- **Multimodal Weight Allocation Module (MWAM)**: Dynamically allocates modality weights within each batch to determine the relative contribution of each modality.\n\nThis design leads to improved performance in several experimental settings.", "questions": "- **Question 1**: What is the motivation for applying the Discrete Cosine Transform (DCT) instead of other frequency-domain transforms such as the Discrete Fourier Transform (DFT) or Fast Fourier Transform (FFT)? In vision tasks, DFT/FFT captures both magnitude and phase information, whereas DCT focuses on real-valued frequency components and is often used for compression. It would be helpful to understand why DCT was specifically chosen in this context and how it benefits the representation of visual modalities.\n- **Question 2**: What are the main differences between this paper and other frequency-domain deep learning approaches? Previous studies have already applied Fourier transforms either between layers [1] or at the input space [2]. From the paper, the primary difference seems to be the use of DCT instead of FFT. Since both operate in the frequency domain, this substitution alone does not convincingly demonstrate methodological novelty. \n- **Question 3**: What is the rationale for using the reflected sigmoid function with multiple hyperparameters in the Eq. (4)? This design choice seems critical, as it raises concerns that the observed performance improvements might depend heavily on hyperparameter tuning rather than the proposed MWAM mechanism itself. I could not find a clear explanation or reference supporting this design choice, and those 4 hyperparameters.\n- **Question 4**: Have the authors considered evaluating MWAM on datasets that include additional modalities such as audio or text? This would help demonstrate the generalizability of the proposed approach beyond purely visual modalities.\n\n[1] Guo, Shi, et al. \"Spatial-frequency attention for image denoising.\" (preprint)\n\n[2] He, Xilin, et al. \"Towards combating frequency simplicity-biased learning for domain generalization.\" (NeurIPS 2024)", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Strength 1**: Preprocessing input data in the frequency domain can enhance efficiency in capturing the power and meaningful information of each modality within the input space.\n- **Strength 2**: Since this approach focuses on extracting informative representations from the input space, it can be broadly and effectively applied to various vision tasks, such as segmentation, classification, and object detection.", "weaknesses": "- **Weakness 1**: Although this paper focuses on multimodal learning, its proposed approach is primarily applicable to vision-based tasks. It might seems difficult to extend the method to other modalities, such as text, where applying DCT directly is not feasible. Additional processing steps would be required to make DCT applicable to such modalities.\n- **Weakness 2**: The method involves a large number of hyperparameters that can significantly influence the results. For example, variations in $\\alpha$, $\\beta$, $\\lambda$, $\\gamma$, and $\\sigma$ in Eq.(4) and (5) may lead to substantially different outcomes. This complexity makes it challenging to determine which hyperparameters are critical or not. Although the authors provide ablation studies in Table 9, a clearer justification of the importance of each hyperparameter would strengthen the work. Otherwise, it would be better to change the reflected sigmoid function used in the weight allocation process.\n- **Weakness 3**: The performance analysis lacks clarity in certain aspects. For instance, in Table 2, results for the vanilla GSS model are missing, even though GSS + MWAM achieved the best overall performance. The authors compare this with LS3M; however, since GSS and LS3M are fundamentally different approaches, this comparison may not be entirely fair. It would be more appropriate to compare Original vs. Original + MWAM to more fairly and effectively highlight the contribution and novelty of MWAM. Same comments for other Tables.\n- **Weakness 4**: The overall structure of the paper and writings could be improved for better coherence. For instance, in Section 3.1, the authors present empirical analyses on the influence of different frequency components, while all theoretical discussions are deferred to the appendix. Given that one of the paper’s novelty is the theoretical and empirical validation of modality dominance, I think it is required to include at least a concise summary or key results of the theoretical validations within the main text, even if the detailed derivations remain in the appendix.\nAdditionally, I would recommend that important equations and their explanations be formatted like Definitions, Lemmas, and Theorems. (e.g., Appendix A.1)\n\n\nMinor Issues:\n\nWords change at Abs \\& Conclusion: **efficiency** method $\\Rightarrow$ **efficient** method\n\nTypo at line 296: show the sensitivity **o** these factors  $\\Rightarrow$ show the sensitivity **of** these factors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)