MASAM: Multimodal Adaptive Sharpness-Aware Minimization for Heterogeneous Data Fusion
Authors: Zijie Chen
Kejing Yin
Wenfang Yao
William K. Cheung
Jing Qin
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal learning
Modality imbalance
Generalization.
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A method designed to efficiently fuse highly heterogeneous data

Abstract: Multimodal learning requires integrating heterogeneous modalities, such as structured records, visual imagery, and temporal signals. It has been revealed that this heterogeneity causes modality encoders to converge at different rates, making the multimodal learning imbalanced. We empirically observe that such an imbalance is related to the sharpness of the solution. Modality encoders that converge faster could be dragged into sharp regions due to inter-modal interference, degrading the generalization capability of unimodal features learned. Sharpness-Aware Minimization is effective in improving generalization via finding solutions in flat regions. However, its application in multimodal scenarios is challenging: 1) SAM overemphasizes the dominant modality, inducing misaligned perturbations in weaker modalities, and 2) the perturbation gradient calculation is affected by interference from other modalities. To address these issues, we propose Multimodal Adaptive Sharpness-Aware Minimization (MASAM), which optimizes different modalities based on their dominance. We design an Adaptive Perturbation Score (APS) using convergence speed and gradient alignment to identify dominant modalities for SAM application. Our Modality-Decoupled Perturbation Scaling (MDPS) then reduces inter-modal interference during optimization, better aligning each modality with shared information. Extensive empirical evaluations on five multimodal datasets and six downstream tasks demonstrate that MASAM consistently attains flatter solutions, achieves balanced multimodal learning, and subsequently surpasses state-of-the-art methods across diverse datasets and tasks. Code is available at https://github.com/Orange2107/MASAM-Multimodal-Adaptive-SAM.

Published as a conference paper at ICLR 2026
MASAM: MULTIMODAL ADAPTIVE SHARPNESS- AWARE MINIMIZATION FOR HETEROGENEOUS DATA FUSION
Zijie Chen Kejing Yin',, Wenfang Yao², William K. Cheung', Jing Qin'
I Department of Computer Science, Hong Kong Baptist University
2 Institute of Systems Medicine and Health Sciences, Hong Kong Baptist University
3 Division of Artificial Intelligence, Lingnan University
4 School of Nursing, The Hong Kong Polytechnic University
ABSTRACT
Multimodal learning requires integrating heterogeneous modalities, such as struc- tured records, visual imagery, and temporal signals. It has been revealed that this heterogeneity causes modality encoders to converge at different rates, making the multimodal learning imbalanced. We empirically observe that such imbalance is related to the sharpness of the solution. Modality encoders that converge faster could be dragged into sharp regions due to inter-modal interference, degrading the generalization capability of unimodal features learned. Sharpness- Aware Minimiza- tion is effective improving generalization via finding solutions in flat regions. However, its application in multimodal scenarios is challenging: 1) SAM overem- phasizes the dominant modality, inducing misaligned perturbations in weaker modalities, and 2) the perturbation gradient calculation is affected by interference from other modalities. To address these issues, we propose Multimodal Adaptive Sharpness-Awain Aware Minimization (MASAM), which optimizes different modalities based on their dominance. We design an Adaptive Perturbation Score (APS) using convergence speed and gradient alignment to identify dominant modalities for SAM application. Our Modality-Decoupp Perturbation Scaling (MDPS) then reduces inter-modal interference during optimization, better aligning each modality with shared information. Extensive empirical evaluations on five multimodal datasets and six downstream tasks demonstrate that MASAM consistently attains flatter solutions, achieves balanced multimodal learning, and subsequently surpasses state- of-the ff-the-art art methods across diverse datasets and tasks. Code is available at https: / //githuis com/ http///aanngeel /MA/AAMM Multimodai timodal Adapt i vee SAM.
INTRODUCTION
Multimodal learning leverages complementary information from heterogeneous data sources and has achieved remarkable progress in domains such as audio-video understanding (Huang al., 2023) and clinical decision support (Stahlschmidt et al., 2022). Despite these successes, prior studies have highlighted the persistent issue of modality imbalance, where conver-gn modalities dominate training and weaker modalities are under-optimized, often leading to suboptimal fusion or even inferior performance compared to unimodal models (Wang al., 2020; Peng et al., 2022; Wei & Hu, 2024). A range of strategies, gradient modulation (Li et al., 2023), iterative unimodal optimization (Zhang et al., 2024), and representation regularization (Fan et al., 2023), have been developed to alleviate this problem, but these approaches primarily modulate gradient magnitudes and overlook deeper geometric factors, such as the sharpness the loss landscape and the solutions.
Such geometric factors have close linkage to the generalization capability and also modality balance in the multimodal settings: flatter minima typically correspond to more robust and stable solutions (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Foret et al., 2021). In multimodal settings, heterogeneous modalities exhibit inherently divergent convergence dynamics. Eyen when a
"Correspondence author: Kejing Yin wwwe@@eooompmhhccceocc

---

Published as a conference paper at ICLR 2026
8m Cluse O + tm
Normalized Hessian Trace
1.2
CXR Trace (Late Fusion)
1.0
CXR Trace (Ours)
Von Lm (mm)
0.8
0.6
0.4
0.2
Start Ours Late Fusion
V Ven Cueed (
Perturbation
Epoch
(a) Hessian trace of CXR encoder (MIMIC)
(h) Optimization trajectory comparison
Figure 1: (a) The Hessian trace analysis for CXR modality in the MIMIC dataset, where a lower normalized Hessian trace represents flatter minima. It suggests that the CXR encoder in the naive late fusion model converges to a sharp region, whereas that in our MASAM converges to much flatter region. (b) An illustration of the optimization trajectory of naive late fusion and MASAM.
modality progresses towards a flatter solution, joint optimization can continually disrupt its trajectory through cross-modal interference, preventing the model from reaching jointly flat optimum and thereby making multimodal learning imbalanced and limiting its generalization capacity. We em- pirically observe this phenomenon in real-world datasets. Fig. 1a plots the Hessian trace (a proxy for sharpness) of modallty-sppecifi encoders for the MIMIC dataset (EHR + CXR), where the CXR encoder initially converges towards flatter region (smaller Hessian Trace) but later becomes sharper as the unstable optimization of the EHR encoder disrupts its trajectory. Fig. lb provides a concep- tual illustration, showing that cross modal interference under Late Fusion diverts the optimization trajectory into sharper regions.
Sharpness- Aware Minimization (SAM) has demonstrated consistent improvements in gen- eralization for manglemmodalit deep learning by
MASAM
AGM
MMPareto
PMR
OGM
MLA
G-Blend
Late Fusion
steering optimization towards flat regions of the
CREMA-D
loss landscape (Foret al., 2021). This benefit stems from the fact that flat minima are associ- ated with robustness parameter perturbations and data variability (Hochreiter & Schmidhu- ber, 1997; Keskar al., 2017), making SAM widely adopted technique for enhancing model reliability diverse domains. We conjecture that similar ideas could be used to mitigate the modality imbalance issue in multimodal learn- ing. However, directly applying SAM to mul- timodal settings introduces fundamental chal- lenges that undermine its effectiveness: (1) Am- plified modality imbalance: Different modal- ities exhibit heterogeneous convergence speeds and gradient magnitudes. SAM's uniform sharp- ness minimization thus disproportionately bene- fits dominant modalities with faster convergence, while under-optimizing slower yet informative ones (Fan et al., 2024). (2) -odality-agnostic perturbation: Beyond imbalance in magnitude, modalities also possess distinct loss landscape geometries. The perturbation applied by SAM is agnostic to differences in loss landscape geometries, forcing encoders of different modalities to adapt to a shared trajectory rather than their own flat regions. This one-size sizt--its--llll approach leads to undermined multimodal generalization.
0.78
0.74
UPMC Egod101
ADNI
0.68
0
0.92
0.91
0.58
0.58
D.44
0.68
0.47
MIMIC-IV MIMIC-IPD.
Kinetics
(Moratlity
0.48
0.49
MIMIC-IV (Phenotype)
Figure 2: The proposed MASAM model achieves the best performance across all datasets and tasks.
2

---

Published as a conference paper at ICLR 2026
To address these challenges, we propose Multimodal Adaptive SAM (MASAM), optimization framework tailored for multimodal learning. To mitigate amplified modality imbalance, MASAM introduces an Adaptive Perturbation Score (APS) to quantify the dominance of each modality through its convergence speed and gradient alignment with the fusion objective. Perturbations are then selectively applied dominant modalities, stabilizing their convergence in flatter regions while preventing weaker modalities from being affected by misaligned perturbations from the dominant modality. To address modality-agnostic perturbation, MASAM develops Perturbation Scaling (MDPS) that adjusts perturbation strength according to cross-modal gradient aligmment. This enables each modality to explore flatter minima along directions that align with shared information while remaining robust to heterogeneous loss landscapes. Together, these mechanisms make SAM modality-aware allowing multimodal models to retain the benefits of flatness-driven generalization without suffering from imbalance or incompatible perturbations. Fig. 2 summarizes the quantitative evaluation on six diverse multimodal datasets, demonstrating the effectiveness of MASAM.
Mlodality-Decouppe
Our main contributions are summarized as follows:
We identify two fundamental limitations: amplified modality imbalance and modasity perturbation, which hinder the direct application of SAM to multimodal learning.
We propose MASAM, multimodal optimization framework that explicitly tackles modality imbalance and heterogeneous loss geometries through Adaptive Perturbation Score (APS) and Modality-Decoupled Perturbation Scaling (MDPS).
We conduct extensive experiments across diverse domains, from widely used multimodal bench- marks (CREMA-D, Sinetics-Soundee UPMC-Food101) to noisy clinical datasets (MIMIC-IV, ADNI), and extend MASAM to tri-modal learning UR-FUNNY. Results show that MASAM significantly outperforms existing basclines, consistently addresses modality imbalance issues by achieving flatter optima, and obtains better robustness against noisy labels.
PRELIMINARIES AND RELATED WORK
Preliminaries on Sharpness-Awe Aware Minimization Sharpness-Aware Minimization (SAM) (Foret et al., 2021) seeks parameters in flat regions in the loss landscape. Formally, SAM minimizes both the loss value and the sharpness of the loss surface, where sharpness is defined as the maximum increase in loss under small perturbation within l2-ball of radius p of the parameters A:
max L(O + - [(0) + C()).
min
By applying first-order Taylor expansion, the inner maximization reduces to perturbing parameters in the gradient direction:
VL(0) VL(O)]I
min L(O + E), where é -
Intuitively, this procedure penalizes sharp directions in the loss surface, steering training towards flatter minima that enhance generalization and robustness (Andriushchenko et al., 2023). Several variants of SAM refine the perturbation sharpness estimation (Wang al., 2023; Wu et al., 2024a; Li et al., 2024), but all are developed for unimodal tasks, leaving their applicability to multimodal optimization unexplored.
Related Work on Balancing Multimodal Learning Recent efforts have been made to mitigate the modality competition issue, where the learning is dominated by stronger modalities. Existing works mostly rely on gradient modulation, such as G-Blend (Wang et al., 2020), OGM (Peng et al., 2022), and AGM (Li et al., 2023). They tackle the modality imbalance by dynamically rescaling gradients, but do not take the geometry of the loss surface into consideration, potentially leading to poor generalization. For more discussions of related work, see Appendix
Why SAM Fails in Multimodal Learning? While SAM and its variants have demonstrated consistent improvements in unimodal learning, directly applying SAM to multimodal learning is problematic since the gradients of the fusion objective are coupled across modalities. When
3

---

Published as a conference paper at ICLR 2026
Modality Decoupled Perturbation Scaling
Modulity Decoupled Parturbation Scaling
Decoupled
Paramelere
Perutation
Lipans
Dractsetaanl
Potateien SSAMS
Updiaa
Zre
Brac
Vi. Erere
Pairnd Data
fuse
('ars
(SAM)
fm
g
mm
Unimodal Loss Landscape
Destvard
Forvedd
Modality Decoupled Perturbation Scaling
Colect Modial
Figure 3: Overview MASAM. We propose to adaptively apply perturbations to specific modalities, guiding each modality towards its own flat minimum to address modality imbalance. First, introduce the Adaptive Perturbation Score (APS) to identify strong modalities training. Then, Medaalty-Decoupled Perturbation Scaling (MDPS) aligns gradients between unimodal and fusion objectives, mitigating cross-modal interference under SAM perturbations.
one modality dominates the prediction, weaker modalities receive distorted gradients, and SAM perturbations derived from them fail to ensure modality-specifie flatness. We formally summarize this observation as follows.
Observation (Difficulties of applying SAM multimodal learning). Consider multimodal learning with at least two modalities, trained under joint late fusion loss Lruse. If one modality is dominant (i.e., contributes disproportionutely to the fused prediction), then:
The gradient Ve.. Cruse for each weaker modality m is biased towards the dominant modality, rather than aligned with its own unimodal objective.
The SAM perturbation direction for modality m
Vo... Lruse
Cyrl P I Ltusell2
is therefore also biased, preventing weaker modalities from converging to their flat minima.
As a result, applying SAM directly to multimodal training can exacerbate modality imbalance and fail to achieve modality-specifie flatness.
We provide a detailed derivation in Appendix This observation shows that SAM's perturbation mechanism, while effective in unimodal training, becomes counterproductive in multimodal settings, motivating the moddllty-adaptivn variant we propose in this paper.
MASAM: MULTIMODAL ADAPTIVE SHARPNESS- SWARE MINIMIZATION
3.1 THE MASAM FRAMEWORK
Prior analysis demonstrates that multimodal training, the dominant modality often steers the joint optimization process, resulting in insufficient exploration and suboptimal learning for weaker modalities. Moreover, the perturbation direction in SAM tends to align with the stronger modality's gradient, further impeding the weaker modality's 's ability converge to flatter minima. To mitigate these challenges, we propose a method that dynamically selects which modality to perturb and enforces gradient alignment across modalities. The overall framework is illustrated in Fig. 3.
Multimodal fusion Without loss of generality, we assume two modalities m and m2g For each data sample indexed by we consider a paired multimodal instance with the downstream task label denoted as (I:" _x ma yt). The data modalities x: and In are encoded using modality-specinf encoders, denoted Φin and Φm, and parameterized by 0m, and O mmg, respectively. Then, they are combined for downstream tasks.
Luse pluse (2,8
2, fuse
Fusion (2 22 m2, O W
12
m2
b.
4

---

Published as a conference paper at ICLR 2026
2: Omg ( mm)), z Pmz (x." mmz),
(5)
where || denotes concatenation, ffuse is the task-specific classification head, and gluse the final prediction. We adopt unified cross-entropy objective, denoted as CE. The fusion loss Ctuse and the auxiliary unimodal objective C. of modality m are defined as:
N Σ F=1
Liuse
CE
fuss),
Cm = N Σ CE(y,,8"), m E {mi,
(6)
where y = (z." is the prediction from modality m using its unimodal feature representation.
The final training objective combines the fusion loss and unimodal auxiliary losses:
Lotal
Luse
+
Am f.ns +
Amzmmas
where Am, and Xm, are hyperparameters that control the strengths of the unimodal objectives.
Adaptive perturbation score (APS) As noted prior works (Wang et al., 2020; Li al., 2023), different modalities exhibit distinct convergence behaviors and make uneven contributions to the joint optimization process during training. A modality with a faster decrease the uni-modal loss C.n. typically indicates that its information is being efficiently and continuously learned, thus suggesting stronger contribution. To capture this phenomenon, we introduce a measure for modality learning speed based moving averages:
Decay!! (1) = max (0, (/, MA E {mi, m23 m
(8)
where MA 3- 41 9) Lm is the moving average of the loss values during training, I denotes the number of training steps. MA provides stable estimate of the long-term loss trend, while Decay reflects the short-term reduction relative this trend. A higher decay value indicates faster learning progress, implying that the modality is contributing more actively at the current step.
In addition to learning speed, we also consider gradient alignment as an indicator of modality strength. Our earlier analysis suggests that strong modalities, which convey richer shared information, tend to dominate the optimization trajectory the fusion objective. This manifests as a high alignment between the gradients of the uni-modal loss and that of the fusion loss, since the latter is largely stecred by the stronger modality. Conversely, weaker modalitics may have misaligned or even conflicting gradient directions with respect to the fusion gradient, resulting in limited influence during joint learning. This perspective also observed in Fan et al. (2023) and Guo et al. (2024). We therefore define gradient-basee based modality dominance metric
(Vom. Cluse, Vo.. Lm) 112 Lm.
(8) Tma
E (mi,
where Ven.om and Vo., Linse denote the gradients from the unimodal objective and the fusion objective, respectively. This unimodal loss not only promotes the acquisition of modality-specific information but also serves as proxy to assess the modality's standalone learning dynamics. By combining both perspectives, we define our Adaptive Perturbation Score (APS) as:
APS(! a Decay!!) + a) 1210 m (m11,
(10)
where a E (0, 1] is a tunable hyperparameter that balances learning speed and gradient consistency in determining the dominance of each modality. To mitigate the risk of strong modalities being pushed away from flatter minima due to the continued updates driven by weaker modalities, we first identify the strong modality as the one with higher APS in each training step. Then, we impose modality-speciff 3AM-basee regularization over the strong modality.
Modallty--dcouplec perturbation scaling (MDPS) joiot learning, the final prediction is derived from the output of the fusion module, which primarily captures modality-shared information that is directly relevant to the downstream task. Consequently, the gradient of the fusion objective, Ve.. Ltuse- can be interpreted as the direction that guides the learning of shared representations. Motivated by this, we apply perturbations along the fusion gradient direction to the dominant modality identified by APS. However, as discussed in Observation and Appendix C, the gradient Vo., Lruse has coupled
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: MASAM: Multimodal Adaptive Sharpness-Aware Minimization
Input: Training dataset D; Perturbation radius p. weights {Am]; momentum μı; score weight a.
Output: Modality encoder parameters 10 and other model parameters 8uther
for each iteration with mini-batch
do
Compute Ctuse and (5)) (5))m=1;
Update getse by gradient descent;
for each modality m E {1,..., M3 do
Compute modality-speeitff APS score APSm. using Eqs. (8) to (10);
Select dominant modality by m arg max APSm:
Obtain the perturbed parameters 0.. + 6m using Eq. (11);
Update dominant modality encoder parameters using Eq. (12); for each non dominant modality m E (l.... M) {m" do
Update modality encoder parameter 0.,. using Eq. (13);
effect from different modalities; therefore, directly applying the SAM perturbation may mislead
weaker modalities during optimization. To address this issue, we propose method that dynamically adjusts the perturbation magnitude based on the degree of alignment between unimodal and fusion gradients, yielding
Vo., Cruse
(Vo., Cruse: Ve.. Cm.) Ve Liuse Ve L2 Ciusel2
Vem. Cruse
6m p mm
(11)
VV o.. Liuse ll2
where 6m is the perturbation applied to Om. is scalar that controls the step size of the perturbation (consistent with the definition Eq. (3)), and T/m is the cosine similarity between the unimodal and fusion gradients, defined in Eq. (9). This scaling can also be interpreted projection of the gradient of unimodal loss onto the gradient direction of the fusion loss, thus achieving modality- decoupled perturbation.
Parameter Update We divide all model parameters into three disjoint subsets, where (0.. } and (0m |m a m²} denote the parameters of the dominating and non-dominating modalities, respectively, and gother denotes all other model parameters. In each iteration, we first update gother using a base optimizer, e.g-, SGD Adam. Then, identify the dominant modality using the APS score by m = arg max APSm. MASAM then employs MDPS as defined in Eq. (11) to perform sharpness- aware optimization that eliminates interference from other modalities. The update direction given by the combination of the perturbed fusion gradient and the unimodal loss gradient:
Von Ciuse V Ciusell2
g++1 6"
Th
mm Lluse 8% + Pt 7/n
V
Ve Lm (m
{m*).
(12)
For each non non-dominant modality, MASAM updates the parameters using the gradients of both the fusion and unimodal objectives computed at the current parameters:
0++1 = 0% 7/ Ve... Liuse (0',. + Ve,, Cm (0th) m m
(13)
We summarize the learning algorithm for arbitrary number of modalities in Algorithm 1.
3.2 THEORETICAL ANALYSIS
We amalyze the convergence of MASAM under standard smoothness and step size conditions, building on the incxact gradient descent (IGD) framework of Khanh et al. (2024). Theorem 1 (Convergence of MASAM). For the modality-specitty MASAM update as in Eq. (12): suppose the fusion loss Liuse has -fisschipschit. continuous gradients with Lfuse > 0. Assume the learning rate sequence {m and the perturbation radius sequence {pt} satisfy
2 T-tuse
(i) 00, (i)) TJ. 4
(ii)
Σ Tppe < oc, (iv) lim Simsup/r
(14)
1-220
Then {6th) converges to a stationary point of the multimodal objective Eq. (7).
The complete proof of this result is presented in Appendix D. The conditions in Theorem are practical in implementation, given that the fusion loss cross-entropy loss and the radius sequence {pe} can approximate constant (see Khanh et al., 2024, Remark 3.6).

---

Published as a conference paper at ICLR 2026
Table 1: Unified comparison across datasets, Columns report the primary evaluation metric used for each dataset. Underlined values indicate the strongest baseline, while bold highlights our method. Relative Gain is computed against the strongest baseline in each column. All results are averaged over four runs with different random seeds.
Phenotype AUPRC
Mortality
CREMA-D
KS
UPMC-Food101
ADNI
UR-FUNNY
Method
AUPRC
Accurecy
Accuracy
Accuracy
mAP
Accuracy
Late Fusion
0.475
0,567
0,660
0.636 Jomn
0:907
0.826 JA3K
0.620
G-Blend (Wang al., 2020)
0,480
0,584
0.717
0.670
0.924
0.818
N.A.
OGM (Peng et al., 2022)
0.475 anores
0,577 -tree
0.769 -00e
0.677 -0oen:
0.907 40000
0.812 44,050
0.632 #1
AGM (Li et al., 2023)
0.472 20012
0,572 Le1e2
0.746 Looc
0.688 LowKer
0.908 un000
0.817 200100
0.630
LID
MI.A (Zhang et al., 2024)
0,474 s000s
0,582 -0r0x
0,705 -008
0.681 20012
0.928 s0301
0.836 +9.005
0.624
MMPareto (Wei & Hu, 2024)
0.479 suoker
0,585
:0160
0,717 S00N
0.658 su1e7
0.925 stuees
0.823 59,133
N.A.5
PMR (Fan ct al., 2023)
N.A.T
0,572 :016L
0.713
0,631 2unz
0.920 sHaRG
0.820 54:022
N.A.5
InfoREG (Huang ot al., 2025)
0.481 Dooe
0.581
Lex0x
0.723
JAMES
0.671 20ons
0.925 JoUaes
0.820 200135
0.625 LEDNA
AUG (Jiang et al., 2025)
0.472 woom
0.579
0.770
0.689 LIOKES
0.923 anars
0.847 JADKS
N.A.
DrFuse (Yao et al., 2024)
0.479 20001
0,576
N.A. F
N.A.F
N.A.F
0.836 HU:HI
N.A.
MedFuse (Hayar et al., 2022)
0.470
0,549
corets
N.A. F
N.A.F
N.A.F
0.828 593133
N.A.
MASAM (ours)
0.498
0.603
0.814
0.740
0.935
0.857
0.644
Relative Gain (%)
+3.53
+3.08
+5.71
+7.40
+0.75
+1.18
+1.90
PMR relies on a contrastive learning module, which canmot be applied to the multi label phenotype task.
DrFuse Mediuuse are tailored clinical data; thus, we exclude them datasets and tasks in other domains. These baselines are not designed for the ti--modality setting.
4 EXPERIMENTS AND RESULTS
We evaluate the proposed method on diverse multimodal datasets: two audio-video datasets, CREMA- D (Cao et al., 2014) and Kinetics-Sounds (KS) (Arandjelovic & Zisserman, 2017 (), one image-text dataset, UPMC-Food101 (Wang et al., 2015), two clinical multimodal datasets, MIMIC (Johnson al., 2023; 2019) and ADNI (Jack Jr al., 2008), and a tri-modality dataset UR-FUNNY (Hasan et al., 2019). On the MIMIC dataset, we perform two clinical tasks, the Phenotype Classification and the Mortality Prediction. For comparison, we include several representative methods for addressing modality imbalance, including InfoReg (Huang et al., 2025), AUG (Jiang et al., 2025), OGM (Peng et al., 2022), AGM (Li al., 2023), MMPareto (Wei & Hu, 2024), PMR (Fan et al., 2023), MLA (Zhang et al., 2024), and G-Blend (Wang et al., 2020), as baseline models. For clinical datasets, we additionally include DrFuse (Yao et al., 2024) and MedFuse (Hayat et al., 2022), which tailored for clinical multimodal learning. Due to page limits, we report AUPRC for the two MIMIC tasks, mAP for ADNI, and accuracy for CREMA-D, KS, 1PMC-Food and UR-FUNNY, with additional metrics in Appendix E.3.
We follow pipelines from existing works to preprocess the datasets, and split them into training, validation, and test sets with 7:1:2 ratio. We use identical encoders across all models within each dataset for fair comparisons and determine the hyperparameters via grid search. All experiments are performed using a server with four NVIDIA RTX 4090 GPUs. The details of data preprocessing and hyperparameter settings can be found in Appendices A.3 and A.4.
4.1 OVERAL. PERFORMANCE ACROSS DATASETS
MASAM consistently achieves significant improvements across all datasets. The overall exper- imental results are summarized in Table and Fig. 2. On the KS dataset, MASAM attains 0.74 accuracy, providing a substantial 7.4% improvement over the strongest baseline (AUG). Similar performance gain is also observed on CREMA-D, where MASAM reaches 0.814 accuracy with a 5.71% relative gain. On clinical datasets, despite their high noise levels and missing rates (statistics in Appendix A.4). MASAM achieves reliable improvements over the best baselines: 3.53% for phenotype and 3.08% % for mortality on MIMIC, and 1.18% higher mAP on ADNI. We provide detailed MIMIC performance analysis across individual disease phenotypes in Appendix E. On UPMC-Food 101, MASAM achieves a 0.75% % accuracy gain; although the improvement is modest, we perform a statistical test that confirms the significance of the improvement (p- = 0.0046 < 0.005). On UR-FUNNY, MASAM achieves 0.641 accuracy, yielding 1.42% % improvement over the strongest baseline. Overall, the promising results show that MASAM consistently outperforms existing methods across diverse and highly heterogeneous multimodal datasets.
7

---

Published as a conference paper at ICLR 2026
MASAM
Late Fusion
DGM
AGM
MMPareto
a MLA
PMR
G-Blend
MedFuse
DrFuse
--.. Unimodal Baseline 1
--- Unimodal Baseline
CREMA-D
KS
UPMC-Food101
Video Performance
Video Perforaace
Image Performance
0.64
0.56
0.45
0.64
0.48
0.40
0.60
0.40
0.35
0.56
0.60
0.62
0.64
0.66
0.68
0.525
0.550
0.575
0.600
0.640
0.645
0.650
0.855 0.860
Audio Performance
Audio Performance
Text Performance
ADNI
MIMIC-Mortality
MiMIC-Phenotype
MRI Performance
CXR Performance
0.32
CXR Performance
0.512
0.28
0.504
0.405
0.496
0.390
0.488
0.765 0.770 0,775 7.0 0.785 EHR Performance
0.546 0.552 0,558 0.564 0.570
0.410 0,15 0,420 0.425 0.430 EHR Performance
EHR Performance
Figure 4: Unimodal performance across different datasets and modalities. Unimodal Baseline refers to model trained and evaluated using only single modality. Dots closer to the upper-right comnrr indicate stronger unimodal performance.
4.2 MODALITY BALANCE AND LOSS LANDSCAPE FLATNESS
MASAM consistently improves unimodal performance across all datasets, demonstrating bal- anced multimodal learning. We further evaluate unimodal performance by freezing the encoders obtained from multimodal training and training classifier head on top. The results are summarized Fig. where dots closer to the upper-right comner indicate overall stronger andle-modality perfor- mance. On CREMA-D and KS, MASAM surpasses all baselines, exceeding the unimodal Audio baseline and approaching the unimodal Video baseline. On the clinical dataset MIMIC, which is characterized by substantial noise and missingness, most baselines fail to obtain better unimodal performance than the models trained using unimodal data, indicating severe modality imbalance during learning. In contrast, MASAM demonstrates significant advantage: it achieves consistent superiority over all multimodal baselines, and also significantly exceeds the unimodal baselines for both modalities, suggesting that MASAM is effective in balanced multimodal learning.
MASAM enables all modalities to converge jointly to flatter minima. To better understand the advantages of MASAM in multimodal learning, we further visualize the loss landscape for the MIMIC dataset in Fig. 5 using methods presented in Li et al. (2018). Visualizations for other datasets are provided in Appendix F.2. On both phenotype and mortality tasks, MASAM consistently converges to flatter regions than all baseline methods for each modality encoder. While existing methods make notable progress in mitigating modality imbalance, they only rely on gradient modulation, which overlooks the geometry of the loss landscape. As a result, they may not ensure convergence to flat minima, leaving modality encoders prone to sharp regions and more sensitive interference from other modalities. In contrast, MASAM overcomes these challenges by steering all modality encoders towards flatter regions the loss landscape along their own optimization trajectories, reducing influence of other modalities on their update directions and ensuring that each modality converges stably to a flatter solution. We provide further analysis of flat minima in Appendix
4.3 ROBUSTNESS TO LABEL NOISE AND SENSITIVITY ANALYSIS
In many real-world scenarios, label noise often inevitable for multimodal data. We further conduct experiments by injecting different levels of label noise (20%-60%) into the CREMA-D and KS datasets. The results are summarized in Fig. 6. MASAM consistently outperforms all baselines across varying noise levels. This is because MASAM guides each modality converge towards flatter minima; therefore, MASAM could achieve better generalization and robustness under noisy supervision. We conduct sensitivity analysis over key hyperparameters (a, 3, and p) on MIMIC- Phenotype and KS, with key results summarized in Appendix E.4. MASAM remains robust across wide range of hyperparameter settings.
8

---

Published as a conference paper at ICLR 2026
G Blend
Mr4Pareto
OGM
DrFuse
Late Fusion
MASAM (Ours)
0,300
0.275
6
Loss Value
Value
4-
0.225
0.200
2-
0.175
0
0.150
-0.4
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
(a) The MASAM landscape mortality prediction with respect CXR (left)
EHR (right) encoders.
20
LoG3
J05
LO68
Lo10
Loos Waike
Loss Vilie
Loss Vauue
Loss Value
(b) G-Blend
(c) DrFuse
(d) Late Fusion
(c) MASAM (Ours)
Figure 5: Visualizations of the loss landscape. (a) 2D visualization of the MASAM loss landscape for mortality prediction. (b-e) 3D visualizations of representative methods for disease phenotype classification with respect to all model parameters. MASAM consistently achieves flat solutions jointly for both modalities.
MASAM
0.740
0.677
0.609
0.501
Late Fusion
0.660
0.547
0.477
0.389
0.7
Late Fusi on
0.556
0.65
Models
0.572
0.378
Accuracy
Models
AGM
0.687
0.568
0.485
0.381
0.60
MLA
0.705
0.594
0.489
0.359
0.6
0.681
0.590
0.523
0.416
0.55
0.769
0.709
0.590
0.379
0.5
OGM
0.585
0.50
PMR
0.713
0.549
0,473
0.360
PMR
0.631
0.525
0.469
0.353
0.45
MMPareto
0.717
0.559
0.495
0.393
0.4
0.658
0.571
0.513
0.394
0.40
0%
20%
40%
60%
0%
20%
40%
60%
Noise Level (%)
Noise Level (%6
CREMA-D noise robustness heatmap.
KS noise rohustness heatmap.
Figure 6: The performance of MASAM under varying noise levels the CREMA-D (left) and KS
(right) datasets.
4.4 ABLATION STUDY
To verify the contributions of each component in MASAM, we conduct an ablation study on both the MIMIC-Phenotype and KS datasets (Tables 2a and 2b) by systematically removing individual components and assessing their impact on performance. The variants include: #1 removes APS-based dynamic modality selection, #2 removes decoupled gradient perturbation, #3 employs SAM without APS or gradient decoupling, and uses standard Late Fusion strategy without sharpness-aware regularization. See Appendix F.5 for additional evaluation metrics and Appendix E.6 for comparisons of alternative APS and MDPS designs.
(1) Adaptive perturbation score. The APS module demonstrates significant contribution, as shown in our ablation experiments. By comparing MASAM with #1 as well as comparing #2 with #3, observe that APS achieves notable improvements of 2.89% in MASAM and 2.72% in SAM-only setting. A similar trend is observed on Kinetics-Sounds, where APS yields gains of 2.21% when comparing MASAM with #1 and 4.93% when comparing #2 with #3. This demonstrates that

---

Published as a conference paper at ICLR 2026
Table Ablation study of MASAM. Tables and 2b report results on MIMIC- Phennoype and KS, respectively. v indicates the module is enabled. Experiments are numbered for easier reference in the text. The first row shows our proposed model, while rows through represent different ablation experiments, All results are averaged over 4 random seeds.
(a) MIMIC- MIMIC-PPenotype (AUPRC)
Method
APS
MDPS
SAM
AUPRC
vs. APS
w/o MDPS
vs. SAM Only
vs. Late Fusion
MASAM
0.498
+2.89%
+1.43%
+4.18%
+4.84%
w/o APS
X
/
V
0.484
-1.43%
+1.26%
+1.89%
2
w/o MDPS
/
X
V
0.491
+1.45%
+2.72%
+3.37%
SAM Only
0.478
-1.24%
-2.55%
+0.63%
Late Fusion
X
X
X
0.475
-1.86%
-3.26%
-0.63%
(b)
Kinetics-Soundh (Accuracy)
# Method
APS
MDPS
SAM
Acc.
w/o APS
w/o MDPS
vs. SAM Only
vs. Late Fusion
MASAM
0.740
+2.21%
+2.35%
+7.40%
+16.35%
w/o APS
X
/
0.724
+0.14%
+5.05%
+13.84%
2
w/o MDPS
V
X
0.723
-0.14%
+4.93%
+13.68%
SAM Only
X
X
0.689
-4.84%
-4.70%
+8.33%
4 Late Fusion
X
X
X
0.636
-12.11%
-12.66%
-7.62%
APS effectively resolves the optimization imbalance across modalities, ensuring that each modality converges flatter regions, which leads to better overall performance.
(2) -edouity-decoupeed perturbation scaling. The decoupled gradient module also plays critical role. Comparing MASAM with #2 and with #3, we find that the decoupling mechanism provides relative improvements of 1.43% in MASAM and 1.26% in SAM-only setting, respectively. On Kinetics-Sounds. MDPS brings consistent benefits as well, with improvements of 2.35% when comparing MASAM with #2 and 5.05% when comparing #1 with #3. This affirms that our proposed MDPS plays a crucial role in effectively reducing cross-modal interference and ensuring a stable training trajectory.
(3) Sharpness-aware minimization. By comparing experiments and 3, and 2 and 3, we observe that our APS and MDPS modules lead to performance improvements of 1.26% and 2.72% in the SAM-only model, respectively. On Kinetics-Sounds, combining SAM with APS and MDPS yields even larger relative gains over the SAM-only baseline (Table 2b). This demonstrates that the two proposed modules effectively address the challenges of SAM in multimodal settings: APS mitigates SAM's reliance on the dominant modality, while MDPS ensures independent optimization trajectories for each modality, avoiding the issue of forcing modalities adapt to shared optimization path, thus enhancing multimodal generalization.
5 CONCLUSION
In this paper, propose MASAM, multimodal adaptive sharpness-aware minimization framework to address the modality imbalance issue via attaining flat minima jointly for different modalities. At its core, MASAM dynamically identifies dominant modalities different training stages via the adaptive perturbation score (APS), and applies modality modality-decouppl decoupled perturbation scaling (MDPS) to mitigate the cross- moos- interference in SA.--style perturbation. This design allows each modality to preserve its own independent optimization trajectory while still benefiting from shared information. Extensive empirical evaluations across six multimodal datasets and seven downstream tasks from diverse application domains show that MASAM could attain flat minima for all modalities simultane- ously, achieving balanced multimodal learning, and consequently outperforming all state-of-ther torthe-rrt multimodal learning methods compared. The major limitation of MASAM is that incurs additional training overhead due to the CAsebased optimization and extra backward passes. Our future work will focus exploring lightweight variants improve computational efficiency while preserving performance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is partially supported by the National Natural Science Foundation of China (62302413), the Health and Medical Research Fund (232203 12), an Innovation and Technology Fund of Hong Kong Innovation and Technology Commission (project no ITSS/202/3) and grant under the Collaborative Research with World-leading Research Groups scheme of The Hong Kong Polytechnic University (project no G-SACF).
ETHICS STATEMENT
All datasets used in this work are publicly available. Among them, the two datasets containing clinical data, MIMIC (Johnson et al., 2023; 2019) and ADNI (Jack Jr et al., 2008), were rigorously de-identified by the official institutions prior to release, ensuring that personal privacy of patients compromised. These datasets are employed strietly for research purposes. The work presented here does not pose any foreseeable risk to individuals society.
REPRODUCIBILITY STATEMENT
All datasets used in experiments are publicly available. All source codes will be released, including both the model framework and the dataset processing pipeline. Additional details of the preprocessing steps are provided in Appendix A.4. For hyperparameter selection, we adopt grid search, and the best configurations for each dataset are detailed in Appendix A.3 for reproducibility.
REFERENCES
Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. Advances in Neural Information Processing Systems, 36: 47032-4700 2023.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp- 609-617, 2017.
Akbar Asgharzadeh- -Bnab,, Hashem Kalbkhani, and Sina Azarfardian. An Alzheimer's disease classification method using fusion of features from brain magnetic resonance image transforms and deep convolutional networks. Healthcare Analytics, 4:100223, 2023.
Houwei Cao, David Cooper, Michael Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE transactions affective computing. 5(4):377-390, 2014.
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. PMR: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20029-20080 2023.
Yunfeng Fan, Wenchao Xu, Haozhao Wang, and Song Guo. Cross-modal representation flattening for multi-modal domain generalization. Advances in Neural Information Processing Systems, 37: 66795, 2024.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Zirun Guo, Tao Jin, Jingyuan Chen, and Zhou Zhao Classifier-guided gradient modulation for enhanced multimodal learning. Advances in Neural Information Processing Systems, 37:133328- 133344, 2024.
Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Phillipe Morency, and Mohammed (Ehsan) Hoque UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference Empirical Methods in
11

---

Published as a conference paper at ICLR 2026
Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2046-200000 2056, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1/D19-122 1211. URL https: /www aclweb.
org/ hess/nnhhllogyyy/ /D1 9- 1211.
Nasir Hayat, Krzysztof J Geras, and Farah E Shamout. MedFuse: Multi- mudai fusion with clinical time-series data and chest X-ray images. In Machine Learning for Healthcare Conference, pp. 479-503. PMLR, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1442, 1997.
Chengxiang Huang, Yake Wei, Zequn Yang, and Di Hu. Adaptive unimodal regulation for bal- anced multimodal information acquisition. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 25854 25863, 2025.
Po-Ye Yo-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, haoqi fan, Yanghao Li, Shang- Wen Li, Gargi Ghosh, Jitendra Malik, and Christoph Feichtenhofer. MAVIL: Masked audio-video learners. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 20371-20395. Curran Associates, Inc., 2023.
Clifford Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L. Whitwell, Chadwick Ward, et al. The Alzheimer's 's disease neuroimaging initiative (ADNI); MRI methods. Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonunce in Medicine, 27(4): 685-691, 2008.
Qingyuan Jiang, Longfei Huang, and Yang Yang. Rethinking multimodal learning from the perspec- tive of mitigating classification ability disproportion. In NeurIPS, 2025.
Alistair EW Johnson, Tom Pollard. Seth Berkowitz, Nathaniel R Greenbaum, Matthew Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. MIMIC-XRR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317,001
Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, al. MIMIC-IV, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017.
Pham Khanh, Hoang-Chee Chau Luong, Boris Mordukhovich, and Dat Tran. Fundamental convergence analysis of sharpness-aware minimization. Advances in Neural Information Processing Systems, 37:13149 13182, 2024.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher SAM: Information geometry and sharpness-aware minimisation. In International Conference on Machine Learning, pp. 11148 11161. PMLR, 2022.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances Neural Information Processing Systems, 31, 2018.
Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei. Chunxiao Li, and Yi Zhou. Boosting performance with adaptive Gradient Modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22214-221 22224, 2023.
model
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang, Friendly sharpness- aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5631-56400, 2024.
Xiaokang Peng. Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fi fly gradient modulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8238- 8247, 2022.
12

---

Published as a conference paper at ICLR 2026
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look at the Hessian eigenspectrum of deep neural networks and its applications to regularization, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9481-9488, 2021.
Sören Richard Stahlschmidt, Benjamin Ulfenborg, and Jane Synnergren. Multimodal deep learning for biomedical data fusion: a review. Briefings in bioinformatics, 23(2):bba5669, 2022.
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3769-3778, 2023.
Weiyao Wang. Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 22695-12705. 2020.
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recogni- tion with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1-6. IEEE, 2015.
Yake Wei and Di Hu. MMPareto: Boosting multimodal learning with innocent unimodal assistance. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp.
2024.
Yake Wei, Siwei Li, Ruoxuan Feng, and Di Hu. Diagnosing and re-learning for balanced multimodal learning. In European Conference on Computer Vision, pp. 71-86. Springer, 2024.
Tao Wu, Tie Luo, and Donald C Wunsch II. CR-SAM: Curvature regularized sharpness-awe aware minimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 61 6152, 2024a.
Zhenbang Wu, Anant Dadu, Nicholas Tustison, Brian Avants, Mike Nalls, Jimeng Sun, and Faraz Faghri. Multimodal patient representation learning with missing modalities and labels. In The Twelfth International Conference on Learning Representations, 2024b.
Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Advances in Neural Information Processing Systems, 34:18722-18735 2021.
Wenfang Yao, Kejing Yin, William K Cheung. Jia Liu, and Jing Qin. DrFuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 16416-16424, 2024.
Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. Flex-MoE: Modeling arbitrary modality combination via the flexible mixture-of-experts. Advances in Neural Information Processing Systems, 37:98782-98805, 2024.
Xiaohui Zhang, Jachong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27466,2746, 2024.
Yingying Zhu, Minjeong Kim, Xiaofeng Zhu, Daniel Kaufer, Guorong Wu, Alzheimer's Disease Neu- roimaging Initiative, et al. Long range early diagnosis of Alzheimer's disease using longitudinal MR imaging data. Medical image analysis, 67:101 2021.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTAL DETAILS
A.1 1 DATASETS AND BASELINES
This section presents the datasets used and baseline models compared in our study.
MIMIC (Johnson et al., 2023; 2019): MIMIC-IV contains de-identified clinical records of patients admitted to ICUs at Beth Israel Deaconess Medical Center between 2008 and 2019. Following prior works (Hayat al., 2022; Yao et al., 2024), we extract eight important physiological variables along with the Glasgow Coma Scale (GCS) recorded within the first 48 hours of ICU stay as the EHR modality input. For the image modality, we use the most recent anterior-postene image acquired within the first 48 hours of ICU stays. The downstream prediction tasks include disease phenotype classification of 25 diseases and in-ICU mortality prediction. Given the inherent class imbalance in clinical data and the importance of correctly identifying positive cases, we adopt the Area Under the Ceeiiion-RReall Curve (AUPRC) as the primary evaluation metric, and report the Area Under the Receiver Operating Characteristic Curve (AUROC) as complementary indicator.
(AP) view X-ray
ADNI (Jack Jr et al., 2008): The ADNI project has evolved through four phases, namely ADNI- 1, ADNI-GO, ADNI-2, and ADNI-3, and has developed into systematic and comprehensive multimodal research database that provides rich information for clinical prediction of Alzheimer's disease. Following the MUSE (Wu et al., 2024b) preprocessing pipeline, we selected demographic information, genetic data, and cognitive assessment scores as tabular inputs, and employed MRI scans as the visual modality. These multimodal data were utilized to perform classification across three diagnostic categories: Alzheimer's Disease (AD), Mild Cognitive Impairment (MCI), and Cognitively Normal (CN).
CREMA-D (Cao et al., 2014) and Kinetics-Sounds (Arandjelovic & Zisserman, 2017): Both datasets contain paired audio and video modalities. CREMA-D consists of audiovisual recordings of actors reading 12 English sentences under six common emotional states (angry, happy, sad, neutral, disgust, and fear). with perceptual validation provided by 2,443 crowd workers. The Kinetics dataset contains short video clips of approximately ten seconds collected from YouTube, covering a wide range of daily activities, sports, and social interactions. It is commonly used for action recognition tasks involving 31 distinct categories. Both datasets are processed in accordance with the OGM (Peng al., 2022).
UPMC-Food101 (Wang et al., 2015): This dataset consists of text data paired with food images. Each sample contains a food image and the corresponding web text, covering 101 distinct food categories. It is commonly used as a multimodal benchmark for food classification. We use accuracy as the primary evaluation metric.
UR-FUNNY (Hasan et al., 2019): This dataset contains three different modalities: text, audio, and video, and each sample used for a binary humor classification task. We process this data following Li et al. (2023) and select accuracy as the primary evaluation metric.
A.2 BASELINES
Baseline methods
We compare MASAM with the following existing methods.
Uni-modal: Models trained independently on each single modality using its dedicated encoder.
Late Fusion: A standard fusion strategy that concatenates modality-specifii features at the decision level after separate feature extraction.
G-Blend (Wang al., 2020): It modulates the contribution of each modality during training by incorporating generation quality and overfitting sensitivity metrics.
AGM (Li et al., 2023): Dynamically reweights modality gradients based on their Shapley value- estimated contributions to the overall loss.
MLA (Zhang et al., 2024): It adopts a shared prediction head while alternately updating unimodal branches, enforcing directional orthogonality across modalities.
MMPareto (Wei & Hu, 2024): It resolves optimization conflicts between unimodal and multimodal objectives via Pareto-based gradient projection.
PMR (Fan et al., 2023): It mitigates modality imbalance by promoting weaker modalities through prototypical clustering while regularizing dominant ones with prototyp--hased entropy.
14

---

Published as a conference paper at ICLR 2026
Table 3: Best hyperparameter configurations across datasets, p. score weight, and momentum are selected from predefined search spaces.
Dataset
Score Weight
Momentum
ADNI
0.5
0.3
0.9
CREMA-D
0.6
0.3
0.9
UPMC-Food101
0.3
0.3
0.9
Kinetics-Sounds
0.7
0.3
0.9
MIMIC-Mortality
0.1
0.3
0.9
MIMIC-Phenotye
0.5
0.3
0.9
UR-FUNNY
0.5
0.3
0,9
MedFuse (Hayat et al., 2022): It sequentially aggregates multimodal representations using an LSTM-based fusion architecture, designed for partially paired clinical data.
DrFuse (Yao et al., 2024): integrates EHR and CXR modalities by disentangling modality- specific features and aligning shared representations for robust fusion.
AUG (Jiang et al., 2025): It enhances modality balance from the perspective of classifier capability, using boosting-style strategy strengthen the weak modality's ' classifier.
InfoReg (Huang al., 2025): It balances the information acquisition of different modalities in the early stage by applying regularization to modalities with overly high information gain, leading to more balanced learning process.
A.3 SEARCH SPACE
We conducted each experiment with four random seeds ((42, 123, 1234, 20244) and report the averaged results. Different datasets correspond distinct search spaces, and the optimal hyper- parameters obtained from grid search are reported in Table 3 to ensure the reproducibility of our results. Specifically, we search over p E {0.1, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9), E {0.1, 0.3, 0.5). and B E {0.5, 0.7, 0.9}
A.4 CLINICAL DATA PROCESSING
A.4.1 MIMIC DATASET
We extracted EHR and CXR data within the first 48 hours of ICU admission from MIMIC-IV (Johnson et al., 2023) and MIMIC-CXR (Johnson et al., 2019), following the preprocessing steps described in DrFuse (Yao al., 2024) and MedFuse (Hayat et al., 2022).
For EHR data, we excluded variables with more missing ratio greater than 80%, such as those in the blood gas category. This resulted in 12 variables, including Heart Rate, Diastolic Blood Pressure, Mean Blood Pressure, Systolic Blood Pressure, Respiratory Rate, Temperature, Peripheral Capillary Oxygen Saturation, Glucose, Glasgow Coma Scale Total Score, Glasgow Coma Scale Motor Response, Glasgow Coma Scale Verbal Response, and Glasgow Coma Scale Eye Opening Response. The missing ratio information shown in Fig. We additionally added a masking column for each variable to indicate whether the value is missing at each time step, enabling the model to learn missingness-relant related patterns. Missing values were imputed using the median, and variables were normalized with statistics computed from the training set. For CXR images, we used the most recent AP-view chest X-ray taken within the first 48 hours of the ICU admission.
We also analyzed the distribution of disease phenotypes in MIMIC dataset, as shown in Table 4. The dataset contains 7,623 samples across 25 different clinical conditions, with varying positive ratios ranging from 5.26% to 49.26%. Notably, some labels exhibit substantial imbalance. which poses additional challenges for robust multimodal learning. This diverse distribution ensures comprehensive evaluation of multimodal fusion methods across different disease prevalence levels.
15

---

Published as a conference paper at ICLR 2026
EHR Variables Missing Ratio Missing Avaiable %
Glucose
88.3%
Glasgow Scele
84.0%
Response
Glasgow Comn Scale
Response
Glasgow Coma Scale
Eye Opening Response
Temperature
82, 0%
Peripheral Capillary
Oxygen Saturation
Mean
Blood Pressure
Blood Pressure
Diastalic
36.4%
Blood
Systalic
36.4%
Heart Rate
Respiratory Rate
60
80
Percentage (%)
Figure 7: Missing ratios of EHR variables in our dataset. We use GCS as an abbreviation for the Glasgow Coma Scale and BP as abbreviation for Blood Pressure. Variables with missing ratios greater than 80% were excluded from our analysis.
Table 4: Statistical overview of disease phenotypes in MIMIC dataset. The table shows the distribution of positive and negative samples across 25 clinical conditions, along with their prevalence rates.
Disease Phenotypes
Total
Positive
Negative
Positive %
Fluid and electrolyte disorders
7623
3755
3868
49.26
Essential hypertension
7623
3431
4192
45.01
Disorders of lipid metabolism
7623
2903
4720
38.08
Cardiac dysrhythmias
7623
2858
4765
37.49
Acute and unspecified renal failure
7623
2696
4927
35.37
Respiratory failure
7623
2400
5223
31.48
Congestive heart failure
7623
2087
5536
27.38
Coronary atherosclerosis
7623
1935
5688
25.38
Septicemia
7623
1855
5768
24.33
Complications of surgical procedures
7623
1668
5955
21.88
Shock
7623
1605
6018
21.05
Pneumonia
7623
1578
6045
20.70
Chronic kidney disease
7623
1512
6111
19.83
Diabetes mellitus without complication
7623
1499
6124
19.66
Hypertension with complications
7623
1401
6222
18.38
Other liver diseases
7623
1320
6303
17.32
COPD and bronchiectasis
7623
1159
6464
15,20
Other lower respiratory disease
7623
980
6643
12.86
Pleurisy; pneumothorax
7623
808
6815
10.60
Diabetes mellitus with complications
7623
794
6829
10.42
Conduction disorders
7623
773
6850
10.14
Acute cerebrovascular disease
7623
762
6861
10.00
Acute myocardial infarction
7623
694
6929
9.10
Gastrointestinal hemorrhage
7623
579
7044
7.60
Other upper respiratory disease
7623
401
7222
5.26
16

---

Published as a conference paper at ICLR 2026
Table 5: Missing ratios and descriptions of variables in the ADNI dataset.
Categury
Name
Missing Ratio (%)
Description
Demogrophics
AGE
0.17
Age (ycars)
PIGENDER
Gender
PTEDUCAT
0.00
Yeurs of education
PTETTCCT
0.00
Ethnicity
PIRACCAT
Race calogory
PTMARRY
0.04
Marital status
Genetics
APOE4
8.64
APOE e4 allete copies (0/1/2)
CSF/PET Biomarkers
ABETA
49.77
CSF 9-amyloid protein
CSF
PTAU
49.77
CSF phosphorylated Tau
FDG
37.45
FDG.PET cerebral metabolism
PIB
PIB-PET amyloid deposition
AV455
$3.20
AVTS-PET amyloid deposition
FBB
85.78
Flotbetaben PET
Cognitive Scorey
Clinical
Rating
ADAS11
0.41
Alzheimer's s Disease Assessment Scale, items
ADAS13
0.91
Alzheimer's Discase Assessment Scale, 13 items
ADASQ4
Alzheimer's Discase Assessment Scale,
MMSE
0.04
Mini-Mental State Examination
RAVLT immediate
0.33
Rey Auditary Verbal Learning Test Immeddate recall
RAVIT leamingg
0.33
RAVLT Learning
RAVLT forgetting
0.37
RAVLT Forgetting
RAVIT pere, Forget
Percent forgetting
TAQ
1.12
Functional Activilies Questionnaire
MOCA
35.26
Montreal Cognitive Assessment
A.4.2 ADNI DATASET
We selected demographics, genetics, CSF/PET biomarkers, and cognitive scores the tabular modality, while excluding variables with a high missing rate (>90%). The missing ratio of these tabular features is summarized in Table For missing values within variables, we employ mean imputation and include corresponding mask indicators, while all variables are normalized using statisties computed from the training set. For prediction, we used baseline data for each subject and followed prior studies (Wu et al., 2024b; Asgharzadeh-Bonn Bonab et al., 2023; Yun et al., 2024) by merging EMCI (early mild cognitive impairment) and LMCI (late mild cognitive impairment) into single MCI group, resulting in three classes: Alzheimer's Disease (AD), Mild Cognitive Impairment (MCI), and Cognitively Normal (CN). For the imaging modality, we adopted T1-weighted MRI scans; when a subject's 's bascline MRI was unavailable, substituted scans acquired within six-month window (Zhu et al., 2021).
B RELATED WORK
B.1 SHARPNESS-AWAR MINIMIZATION (SAM) AND ITS VARIANTS
Improving model generalization has long been associated with finding flatter minima in the loss landscape. SAM (Foret et al., 2021) explicitly encourages flat solutions by minimizing the worst-case loss within a neighborhood of the parameters. Building on this idea, several variants have been proposed:
FisherSAM (Kim al., 2022): Reformulates the parameter space Riemannian manifold using the Fisher information matrix. Sharpness then measured as a distributional distance between perturbed and original parameters, offering a more principled curvature metric.
SAGM (Wang et al., 2023): Extends SAM by simultaneously considering three objec- tives tives-eemirti empirical risk, perturbed loss, and gradient similarity- guide optimization towards flatter and more stable regions.
17

---

Published as a conference paper at ICLR 2026
CR-SAM (Wu al., 2024a): Observes that curvature estimation is highly sensitive to gradient scales, and introduces a scale-invariant approach to normalize curvature estimation, thus improving stability across different architectures.
FriendlySAM (Li et al., 2024): Proposes directionally perturbed gradients, which diversify per- turbation directions rather than restricting them to the normalized gradient, leading to improved generalization.
These approaches have refined the measurement of sharpness and the design of perturbations, but they are all developed unimodal contexts. Their direct extension to multimodal learning is non-trivial because different modalities exhibit heterogeneous convergence dynamics and distinct loss landscape geometries.
B.2 ALLEVIATING MODALITY COMPETITION IN MULTIMODAL LEARNING
A central challenge in multimodal learning is modality competition, where strong modalities dominate optimization, causing weaker but informative modalities to be under-optimized. This phenomenon often leads degraded multimodal fusion and even underperformance compared unimodal baselines (Wang et al., 2020), Several strategies have been proposed to mitigate this problem:
Gradient modulation approaches:
GBlend (Wang et al., 2020): Introduces gradient blending coefficients based on the overfitting- to-generalization ratio of each modality, dynamically reweighting their contributions during training.
OGM (Peng et al., 2022): Promotes exploration of weaker modalities by slowing down the optimization of stronger ones.
AGM (Li et al., 2023): Accelerates weaker modalities by explicitly defining their relative strength
and adjusting gradients accordingly.
PMR (Fan et al., 2023): PMR leverages prototypical representations diagnose modality
strength via prototype-basee based separability, regularizing strong modalities to prevent dominance and enhancing weak modalities by encouraging more discriminative prototype alignment.
Training schedule and gradient conflict approaches:
MLA (Zhang et al., 2024): Proposes an iterative training strategy that alternates across modalities, combined with an orthogonality constraint on modality-specific gradients to reduce interference. .MMarrto (Wei & Hu, 2024): Frames multimodal optimization as a multi-objectivg problem and applies Pareto optimization to resolve conflicts between unimodal and multimodal gradients. Diagnosing-Relcamning (Wei et al., 2024): Employs a global soft reinitialization strategy, resetting modalities to varying extents during training balance learning progress.
-
While these methods effectively rebalance optimization across modalities, they do so primarily by modulating gradient magnitudes or training schedules. This helps address imbalance but does not account for the loss landscape geometry of each modality. Consequently, they cannot guarantee that each modality converges to its own flat minimum, leaving the generalization benefits of flatness-aware optimization unexplored.
In summary, prior work on SAM and its variants focuses on flatness for unimodal tasks, while multimodal learning methods emphasize gradient-based balancing across modalities. However, no existing method explicitly addresses how sharpness-aware optimization interacts with multimodal imbalance. This motivates our proposed MASAM framework, which integrates Ratness-aware perturbation with modality-spedee adaptation to overcome the limitations of both lines of research.
C DERIVATION OF OBSERVATION
Without loss of generality, here we assume late fusion two modalities. In the following content continue to notations in Section 3.1. In late fusion methods, although each modality appears to be optimized independently, their training remains inherently coupled through the shared fused prediction. Specifically, the gradient of the fused loss with respect to the parameters of modality m
18

---

Published as a conference paper at ICLR 2026
can be expressed as:
N C
Fusse
Difie Öz dz!""
N
Dipm (x:²t) oi d0m
Ve... Liuse
fuse Vi,e)) Vie
(15)
Difiue Oz!use dz 00m
Here, the fused prediction is defined as shec = g (W " W .z," +b.) where g(.) denotes the task-specific activation function: a softmax function in multiclass classification tasks with categorical cross-entropy loss, or a sigmoid function in multilabel tasks with binary cross-entropy loss. This formulation follows the standard late fusion paradigm, where the fused weight vector We: in Eq. (4) can be decomposed into mopality-specific parts W and W From this formulation, is evident that dominant modality (e.g., m2) with stronger discriminative signals will exent more influence on the final prediction through a larger weighted contribution (W dominance causes the computed loss gradient be skewed in favor of the stronger modality, while the weaker modality (e.g., mi receives gradients that are misaligned with its own representation, thereby limiting its learning capacity and reducing effective exploration.
o, =,²)). This
This issue also extends to the computation of the SAM perturbation Eq. (3). Since the gradient itself is already biased by the dominant modality, the resulting perturbation vector for the weaker modality becomes largely determined by the optimization direction favored by the strong modality. As result, this perturbation may not only fail to benefit the weaker modality but may even steer in suboptimal or harmful direction, thereby undermining the effectiveness of SAM and exacerbating the imbalance across modalities.
D PROOF OF THEOREM
We now provide full proof of Theorem I using the framework of inexact gradient descent (IGD).
Let g vm Ciuce denote the normalized gradient of the fusion objective with respect to the Ven Ca
encoder parameters of modality at step I.
Lemma 1 (Perturbation Error Bound for Fusion Gradient). For each modality m, given the Liuse- Lipschitz continuity of V Liuse, the perturbed gradient of Liuse satisfies
V ( + Pt. Y'n Bim)) V Vou,o Lruse
Lruse
(16)
Proof. By Lipschitz continuity of V VLusee i.e.,
Von Cruse (0%, + P: yim gm) V Cruss (0t))l S L L Ipp 7mn 1²
Eq. (16) is obtained as |[ and g
Lemma (MASAM as IGD instance). The MASAM update (Eq. (12)) modality m is an instance of the inexact gradient descent method as:
g++1
0r m..n.. with inexact condition l9gh V Ven, (0t.)) S LLuse
(17)
Proof. Let gin Vo Cruss (Oth + Pr 7m gm) Vo,, L. (0%.). Then the MASAM update gives Ot-1 = oe "nim Together with Eq. (7) and Eq. (16), the inexact condition can be obtained as:
1 Va.. Crotal (Oth) -|vl= Liuse (0th + Pt 7m Bin) Ven L. (0 Vo., Lroral (0th)
V 6m Lluse (0th + Pt y/n g.m) + Van Lm (B Von, Liuse (Pth) Von (8%h))
V 8m. Linso (0 + Pt. Im B(m)
V pm Linso (0m)) < Leuso
We next provide the proof of Theorem Theorem 3.3).
based on Lemma 1, Lemma 2, and Khanh et al. (2024,
19

---

Published as a conference paper at ICLR 2026
Table 6: This table compares the AUPRC of different models across 25 disease phenotypes. Bold values indicate the best performance, and underlined values indicate second-best performance. The Average Rank metric ranks all models for each phenotype individually and calculates their mean ranking across all 25 disease labels. MASAM consistently ranked first second across nearly all phenotypes and achieved the highest average rank overall, validating the effectiveness of our approach.
Clinical Condition
Uni-EHR
Uni-CXR
G-Blend
MASAM
Acute renal failture
0.593
0,519
0,590
0.577
0,578
0.587
0.594
0.599
cerchrovascular disease
0.487
Acute myocardial infarction
0.188
0.230
0.253
Cardiac thsmas
0.638
0.627
0.625
0.653
Chronic kidney discase
0.472
0.461
0,550
0.535
0,528
0,542
0.539
0.556
COPD and bronchiectasis
0.328
0.361
0.424
0.413
0.407
0.403
0.420
0.572
0.435
Surgical complications
0.331
0.286
0.344
0.338
0,335
0.335
0.333
0.690
0.335
Conduction disorders
0.214
0.597
0.615
0.626
0.618
0.616
0.621
0.199
0.634
CHF; nonhypertensive CAD
0.548
0.671
0,703
0.091
0.682
0.689
0.699
0,506
0.719
0.472
0.579
0,610
0.598
0.605
0.588
0.599
0.417
0.621
DM with complications
0.591
0.238
0,571
0.566
0.556
0.539
0,577
0.177
0,587
DM without complication
0,380
0.295
0.400
0.395
0.395
0.393
0.404
0.207
0.407
Disorders of lipid metabolism
0.563
0.551
0.585
0,575
0,573
0.584
0.578
0.241
0,586
Essential hypertension
0.542
0.524
0.566
0.565
0.565
0.568
0.562
0.217
0.576
Fluid and electrolyte disorders
0.678
0.625
0.689
0.677
0.676
0.681
0.687
0.436
0.686
Gastrointestinal hemonhage
0.171
0.168
0.199
0.200
0.188
0.203
Secondary hypertension
0,506
0.511
Other diseuses
0.399
0.419
0.445
Other respiratory discase
0.160
0.178
0.173
0.185
Other upper respiratory disease
0.206
0.161
0,223
0.247
0,219
0.265
0.238
0.544
0.309
Pleurisy: poeumothorax
0.134
0.238
0.232
0.230
0,215
0.210
0.229
0.415
0.235
Pocumonia
0.392
0.392
0.438
0.431
0.428
0.433
0.434
0.335
0,441
Respiratory failure
0.612
0.564
0.644
0.637
0.642
0.643
0.643
0.617
0.650
Septicemin
0.569
0.431
0.566
0.565
0.567
0.565
0.565
0.699
0.576
Sback
0.571
0.440
0,581
0.572
0.579
0.589
0.590
0.608
0.594
Average Rank
6.38
6.72
4.36
4.98
5.16
5.16
4.72
4.16
2.36
Proof of Theorem 1. Given that Khanh et al. (2024, Theorem 3.3) already established the conditions for IGD method and that the IGD format of MASAM is given in Eq. (17), the convergence of MASAM requires that
The
0c,
(ii)
40,
(iii)
Liuse Pe. < 00, (iv) lim sup LiusePt < 2, (18)
t-000
t=1
which is equivalent to the conditions in Eq. (14). Then the convergence of MASAM can be obtained.
E ADDITIONAL. EXPERIMENT RESULTS
E PREDICTION PERFORMANCE ACROSS DISEASE PHENOTYPES
We report the detailed breakdown of prediction performance across different disease phenotypes in Table Table 6.
E.2 LOSS LANDSCAPE VISUALIZATION
We visualize the loss landscapes across different baseline models for all datasets: MIMIC-Phenotype (Fig. 8), MIMIC-Mortality (Fig. 9), CREMA-D (Fig. 10), Sinetics-Soundl (Fig. 11), UPMC-Food (Fig. 12), and ADNI (Fig. 13). Across all these figures, clearly observe that our proposed MASAM method converges to a noticeably flatter region than all other models, demonstrating stronger generalization and robustness.
20

---

Published as a conference paper at ICLR 2026
30
30
25
25
25
20
15
10
10
-1.0
-1.
-1.0
-10
1.0
-0
-0.3
-0.5
-0.5
-0.
-0.
-05
0.0
0.0
0.0
0.0
0.0
0.0
0.5
0.5
0.5
0.5
0.5
0.5
1.0
1.0
1.0
1.0
1.0
1.0
1t
Loss Value
Loss Value
Loss Value
(a) G-Blend
(h) OGM-GE
(c) MMPareto
20
15
20
Loss
15
10
-0.5
-0.5
-1.0
-0.4
-0.5
-05
0.0
0.0
0.0
0.0
0.0
0.0
X
0.2
0.5
0.5
0.5
1.0
1.0
1.0
Loss Value
Loss Value
Loss Value
(d) DrFuse
(e) Late Fusion
( MASAM (Ours)
Figure 8: 3D loss landscape visualization of MASAM and baseline models in disease phenotype classification. MASAM consistently converges to wider and flatter region, indicating better generalization and robustness.
E.3 ADDITIONAL METIRCS
In this section, we provide additional experimental results to complement the main findings. Specif- ically, we report AUROC for MIMIC-Phenotype and MIMIC-Mortalit, mAP for CREMA-D, Food101, and KS, and Accuracy for ADNI. These results offer more comprehensive evaluation of the effectiveness of our proposed method.
As shown in Table 7, MASAM consistently achieves notable improvements across diverse datasets and tasks these supplementary metrics, providing strong evidence of the effectiveness of our approach.
E.4 SENSITIVE ANALYSIS
To assess the robustness of our proposed model with respect to hyperparameter choices and to gain insights into the functional roles of key components in our design, we perform sensitivity analysis on three key parameters: a, 3, The parameter controls the weighting between the loss descent rate and gradient alignment in APS computation. A higher a emphasizes the contribution of the loss descent rate when estimating modality dominance. The parameter determines the momentum accumulation rate for loss smoothing; a larger places more emphasis on the historical trend of loss values, making the model's judgment of descent more reliant on long-term behavior, whereas smaller increases sensitivity to short-term fluctuations. The parameter specifies the magnitude of the perturbation step in sharpness-aware optimization. A larger expands the neighborhood over which the loss surface is explored, effectively enlarging the search space.
MASAM demonstrates strong robustness the hyperparameter a, while also highlighting the importance of gradient alignment in quantifying dominant modalities. As shown in Fig. 14a, MASAM consistently outperforms Late Fusion across wide range of values (0.1 to 0.8). Notably, the
21

---

Published as a conference paper at ICLR 2026
Loss
Loss
-0.
0.2
022
-0.2
-02
0.0
0.0
0.0
0.0
0.0
0.0
0.2
0.2
0.2
0.2
02
0.2
0.4
0.4
0.4
0.4
Loss Value
Loss Value
Loss Value
(a) G-Blend
(h) OGM-GE
(c) MMPareto
0
-0
0.0
0.0
0.0
0.0
0.0
0.0
0.2
0.2
0.2
0.2
0.2
0.2
0.4
0.4
0.4
0.4
0.4
0.4
Loss Value
Loss Volue
Loss Value
(d) DrFuse
(c) Late Fusion
(f) MASAM (Ours)
Figure 3D loss landscape visualization of MASAM and baseline models mortality prediction task. MASAM converges to a wider and flatter region, indicating improved robustness and generalization compared to all baselines.
model achieves the best performance when is set between 0.2 and 0.3, suggesting that the degree of gradient alignment plays a more critical role than loss descent rate estimating modality dominance in our settings. A similar trend is observed on the KSdataset (see Fig. 14d), where MASAM again reaches its peak performance around a = 0.2-0.3, further confirming the central role gradient alignment in estimating modality dominance across diverse multimodal settings.
Similarly, MASAM exhibits strong robustness with respect to the momentum parameter 3, further confirming that incorporating global loss trends is beneficial for estimating the learning speed of each modality. As shown in Fig. 14b, MASAM maintains consistently high AUPRC scores across a broad range of 9 values (0.1 to 0.7), with a slight increase observed at 3 0.9. where the model achieves its best performance. A similar pattern is observed on the KSdataset (see Fig. 14c), where the model likewise attains its highest accuracy trends enhances the reliability of modality-wise -wise learning estimation across datasets. Overall, the performance remains stable and significantly superior to that of Late Fusion, demonstrating that leveraging global loss dynamics helps stabilize modality-wise learning assessments.
0.9, further reinforeing that emphasizing long-term loss
MASAM continues to demonstrate strong robustness with respect to the perturbation magnitude As shown in Fig. 14c, the MIMIC-Phenotype dataset exhibits a mild dip at 0.6, followed by a recovery as p increases. In contrast, the KS dataset (Fig. 14f) displays a steadily increasing trend throughout the entire range, with the best performance achieved at larger perturbation radii (p = 0.7-0.9) These observations suggest that the optimal perturbation magnitude can vary across datasets, reflecting differences in loss landscape geometry. Although the KS curve shows more noticeable variation in the small-p region (0.1 (0.3), the performance becomes stable and consistently strong for p 0.3, supporting the robustness of MASAM under wide range of perturbation scales.
22

---

Published as a conference paper at ICLR 2026
40
20
20
10
10
10
1.0
0.0
0.0
0.5
0.0
0.0
0.5
0.0
0.0
0.5
0.5
0.5
0.5
1.0
1.0
1.0
23
23 Loss Value
16
Loss Value
Loss Value
(a) AGM
(b) G-Blend
(c) Late Fusion
40
40
40
30
30
LoGs
Loss
20
10
10
10
-0.5
-0.5
-0.5
0.0
0.5
0.5
0.0
X
0.0
0.5
0.5
0.0
X
0.0
0.5
0.5
0.0
1.0
1.0
1.0
Loss Value
Loss Value
Loss Value
(d) MLA
(e) MMPareto
(f) MASAM (Ours)
Figure 10: 3D loss landscape visualization of MASAM and baseline models in CREMA-D. MASAM
consistently converges wider and flatter region, indicating better generalization and robustness.
E.5 ABLATION STUDY RESULTS MEASURED BY AUROC
We provide the results of ablation studies in AUROC in Table 9.
E.6 APS AND MDPS ALTERNATIVES
To assess whether our design choices are necessary, we evaluate several alternative formulations for APS and MDPS. For APS, we test: (1) replacing our score with the codality-scoring mechanism used OGM (Peng et al., 2022); and (2) setting a in APS, which removes the loss-decay term and keeps only the
component (t) Table 8 summarizes the comparison between MASAM and the alternative designs. For MDPS, we explore two alternative perturbation directions:
(1) using the gradient difference between fusion and unimodal objectives, gi = Ve., Ltuse - Vo.. f.
(2) using the orthogonal component the unimodal gradient, g = Ve__, Lruse. Table summarizes the comparison between MASAM and the alternative designs. We first examine the APS variants (#1 and #2). Using the OGM scoring rule reduces performance on KS by 1.22% and on MIMIC- Phenotype by 1.01%. Removing the loss-decay term by setting a 0 also causes drops of 0.95% on KS and 1.81% on MIMIC-Phenotype. These results indicate that the APS alternatives do not reliably identify the dominant modality for SAM optimization. Next, evaluate the MDPS variants (#3 and #4). The gradient-diffene 5.41% on KS and 7.23% on MIMIC-Phenotype. The orthogonal-gradien variant g also performs worse than MASAM on both datasets. This shows that these alternatives cannot effectively decouple cross-modal interference during SAM perturbation.
variant g leads to a much larger degradation, with drops of
E.7 GRADIENT ALIGNMENT ANALYSIS
Fig. 15 reports the cosine similarity between the unimodal and fusion gradients for the audio- video encoders in KS and the EHR-CXR MIMIC-Phenotype training epochs. Across all modalities,
23

---

Published as a conference paper at ICLR 2026
30
30
25
25
15
10
10
5
10
5
-
9.0
-0.5
0.0
0.5
0.0
-0.5
-0.5
0.0
0.D
0.5
0.0
0.5
0.5
0.0
0.5
-.
1.0
1.8.0
1.8.0
T5 Loss Value
Loss Value
Loss Value
(a) AGM
(b) G-Blend
(c) Late Fusion
20
20
15
Loss
15
Loss
10
Loss
15
10
10
5
0
0.0
0.5
-0.5
-0.5
0.0
0.0
0.0
0.5
0.5
X
0.5
0.5
X
0.5
0.5
1.0
1.0
1.0
222 12
Loss Value
Loss Value
Loss Value
(d) MLA
(e) MMPareto
( MASAM (Ours)
Figure 11: 3D loss landscape visualization of MASAM and baseline models on the Kinetics-Sounds dataset. MASAM converges to a wider and flatter region, demonstrating improved generalization and robustness.
MASAM consistently maintains higher and more stable alignment than all baselines, and the gap widens as training progresses. This empirically supports that our MDPS design effectively aligns unimodal and fusion objectives, leading to more coherent multimodal optimization.
E.8 EXTENSION TO TRI-MODALITY DATASET
We further demonstrate the scalability of MASAM by extending it to a three-modality setting on the UR-FUNNY (Hasan et al., 2019) and comparing with baselines that also support this setting. To extend MASAM the three-modality setting, we compute the modality-specific APS(!) for all modalities each training step and identify the dominant modality m arg max, APS(!) MASAM then applies the MDPS-guided SAM perturbation only to the dominant modality m". while the remaining modalities proceed with their standard gradient updates.
F FLAT MINIMA ANALYSIS
Gradient scaling methods are effective in solving modality competition, but they cannot ensure that each modality converges a flat minima. In this part, we provide an in-depth analysis.
These works (Li et al., 2023; Peng et al., 2022; Wang et al., 2020) adopt gradient-scaling strategies, where each modality's 's gradient scaled by static coefficient based on modality importance score. This leads to an update rule of the form:
0 g++1 = 0' - - V
(19)
The flatness of the loss landscape is often correlated with the largest eigenvalue of the Hessian matrix, Xmaze ( ( (wr) (Yang et al., 2021; Sankar al., 2021; Wu et al., 2024a). To analyze how the update
24

---

Published as a conference paper at ICLR 2026
40
30
30
Loss
20
0.4
Loss
20
0.4
Loss 20
0.0
x
0.0
10
0.0 x
0.2
0.2
-0.2
0.0
0.4
-..
0.0
-02
0.0
0.4
0.4
0.4
20
25
20
45
Loss Value
Loss Value
Loss Value
(a) AGM
(b) G-Blend
(c) Late Fusion
40
30
20
0.4
Loss
0.2
0.2
-02
0.2
0.0
x
0.0
0.0
x
-0.4
0.7
0.0
0.2
0.4
-04
0.2
0.0
0.2
0.4
-0.4 0.2
0.0
0.2
0.4
Loss Value
Loss Value
Loss Value
(d) MLA
(e) MMPareto
(f) MASAM (Ours)
Figure 12: 3D loss landscape visualization of MASAM and baseline models on the UPMC-Food dataset. MASAM converges to wider and flatter region, demonstrating improved generalization and robustness.
affects this flatness, apply first-order Taylor expansion of the Hessian around (( se H(0)) π VH((')
(20)
Since the Hessian is symmetric, we approximate the change in its largest eigenvalue using the Rayleigh quotient:
Amax (H()++1)) 2 Xmaxx (H(0')) c/c. [ V
(21)
where u is the eigenvector corresponding to (maax(H(0))
This analysis shows that simply scaling the gradient by positive coefficient e does alter the direction of the term (H(w) (w))), which governs whether the model moves towards sharper or flatter regions. The effectiveness of such an update in reducing sharpness depends on natural geometry of the optimization landscape, not the algorithm itself.
In multimodal settings, this limitation especially critical: a modality that initially converges towards flatter region may be pulled into sharper areas due to the continued influence of other modalities. Gradient-scaling methods lack explicit mechanisms to prevent this, whereas MASAM explicitly regularizes each modality towards flatter minima, thereby mitigating cross-modality interference and ensuring more stable optimization.
G COMPUTATIONAL EFFICIENCY ANALYSIS
We evaluate whether MASAM incurs additional computational overhead by measuring the average per-step inference latency training latency and the peak GPU memory usage under the best-performing configuration on the MIMIC- PhenlC-Peenotpeee prediction task, the results are reported in Table 10. MASAM achieves inference per-step latency (8.875 ms) comparable to Late Fusion (8.637 ms), indicating
25

---

Published as a conference paper at ICLR 2026
30
30
20
9.00
-.0
4.0
-0.5
0.5
-0.5
0.0
0.0
0.0
0.0
0.5
0.5
0.5
0.5
1.0
1.0
23
20 Loss Value
Loss Value
Loss Value
(a) AGM
(b) G-Blend
(c) Late Fusion
30
Loss
10
10
-..0
9.00
9.0
-0.5
-05
-0.5
-0.5
0.5
0.0
0.0
0.0
0.5
0.5
0.5
0.5
1.0
1.0
1.0
Loss Value
Loss Value
Loss Value
(d) MLA
(e) MMPareto
(f) MASAM (Ours)
Figure 13: 3D loss landscape visualization MASAM and baseline models on the ADNI dataset. MASAM converges to substantially flatter and broader region, illustrating enhanced robustness and
generalization.
0.23
3.60
2.42
0.482
64.00
0.90
0.75
6470
0.22
(a) a Sensitivity (Phenotype)
(b) Sensitivity (Phenotype)
(c) Sensitivity (Phenotype)
0.34
0.32
MASAM
0.6
666
0.6
666
05
(d) a Sensitivity (KS)
(e) 3 Sensitivity (KS)
(1) p Sensiivvity (KS)
Figure 14: Sensitivity analysis of MASAM on three key hyperparameters: a, B, and p with accuracy the metric. The first row reports results the MIMIC-Phenoth dataset, while the second row presents the corresponding results on Kinetics-Sounds (KS). All numbers represent averages across random seeds.
that method does introduce noticeable overhead at inference time. While MASAM requires a moderate amount of additional memory due to SAM-based optimization, its peak usage remains significantly lower than computationally intensive baseline DrFuse. However, method exhibits
26

---

Published as a conference paper at ICLR 2026
Table Unified comparison on additional metries. Columns report AUROC for MIMIC-Phenotype and MIMIC-Mortality, mAP f CREMA-D and KS, and Accuracy for ADNI, Underlined values
indicate the strongest baseline, while bold highlights our method. Relative Gain is computed against
the strongest baseline in each column. All results are averaged over four runs with different random
seeds.
Phenotype
Mortality
CREMA-D
KS
ADNI
Method
AUROC
AUROC
mAP
mAP
Accuracy
Late Fusion
0.748 +0.022
0.903 +00062
0.660
-0.018
0.678
+0.0100
0.749
+00017
G-Blend (Wang al., 2020)
0.752 10,0010
0.902
200000
0.717
20,0134
0.710
1000109
0.751
20,054
OGM (Peng et al., 2022)
0.748
0.906
#000005
0.769
=0.0000
0.729
#000049
0.756
#00033
AGM (Li et al., 2023)
0.746
0.906
200000
0.746
-0,0100
0.747
2000052
0.757
200111
MLA (Zhang et al., 2024)
0.748 +0.00199
0,907
#0,006
0.705
=0.0033
0.723
+0.0121
0.755
#0,058
MMPareto (Wei & Hu, 2024)
0.751 a0.0005
0,904
20.002
0.717
0.0
0.712
1:00096
0.751
2000008
PMR (Fan et al., 2023)
N.A.
0.906
20000S
0.782
00,0070
0,665
4000120
0.755
20,0100
InfoReg
0.753 +0,0008
0,904
+000055
0.792
=0.01
0.734
+000066
0.753
=0,0200
AUG
0.747 26,0015
0.901
#:00079
0.838
=0,0107
0.752
#0.m05
0.755
20,012
DrFuse (Yao et al., 2024)
0.749
+0.0003
0,,05
+0.0011
N.A.
N.A.F $
0.768
+0.0058
MedFuse (Hayat et al., 2022)
0.747
10,0021
0.895
10.0000
N.A.
N.A.F F
0.760 20013
MASAM (ours)
0.760 10,0004
0.915
100000O
0.797
00,000L
0.797
400000
0.778
20.0076
Relative Gain (%)
+0.93
+0.88
-4.89
+5.98
+1.30
PMR relies on contrastive module that is incompatible with multi-label phenotype prediction. DrFuse and MedFuse are tailored for clinical data; thus, we exclude them on CREMA-D and KS. Table 8: Comparison of alternative APS and MDPS designs on both KS and MIMIC- MIMIC-Phenotype datasets. / indicates the default design in MASAM. All results averaged over four random seeds.
Method
KS Accuracy
A Acc (%)
MIMIC-PPenotype AUPRC
A AUPRC (%)
MASAM (Default)
0.740 = 0.0084
0.00%
0.498 0.0010
0.00%
1
MASAM w/ OGM Score
0.731 = 0.0012
-1.22%
0.493 + 0.0013
-1.00%
2
MASAM w/ APS (a 0)
0.733 0.0043
-0.95%
0.489 + 0.0033
-1.81%
MASAM w/ gi
0.700 0.0064
-5.41%
0.462 0.0026
-7.23%
MASAM w/ 82
0.721 = 0.0074
-2.57%
0.489 0.0010
-1.81%
Table Ablation study of MASAM disease phenotypes classification task (AUROC). / indicates the module is enabled. Experiments numbered for easier reference in the text. The first row shows our proposed model, while rows through 4 represent different ablation experiments. All results are averaged over 4 random seeds. This table demonstrates the ablation experiment results using the AUROC metric, revealing that both our proposed APS and MDPS modules make significant contributions. Comparing MASAM with #1, #2, and #3 shows that the APS module substantially improves model performance. Similarly, comparisons between MASAM and #2, as well as between #1 and #3, demonstrate that the MDPS module enhances the model's overall effectiveness.
Method
APS
MDPS SAM AUROC vs. APS vs. w/o MDPS vs. SAM Only vs. Late Fusion
MASAM
v
0.760
+0.93%
+0.26%
+1.06%
+1 1..60%
w/o APS
X
V
v
0.753
-0.66%
+0.13%
+0.67%
2 w/o MDPS
V
X
V
0.758
+0.27%
+0.80%
+1 +1.34%
3 SAM Only
X
X
v
0.752
-0.13%
-0.79%
+0.53%
4 Late Fusion
X
X
X
0.748
-0.66%
-1.32%
-0.53%
longer average training time due the additional backpropagation required by SAM-based optimiza- tion. We plan to explore a lightweight variant of MASAM that retains its SOTA performance while reducing computational overhead.
27

---

Published as a conference paper at ICLR 2026
0.95
G-Blend
0.90
Latefusion
Audio Gradient Similarity
MAGAM MMForcto
Video Similanty
MASAM
0.85
MMFFrato
0:80
0.7
GM
0.75
0.6
0.65
0.60
0.3
10
15
30
10
30
Epoch
Epoch
(a) Audio Gradient Alignment
(b) Video Gradient Alignment
0.9
0.8
EHK Gradent Similarty
Gradlent Similartt
0.85
0.5
0.90
Latunusion
MASAN
KASAN
MMFarcto
MMFarcto
OGM
0
20
30
50
60
10
20
30
50
60
Epoch
Fpoch
(c) EHR Gradient Alignment
(d) CXR Gradient Alignment
Figure 15: Gradient alignment curves for the audio and video modalities KS and the EHR and CXR modalities in MIMIC-Phenofie quantified by the cosine similarity between each unimodal gradient and the fused objective gradient.
Table 10: Computational efficiency comparison methods. Average Training Latency is measured in milliseconds, and Memory reports the peak allocated GPU memory during inference.
Method
Average Inference Time (ms)
Average Training Time (ms)
Memory (MB)
MMPareto
8.48
120.92
2482
DrFuse
19.77
19.99
3691
AGM
13.14
30.96
2027
OGMGE
8.80
20,95
2042
MLA
9.10
41.59
1921
MedFuse
13.93
20.35
1874
Late Fusion
8.64
19.09
1933
AUG
16.93
48.17
1926
InfoReg
12.97
140.64
2152
MASAM
8.88
186.63
2373
28


Reviews:
[{"review_id": "oQk5s9vp1y", "reviewer": "Reviewer_U8Y1", "summary": "This paper proposes the MASAM method, which introduces the Sharpness Aware Minimization (SAM) strategy from machine learning into multimodal learning. By adjusting the loss function to prioritize model robustness, it enables each modality to\nexplore flatter minima along directions that align with shared information while remaining robust to heterogeneous loss landscapes, thereby improving overall performance.", "questions": "Refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "⦁\tThis paper introduces Sharpness-Aware Minimization into the multimodal domain and further designs it specifically for the multimodal context, enhancing its applicability.", "weaknesses": "⦁\tThe baselines are weak. Only 1 work published in 2024 is compared and no works published in 2025 are compared. This comparision is not enough to show its SOTA performance.\n⦁\t The writing is confusing. Fig 1b is not mentioned in the manuscript. Fig 2 is used until in page 6.\n⦁\tTha ablation study should be performed on more datasets. Also, APS and MDPS should be compared to other choices to verift their advantages. \n⦁\tThe experimental section primarily focuses on interpreting dataset performance while neglecting an in-depth analysis of the method’s effectiveness.\n⦁\tThe authors argue that MDPS aligns gradients between unimodal and fusion objectives, please give detailed verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q52FqtcS4Q", "reviewer": "Reviewer_TTzL", "summary": "This paper presents MASAM (Multimodal Adaptive Sharpness-Aware Minimization) to address the challenges of modality imbalance and heterogeneous loss geometries in multimodal learning. The authors introduce Adaptive Perturbation Score (APS) and Modality-Decoupled Perturbation Scaling (MDPS) to ensure balanced multimodal learning while benefiting from Sharpness-Aware Minimization (SAM). Their approach is demonstrated to achieve flatter loss landscapes for each modality, resulting in improved generalization across diverse datasets and tasks.", "questions": "Please check the above section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents both qualitative and quantitative experiments across five multimodal datasets, with six downstream tasks, demonstrating MASAM's effectiveness in diverse domains, including clinical, video, and image-text datasets.\n- The problem of modality imbalance is well-motivated, and the proposed solution is thoroughly explained. The use of SAM in multimodal learning is a novel approach to addressing this problem.\n- The writing is clear, logical, and easy to follow.", "weaknesses": "- The loss landscape analysis is only conducted on the small clinical MIMIC dataset. It would be more convincing if similar visualizations were provided for larger and more diverse \"in-the-wild\" datasets, ensuring the scalability and generalizability of the approach across various real-world data settings.\n- The sensitivity of hyperparameters, such as $\\alpha$ and $\\rho$, is not sufficiently explored. A more detailed analysis of how these parameters vary across different datasets and how they affect the method performance is necessary.\n- The paper focuses on experiments with two modalities (e.g., EHR + CXR, audio-video). An exploration of how MASAM performs with more than two modalities would be valuable for assessing its scalability in more complex multimodal fusion tasks.\n\nMinor Issues:\n\nThe used \"Kinetics\" dataset should be called the \"Kinetics-Sounds\" dataset. Kinetics-Sounds dataset is a subset of Kinetics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VgA0G2TJ5C", "reviewer": "Reviewer_EFgG", "summary": "In this paper, the authors presented MASAM, a novel method that applies sharpness-aware minimization (SAM) to multimodal setting. The paper identifies the issue with directly applying SAM to multimodal setting where modality imbalance could be exacerbated by SAM, and devised a method to decouple the modalities when determining the scales of the perturbations. The proposed method, MASAM, first determines the dominant modality using APS, then applies perturbation on the dominant modality using MDPS. The proposed method was evaluated over 6 tasks across 5 very different datasets from different domains against a large set of baseline fusion methods, and MASAM achieves top performance across all tasks. Additional analysis was conducted to highlight MASAM's robustness to label noise and ability to allow all modalities to converge jointly to a flatter minima, and ablation studies was conducted to demonstrate the necessity for both APS and MDPS modules.", "questions": "I am a bit confused about the setup for calculating Decay. Can you explain why it is calculated as the difference between the previous round's absolute loss and the current round's moving average? Also, in Figure 10b, it seems like a high $\\beta$ works better, but wouldn't a higher $\\beta$ make the Decay smaller and more likely to be zero?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method outperforms many baselines over 5 very different domains, demonstrating MASAM's generalizability across many multimodal application domains.\n\n2. The overall presentation quality of the paper is good. It is overall easy to follow.\n\n3. There is theoretical analysis to show that naive SAM does not work well under multimodal settings, and that MASAM can better converge all modalities.\n\n4. There is additional visualizations and empirical analysis to demonstrate MASAM's more balanced learning across modalities and allows all modalities to converge to a flatter minimum. The additional analysis also shows MASAM's better robustness to label noise.\n\n5. There is ablation study to demonstrate the necessity of both APS and MDPS modules.", "weaknesses": "The paper could be improved by including more justification/analysis for the necessity of both Decay and $\\gamma$ in APS. In figure 10a, it seems like low $\\alpha$ generally performs quite well, so is Decay actually necessary? There needs to be an ablation study to justify this (i.e. what happens if you set $\\alpha$ to 0?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pjw8oW0FqD", "reviewer": "Reviewer_A3G6", "summary": "This paper introduces MASAM, a multimodal optimization framework that adapts Sharpness-Aware Minimization (SAM) to address modality imbalance in multimodal learning. The authors identify two main issues when applying SAM to multimodal settings—amplified modality imbalance and modality-agnostic perturbation—and propose two components: the Adaptive Perturbation Score (APS) and Modality-Decoupled Perturbation Scaling (MDPS). APS dynamically identifies dominant modalities based on convergence speed and gradient alignment, while MDPS decouples perturbations to reduce inter-modal interference. The method demonstrates improved performance across multiple multimodal datasets, achieving flatter minima and better balance among modalities.", "questions": "1. The SAM technique seems to require multiple passes of gradient computation, which raises concerns about the efficiency of the proposed method.\n\nI would consider raise my rating if the above concerns could be sufficiently addressed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents an interesting and well-motivated attempt to extend SAM to multimodal learning, where convergence dynamics differ across modalities.\n2. The introduction of APS and MDPS is intuitive and addresses the challenge of applying sharpness-aware optimization in multimodal settings.\n3. The experimental results are extensive and consistent, showing that MASAM achieves improved generalization and balanced performance across diverse datasets.\n4. The visualizations are impressive.", "weaknesses": "1. Lines 90–92 state that “SAM’s uniform sharpness minimization thus disproportionately benefits dominant modalities with faster convergence.” However, in the proposed MASAM framework, SAM is applied only to dominant modality-specific encoders. This design choice appears contradictory to the stated motivation—if SAM inherently benefits dominant modalities, applying it exclusively to them seems counterintuitive. The logic connecting the problem statement and the solution needs clearer justification.\n2. The introduction of unimodal auxiliary objectives and the APS mechanism represents another gradient-based modulation approach, which has been explored in existing works such as MMPareto, OGM, and related gradient alignment strategies. The paper would benefit from a clearer differentiation from these prior methods, especially in how APS or MDPS uniquely contributes beyond existing gradient-based optimization schemes.\n3. In the discussion section (Line 422), the authors claim that “MASAM enables all modalities to converge jointly to flatter minima.” However, if a modality remains consistently weak (non-dominant) throughout training, no perturbation is applied to it. This raises doubts about whether MASAM truly ensures joint convergence across all modalities or only benefits dominant ones. A more detailed discussion of this limitation is warranted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)