Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning
Authors: Hossein R. Nowdeh
Jie Ji
Xiaolong Ma
Fatemeh Afghah
Venue: NeurIPS 2025 poster ()
Keywords: Sharpness aware minimization
multimodal learning
VLM
Primary area: optimization
TL;DR: In this paper we resolved dominant modality effect by focusing on loss land scape and gradient modulation

Abstract: In multimodal learning, dominant modalities often overshadow others, limiting generalization. We propose Modality-Aware Sharpness-Aware Minimization (M-SAM), a model-agnostic framework that applies to many modalities and supports early and late fusion scenarios. In every iteration, M-SAM in three steps optimizes learning. \textbf{First, it identifies the dominant modality} based on modalities' contribution in the accuracy using Shapley. \textbf{Second, it decomposes the loss landscape}, or in another language, it modulates the loss to prioritize the robustness of the model in favor of the dominant modality, and \textbf{third, M-SAM updates the weights} by backpropagation of modulated gradients. This ensures robust learning for the dominant modality while enhancing contributions from others, allowing the model to explore and exploit complementary features that strengthen overall performance. Extensive experiments on four diverse datasets show that M-SAM outperforms the latest state-of-the-art optimization and gradient manipulation methods and significantly balances and improves multimodal learning. The code will be released.

Modality-Aware SAM:
Driven Gradient Modulation for Harmonized Multimodal Learning
Hossein Rajoli Holcombe Department ECE Clemson University hrajoli@clemson. edu
Jie Ji Holcombe Department of ECE Clemson University jji@clemson. edu
Xiaolong Ma Department of ECE University of Arizona xiaolongma@ar www.aaaaaaaiiaaa....m..mmm edu
Fatemeh Afghah
Holcombe Department of ECE Clemson University faf Eafghah@@lemsonn. edu
Abstract
In multimodal learning, dominant modalities often overshadow others, limiting generalization. We propose Modality-Aware Sharpness-Awen Aware Minimization (M- SAM), model-agnostic framework that applies to many modalities and supports early and late fusion scenarios. In every iteration, M-SAM in three steps optimizes learning. First, it identifies the dominant modality based on modalities' contri- bution in the accuracy using Shapley. Second, it decomposes the loss landscape, or in another language, it modulates the loss to prioritize the robustness of the model in favor of the dominant modality, and third, M-SAM updates the weights by backpropagation of modulated gradients. This ensures robust learning for the dominant modality while enhancing contributions from others, allowing the model to explore and exploit complementary features that strengthen overall performance. Extensive experiments on four diverse datasets show that M-SAM outperforms the latest state-of-the-art optimization and gradient manipulation methods and significantly balances and improves multimodal learning.
Introduction
Multimodal learning 27 has become the cornerstone of emerging deep neural models. These ad- vanced models integrate diverse data types, including text, audio, images, and sensor data, mimicking the multiplicity of sensory information that humans experience.
This capability for integration crucial for range of complex tasks. This integration capa- bility crucial for a range of complex tasks, enabling models to fully grasp context through multiple viewpoints, essential in wide spec- trum of applications such autonomous driv- ing, human-computer interaction, and intelligent virtual agents. By blending various sources of information. these multi-modal approaches can construct richer and more robust representations, potentially leading to improved performance across various domains. However, achieving the ideal balance among modalities presents
(a) AGM
(b) SAM
(c)
Figure 1: Loss landscape visualization of CREMA-D (late fusion) for AGM, SAM, and M-SAM from two different viewpoints.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Vealt de-esan thes
roda ACWISS Fasle)
Epedus
Epodts
Epedts
(a) Audio
(b) Visual
(c)
Multi-modal
Figure 2: The normalized overfitting gap (T) comparisons of joint- hiint-trained. OGM-GE, AGM, SAM, and our proposed M-SAM on the AV-MNIST dataset. Viewing this in color recommended.
unique challenge, as one modality can dominate the learning process, limiting the model's ability to develop unique discriminative capabilities from less dominant data sources. In practice. the imbalanced modality can lead to unexpected results, where the performance of jointly trained models shows minimal improvement or even declines compared to unimodal models. 4 [33. Some studies have found that different modalities often converge at varying speeds, a phenomenon known as uncoordinated convergence 23
28
16]. 30
34
Modality encoders sometimes overfit due to the nature of the input data and the underlying task, especially when overexposed to their training data. [23] 28 6]. This overexposure causes the model to adhere too closely to the noise and unique characteristies of the training data, resulting in poor generalization to new, unseen data. Similarly, the shared network, which integrates features from all modalities, can be overfitted to the combined feature space, reducing its ability to generalize effectively. This uncoordinated convergence can significantly hinder the performance of multi-modal models.
While many researchers 28 argue that multi-modal networks fail to fully utilize individual modalities if their performance drops in mono-modality evaluations compared to single- ingle-modality training, such claims may overlook key aspects of how these networks are structured. For instance, late fusion techniques 41 38 [), where modalities are kept separate until the final layers and allow each modality's ' impact to be more easily assessed. In contrast, early fusion merges modalities from the start and process them together. In these cases, the contributions of individual modalities become intertwined, making their individual effects less distinguishable but not necessarily less significant. The fact that a modality does not perform well individually within complex multi-mode system does not mean it is not providing valuable information. Evaluating these networks requires a more profound look rather than a straightforward comparison of isolated performances.
studies, which utilize gradient modulation techniques, tailor backpropagation for each modality, aiming to find minima that are perfectly suited to each modality. While this approach does outperform previous modulation- modalities and overall, huge gap shows it tends to overfit (18). leading to sharper minima [12] that constrain the model's ability to explore shared and complementary high semantic cues across modalities. The degree of overfitting is evaluated using the normalized overfitting gap, T, measuring the discrepancy between model's training and test performance. It defined as:
Recent SOTA 23 28
works by achieving better accuracy on both
AcCtrain AcCtest
T
Accrest
with 0 <. r S 1, and assuming models are not underfitted, values closer to 0 signify robust gener- alization, whereas values nearing indicate pronounced overfitting, Such a normalized overfitting gap has been established as a proxy for sharpness and generalization ability [T8 7. and it clearly illustrates the advantage of our method in learning flatter, generalizable minima. Figure 2 shows that throughout training, our M-SAM method maintains close match between training and validation accuracy, indicating a well-balanced learning process. Figure [] further highlights differences in curvature by visualizing the loss landscapes. The relationship between flatter, wider minima and reduced overfitting gap is clearly reflected in M-SAM. By relaxing the strict perturbation constraint used in traditional SAM, M-SAM finds broader, flatter minima. This outcome aligns with the findings of FriendIy-SAM 24 that argues traditional SAM not necessarily gets to the flattest minima.

---

In our paper, we present Modality-Awaad SAM (M-SAM), a method designed to boost network generalization by giving precedence to the dominant modality. By steering the learning process towards flatter minima that favor the dominant modality, M-SAM enhances its resilience against parameter updates from other modalities during backpropagation. Additionally, this approach allows non-dominant modalities the freedom to explore and search for optimal features aligned with overall performance. Unlike recent studies that have only slightly improved encoder performance through narrow-focused gradient modulation techniques, M-SAM actively searches for broader, flatter areas in the loss landscape that benefit all modalities and the shared network. This approach prevents system from becoming overly fine-tuned to the particular characteristics of the training data. The proposed method also dynamically adjusts the learning process by continuously evaluating and rebalancing the contribution of each modality in favor of the dominant modality's 's generalization in each iteration that not only enhances the weaker modalities to search for their optima but also ensures more balanced, robust learning across the board. As main contributions, we show that:
Our method is conceptually aligned with findings that dominant modalities can emerge during multi-modal training due to dynamics such as modality competition [15]. We propose a practical optimization- based solution that dynamically adapts to such imbalances during training. Specifically, we introduce Modality- Howae Aware SAM (M-SAM), which detects and prioritizes the dominant modality at each iteration and mini-batch, ensuring its robustness against noisy gradient contributions from weaker modalities and promoting more stable convergence.
M-SAM with a modality-focused generalization capability enhances the resilience and stability of the dominant modality against weight updates from other modalities while simultaneously preventing underfitting phenomena in non-dominant modalities.
Using the architecture proposed by (23 as a baseline, we evaluated M-SAM across diverse datasets and scenarios, including early and late fusion. Our experiments demonstrate state-of-the-an methods in
consistent and substantial performance improvements optimizer design, highlighting the versatility and effectiveness of M-SAM.
2 Related works and backgrounds
2.1 Multi-modal learning
Multi-modal learning is a rapidly growing field in research, aiming to efficiently process data from multiple senses for practical applications. It spans various domains, including multi-modal recognition 37 and understanding audio-visual scenes (3). Researchers have highlighted the advantages of
multi-modal learning over uni-modal approaches 14 while others have delved into the challenges
and failures associated with this type of learning 15 Despite efforts to enhance performance with
more information, studies (5 30 34 36) have shown that many multi-modal learning methods struggle due to discrepancies between modalities. To address these challenges, researchers have proposed innovative approaches such as OGM-GE (28). which adaptively controls optimization by monitoring modality contributions. G- G-Blending while MSES [8] detects overfitting and performs early stopping. MSLR (38] effectively builds late-
34 computes optimal blending based on modality behaviors,
fusion multi-modal models, and AGM
23 boosts performance with adaptive gradient modulation
and fusion strategies. Kontras et al. 20
considered uni-modal loss values beside multimodal--radienn
to modulate encoder gradients, and 335 customized Pareto approach for multimodal scenarios. These
advancements aim to improve the effectiveness of multi-modal learning models.
2.2 SAM
Sharpness- Aware Minimization (SAM) was first proposed by a machine learning technique to enhance model generalization. It achieves this by concurrently minimizing the loss value and the sharpness of the loss function, aiming for flatter minima. SAM has demonstrated efficacy beyond deep neural networks, finding applications in diverse fields such as language models M and fluid dynamics [17), illustrating its adaptability. Adaptive SAM (ASAM) [2 improves SAM by dynamically adjusting the sharpness radius, addressing the parameter scale-dependency issue. Minimization (GSAM) [44) introduces the surrogate gap, simplifying sharpness measurement in local minima. Fisher SAM [9 enhances SAM by using ellipsoids from Fisher information, refining neighborhood structures for better sharpness estimation.
Surrogate Gap Guided Sharpness-Aue
GAM 40 presents first-order flatness focusing the maximal gradient norm within perturbation
3

---

radius which bounds both the maximal eigenvalue of Hessian at local minima and the regularization
function of SAM. SAM finds applications in various leaanigg tasks 26 I 39.. Chen et
al. (3 use SAM on Vision Transformer and
to enhance accuracy and robustness. Na et
al. [26 note SAM's role in parameter compressibility and task transfer. Yeo et al. [39] apply SAM
to medical imaging, improving segmentation while balancing sharpness and warp regularization. However, SAM has not been used in multi-modal learning so far.
3 Multi-modality learning and SAM
3.1 Notation and Problem Definition
In tasks where we deal with datasets composed of multiple modalities, the training set D U (r'm. often includes a variety of data types, where m E [I...., .] indicates the modality to which the data sample (w v/) belongs. These modalities could represent different sources or types of data, like images, audio, texts, and sensor readings. To address the imbalance gradients across these modalities, partition the whole set as D U (Xm Y), which Xm and Y contain data from math modality and it's corresponding label and represents training samples population.
3.2 Preliminaries
Our objective is train system that accurately acquire and process data from all modalities, ensuring that each contributes to the back-propagation considering the optimization stage, despite their differences in representation. From mth modality's perspective, the system, modeled by function f-m(;;mm parameterized by Om, aims to predict the correct label g' for given sample r'. The loss function, among other variants, acts the guide E(j², yy") fine-tune the parameters On during training. The training task would be formulated as:
N
0- = min
C (JT(( ... :::: 31 f5 ... ..::::r;i 02) ... D.M (ār ... FM: (m)) (M))
(2)
where 0 denotes an arbitrary fusion operation, such as early, late, or any hybrid fusion methods and fmh represents the encoder associated with mth modality. 0" is the optimal parameter set we seek, defined such that (0, C L S I M; 0g no; + Ø}. Note that, Om represents a portion of architecture parameters including neural connections, activation function, etc. that carry out the effect of r to the output. Being more precise, Om: : = {0=, U k (1, M) J. m + k), where 8r represents the modality-speciffi encoder architecture parameters of mtth modality. & defines the model's 's parameters associated with the fusion network that fuses modality kth to modality mth. The shared network f and its parameters 4 in f", 8*) influenced by forward/backward propagation of various modalities. Considering these definitions, Eq. 2 without loss of generalization can be comprehended follows; N
0" min Σ ((.(
min Σ (e.................
(3)
where f and 0 in S(,,0) respectively are the model and its parameters. On the other hand, 6n f (ri, Fm ... Thi 0r Os ......., 0" lmyk, 0=M)) is defined as the feature representation produced by the m-th encoder taking into account other modalities" effect through the fusion network's 's parameters, 0mys.
Considering Eq. 3 the loss function of the multi-modal underlying task can be expressed as the The intricate interplay among modalities during model training, coupled with the complexity of deep neural network architectures, poses significant challenges in isolating the individual contributions of each modality to overall performance. Utilizing the Shapley approach as outlined by [29], offers a robust mechanism to decompose the contributions of each modality to the output loss on a per mini-batch basis (Appendix.) B
sum of the losses from all modalities, as L + ا (() $2,..., ( (ar; 0*), ("),,²)) y²).
= (vi + 22 ... tw) = Lı + La + + LM;
where Um represents contribution of the mith modality in the output. Note that Σ M Um 1.

---

3.3 Learning Dynamics and Loss Landscape
Let = (01. 92 ........ Pm..... Par; 8*) denote the combined feature representation that integrates the contributions from all modalities, and #* represents shared weights associated to the downstream task. Given that, the update rules for the parameters are as follows:
0 Oiieets)
1) o [(fmm))))
0 Ommn)
"1
Σ N
1
@L((,vV)
Oub
(5)
Ov
08
i=1
@a((b, Oup dam
w²)
Ov 0bm
0 (
0e
V ((0 mi:e)
=0 0 m.())
"N
00 m
where is the learning rate, L is the loss function, and /. indexes the iteration of the gradient
descent. In Eq. 5 0s
Øø represents the component where inter-modality interaction occurs. During backpropagation, this term adjusts the shared network weights, 0", based on the gradients contributed by all modalities. Considering the decomposition of the loss landscape described in Eq. Fand depicted Fig. as training progresses, the dominant modality, mgg is guided toward a wide, flat minimum in the loss landscape, this ensures that even when inter-modality interactions slightly deviate the networ parameters, 8 and move the dominant modality from its optimal minima, remains within the bounds of wide, stable minimum. Consequently, weight updates in the shared parameter space, 0%, may shift the network parameters slightly. Still, the dominant modality's contribution to the overall loss remains relatively small compared to the case where it converges to sharp minimum.
3.4 SAM
SAM an optimization technique that improves generalization by finding flatter minima by mini- mizing the maximum loss in neighborhood around the parameters. Specifically, consider family of models parameterized by E W C RR:; the loss function over the training dataset SAM aims to minimize the following upper bound of the Probably Approximately Correct (PAC)-Bayesian generalization error for any > O,
(( S max L(B + c) +
2 |1013
To solve the above minimax problem, at each iteration 6, SAM updates
{=sig((2((0--)))1e (V100-1)))VV
(IV
(8)
0. 0 T1 (VL (0, + ()) A0:-1),
Here 1/p-1/9 1.p 0 is hyperparameter,
0 is the parameter for weight decay, and m) the learning rate. By setting p = q = and introducing an intermediate variable u, we have:
0: + PVL (0,-
IVC (0.-1)ll 0.-1 EE,
(9)
0, = 0:-1 (VL (u)) + +00=11).
(10)
3.5 Modality-Aware SAM
To enhance the overall performance of models on multimodal datasets, the SAM optimization process can be adapted by incorporating split loss function (Fq.. 4.. This reformulation targets the dominant modality, making it more robust against changes in network parameters due backpropagation from other modalities. While SAM's modality-agnoste nature originally confines its generalization capability, this new approach ensures that the proposed M-SAM provides strong generalization power in the context of diverse and complex multimodal data. Utilizing Eq 7 and Eq 4, we can reformulate SAM follows:
min max [0 + +c) .... L + c) 1011. Se
(11)
In this regard, the perturbation defined in Fq. can also be divided into components associated with each modality e + ..... + eM where mg is the dominant modality. When
5

---

Algorithm M-SAM Algorithm
Require:
Training dataset = U (U ( y')}}, neural network f()) with parameters 6,
loss function L, mini-batch size b, learning 7), neighborhood size p. weight decay coefficient,,
Ensure: Trained parameters 0*
1: Initialize parameters On.
2: while not converged do
3: Sample minibatch B ".ci.....hhhl ,..., ((0) (of)...... 22))
4: [(O+) Em (( (t), 1²) = Exs U *mf(f(a]
Mi
(x.00),'' y')
Eq.
5:
La(Ot) vaL(f
M 0 01), y¹) ma arg mEft.....d
Um
6:
(0+) Σ Um P(n Fhr 0z), v*) m' EM {m E (I,..., M3, ma)
7:
VLa(O:) Backward La. f()), (O.) Backward(La, f())
8:
& P
(L.(fe)
9:
0, 8,- Th Laa + + (C.) 1
10:
1+1+1
11: end while
DF shan
v. bn
Fr fiee
of uc
rp
Au (K Ae)
La (Nay Y; Cogg 0*)
0 mm
6
my
(a)
Grodient comporinna tooenedeess persmnetree, Fyc, and Feg)
Modalltiee gractents camskdering intoruction modulis
The shared perameter spece's Crodient componentt aummation, ((Fyrla, (gif.g))
updats sajectory
Grddect componsens ttouthrwd networkcs parsmettrs, Far)
Modalltia gracienns aspumingg.wm intorcttion betwoon modalides.
decompesed
Sharpiftec
Landbocape
Figure 3: Illustration of the interaction of gradients in sharp and flat loss landscapes (before and after convergence of MSAM). In (a), the sharp landscape amplifies the dominant modality mo's 's gradient magnitude, often overshadowing the non-dominant modality due to large and differing gradient directions the shared parameter space. In (b), settling the dominant modality m. in flatter minimum reduces its gradient magnitude across a wide range of parameter neighborhoods. allowing the non-dominant modality explore the parameter space more freely and promoting balanced optimization across modalities. Note that the loss landscape decomposition is achieved using the Shapley method by dividing total loss modality-speciff loss in every iteration.
the dominant modality approaches settles on minimum, its contribution to the loss function during perturbations is significantly greater than that of other modalities. Therefore, making this dominant modality robust against perturbations from other modalities allows the latter search for their minima without substantially affecting the dominant modality or the overall performance of the model. However, there is no guarantee that eiT >> &., Vm E (1,2.... AM m # mo)- As a result, the traditional SAM can not provide focused generalization in favor of the dom- inant modality. Inspired by 42] by ignoring the SAM optimization term of non-dominant modalities in Eq. [1] our proposed modality aware SAM optimization target is achieved as Optimization term for the dominant modality
optimization non-dominant modalities
ming
max Lym (0 + Emo) Lymo (0) Sp
+ Lmo (0) (L,(0) Lz(0) + L (0) + 1012.
The gradient update equation, Eq. 10 is modified as follows:

---

0, 01011 17t L (0-_1 + VL1 (0c-1)
L22 (01-1) +... VLM (0x-1) + 20--1).
(12)
oV Leni (02-1)
where &t" calculated as c,"
V (9) 1>
As illustrated Eq. 12 the proposed awwal SAM bypasses sharpness-aware minimization for non-dominant inputs, which are less prone to overfitting due to being overshadowed by the dominant modality. We adopted the Shapely value proposed by [3], elaborated in Appendix, B split the loss into individual modalities' loss. Algorithm, SGD as the base gradient optimizer.
Iegoritmm.Joutineeee outlines the complete M-SAM utilizing
3.6 Stability and Convergence Analysis M-SAM
Following the analytical framework in 22 we establish the convergence behavior of M-SAM under its dynamic update rule and modality-aware selection mechanism. For analytical tractability and without loss of generality, make the following assumptions:
M
1. [(0.) = Σ Um Cm (0+) and its gradient is bounded K-smooth (K (K-Lipschitz), i.e. m=l -
K 00 - 0.Il, vo,, 0. that [|V[())) S Gmux-
Pg for analytical tractability. In theory, 2. Learning Te vi and perturbation Pr Vi
this smooth decay schedule facilitates convergence analysis. In practice, we adopt a step- wise decay, by multiplying the learning rate by 0.1 every 70 iterations, which decays Y even faster. This empirical schedule still satisfies the convergence conditions and does not compromise the theoretical guarantees.
considering d, = 1 - 8, = -- (((,)) and = 02 + Pt mo [(0.)] where Vi means gradient toward the dominant modlaity. Using descent Lemma for k-smooth function write:
Wm C(Os)
-(0++))-(00 < -m((V(()) VC((.)) Kn
|VL(0.11
- (.VV - V2(00) + VL((,)) Kn
[VL(0)]
-m m (V VC(O. VL((,) + (IVC(E))) S
(13)
2
-mIIV2 (VC))) VC((,) VC(())) GK man S
-m] - w [0) Knpp G2 max
VL(0,)I + Kn <
Although the upper-bound term appears to capture potential instability introduced by dynamic modality switching, it can be shown that the bound is function of P: alone. Hence, the convergence rate M-SAM remains invariant to the gradient of the selected modality C(fe), ensuring that modality selection does not affect stability.
by rearranging the inequality, it would be as:
nhe C( < C(O.) [(0r+1) KTPP Gmaz + KT 22 mma
(14)
now do summation over all iterations, 1 t Using telescope sum properties for loss terms on the right side of the inequality and this property, mo Σr VL())))1 < Σ VC(.))
T VL(00))1 S (.,),) + Knopi Gmaz
K
G2
(15)
ti
m.ze

---

Considering that Σt= I 1 log and Σ 277 forms a p-series with p 3 > 1. whose partial sum remains bounded, the only term on the right-hand side influencing the asymptotic behavior the harmonic component. Therefore, the overall convergence rate of M-SAM is O
log T VT'
T
log
VC(O011 SO
That means, As T' increases, the average gradient decreases at rate get for methods like SGD and traditional SAM.
log T vT
This similar what we
Experiments and Discussion
4.1
Dataset
M-SAM is evaluated on three popular multi-modal datasets: AV-MNIST (33], CREMA-D 2 UR-Funny [T and AVE [3]. Details of these datasets are presented Appendix,
4.2 Networks' architecture and preprocessing
To evaluate our approach's 's performance in both late and early fusion strategies across various multi- modal datasets, we followed unified model design based on prior works by 223 25). For early fusion, we used the MAXOUT network [9] for the fusion module. Consistent encoder architectures were maintained for each dataset in both fusion strategies. Specifically, ResNet18 was used the encoder for the audio and visual modalities in the AV-MNIST and CREMA-D datasets. In contrast, the UR-Funny dataset utilized Transformer [32] encoder for all three modalities. To ensure consistency with previous works, we followed the preprocessing steps they applied. For CREMA-D. DD, we extracted 1 frame per minute (fpm) from each clip and processed the audio data into a spectrogram of size 257 x 299 with a window length of 512 and an overlap of 353. We used SGD with 0.9 momentum and 10-1 weight decay as the optimizer. The learning rate was initially set to 10-3 3 and was multiplied by 0.1 every 70 epochs. For UR-Funny, we utilize the preprocessed data introduced by [25]
4.3 Results and discussion
To show the effectiveness of M-SAM, we compared it with mainstream multi-modal optimizer enhancement approaches: MSES g MSLR (38). OGM-GE (2). AGM (23. MM-Pareto [35]
CGGM [IO), and Recon-Boost [3]. Our findings reveal that M-SAM consistently outperforms all
other methods in overall performance and mono-modal accuracy across most scenarios. However, believe its final performance metrics do not solely determine method's efficacy; the trends observed throughout the training epochs also provide valuable insights. Therefore, analyzed the accuracy of the validation set over these epochs. These accuracy curves highlight how the method handles various modalities, which sometimes conffict during training. The smooth and steady progression of M-SAM's overall accuracy curves demonstrates its robustness managing the complexities of multi-modal learning and its ability to harmonize different modalities effectively.
4.3.1 Early Fusion Setup
In early fusion, Table. 2 modality-specific representations are combined before classification, which causes their gradients to interact directly in the shared feature space. Joint-Train performs poorly in this setup. particularly on CREMA-D and UR-Funny, where modality imbalance is more severe. AGM and MM-Pareto provide modest improvements, with MM-Pareto benefiting from its multi- objective loss formulation. CGGM and Recon-Booss also perform competitively. CGGM explicitly supports early fusion by computing modality-speecfff gradients through separate forward passes, even when using shared classifier. It modulates training based on the difference between each modality's 's gradient and the joint gradient, without requiring modality-specific heads. Recon-Boost, while evaluated primarily with decision-level fusion, still offers competitive performance under early fusion. Its alternating update strategy and reconcilement loss help manage imbalance, though its training dynamics are better suited to architectures with modality -specif learners.

---

Table Accuracy (Ace.), and single-modal aceuracy (Acca, Accy, Acci) on the AV-MNIST, CREMA-D, UR-Funny, and AVE datasets using late fusion architecture. Please note that the OGM-GE method could not extend to more than two modality cases their original shape.
MNIST 33
CREMA-D 2
UR UR-Funmy [I]]
AVE 3I
Model
Acc,
Accy
Actim
Area
Adey
Arcmm
Acea
Accy
Acez
Accmmm
Acea
Accy
Aommm
Single-audio
39.61
52.12
59.23
65.43
Single video
65.14
60.37
53.16
64.58
Single-text
63:46
Joins-Train
14.59
62.85
68.41
56.08
44.32
61.19
50.31
53.51
49.78
64.50
59.10
63.72
73.19
MSES
63 34
64.13
49
64.23
76.47
MSLR 55
22.72
62.92
70.62
55.75
47.84
62.93
53.14
53.59
16.93
65.52
70.39
69:41
75.22
25
55.85
58.90
67.91
AGM
23
38.90
63.65
72.14
56.35
54,12
64.72
54.87
49.36
62.22
65.97
70.68
72.34
77.11
Parcoo
64.31
73.22
56.94
52.12
70.31
73.88
77.68
CGGM
[00
39.53
64.13
73.42
57.14
55.07
67.03
55.21
49.83
62.74
67.43
70.91
73.17
77.83
Recon-Hood
40.12
64.18
73.59
57.08
54.83
67.47
55.13
50.08
62.88
67.61
71.13
73.48
78,02
SAM
36.31
64.67
73.17
60.69
51.43
66.32
53.67
51.37
61,48
65.95
66.13
72.83
77.66
M-SAM
41.93
64.97
74,08
62.78
53.22
68.56
51.56
52.67
60.17
68.31
68.27
72.57
79.67
Table 2: Accuracy (Acc.), and single-modal accuracy (Acca Acc,, Acci) on the AV-MNIST. CREMA-D, UR- UR-Funny, and AVE datasets using early fusion architecture.
Model
AV-MNIST
33
CREMA-D 2
JR-FFnny
[T
AVE
3I
Acc,
Accy
Actmm
Acc.
Acce
Accmm
Acc,
Acc,
Ace
Ace
Acca
Accy
Actimm
Joint-Train
24.28
60.14
71.15
55.31
51.72
62.13
54,87
50.86
54.14
65.15
67.40
71.85
76.29
AGM 23
47.79
68.48
72.26
51.42
47.54
64.09
64.88
$5.20
63.36
66.07
68.85
72.46
77.08
MM Pareto
39.82
66.15
72.74
56.90
52.83
65.30
58.32
53.08
60.45
65.92
68.52
72.18
76.83
CGGM
100
41.30
67:27
73.11
57.84
53.67
66.90
60.42
53.40
62.53
66.98
69.01
72.81
77.32
Bocon-Boostt
54.01
66.74
66.83
68.91
SAM
38.56
61.81
73.22
56.35
53.52
66.22
$4.08
49.77
63.86
66.87
68.37
72.02
76.76
M-SAM
67.72
53.71
68.43
70.11
72.46
M-SAM significantly outperforms all baselines across datasets in Accmms with gains of +1.4 % to +2.3 % over the closest competitors, CGGM and Recon-Boost, These improvements are attributable to its optimizer design: M-SAM inherits the generalization benefits of SAM while explicitly prioritizing stable training of the dominant modality. Unlike prior methods that rely static loss weighting or sequential modality updates, M-SAM adjusts its behavior on per-iteration basis, mitigating gradient interference without requiring architecture changes.
4.3.2 Late Fusion Setup
Late fusion provides a strong evaluation setting for CGGM and Recon-Boost. As demonstrated in Table. [ CGGM leverages classifier gradiettt discrepancy to modulate updates, while Recon-Boost alternates training across modalities and applies a reconcilement loss to improve coordination. Both methods consistently outperform AGM and MM-Pareto in Acemm across most datasets. However, M-SAM achieves the highest performance overall. Unlike prior methods that rely on architectural separation sequential training, M-SAM operates entirely the optimizer level. Its intrinsic design encourages consistently flatter convergence across training iterations, preserving the optimization trajectory the dominant modality while allowing non-dominant modalities greater freedom to adapt. This balance emerges not from hand-tuned loss weights but from the geometry of the update itself. Notably, the lower single- modality accuracies of M-SAM, despite significantly higher Accmm. suggest that it encourages the learning of features that are not independently discriminative but become informative through cross-modal interaction, enabling more effective integration of complementary modality features.
5 Conclusion
In this paper, we introduced M-SAM, an optimizer-level framework that extends Sharpness-Awart Minimization (SAM) to address optimization instability in multi-modal learning. M-SAM identifies the dominant modality per batch and aligns the sharpness-aware update direction with its gradient, enabling stable convergence while preserving learning flexibility for other modalities. This mechanism promotes flatter optima without requiring architectural changes explicit loss weighting. As

---

illustrated in Figure 1 M-SAM produces a consistently flatter loss landscape compared to baseline methods such as AGM, improving both generalization and robustmess. Empirical results across four multi-modal benchmarks demonstrate that M-SAM outperforms a range of strong baselines, from gradient modulation methods such as AGM and CGGM to stage-wise training strategies like Recon- Boost. These findings highlight the effectiveness of optimization-awan strategies in harmonizing modality contributions and improving training stability in multi- settings.
Acknowledgments and Disclosure of Funding
This material is based upon work supported by the National Aeronautics and Space Administration (NASA) under award number 8ONSSC23K1399 the National Science Foundation under Grant Number CNS-2232048, and CCF-2553684.
References
[1] Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness generalization, arXiv preprint arXiv:2110.085 2021.
Sharpness-aware aware minimization improves language model
[2] Houwei Cao, David Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-soured emotional multimodal actors dataset. IEEE transactions on affective computing. 3(4):377-390, 2014.
[3] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training strong data augmentations. ICLR, 2022.
[4] Jean-Benoit Delbrouck, Noé Tits, Mathilde Brousmiche, and Stéphane Dupont. A rranformer-basee joint-cncoding for emotion recogmition and sentiment analysis. arXiv preprint arXiv:2006. /5955, 2020.
[5] Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Improving multi modal learning with uni-modal teachers. arXiv preprint arXiv:2106. 11059, 2021.
[6] Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 00029-20038, 2023.
[7] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021.
[8] Naotsuna Fujimori, Rei Endo, Yoshihiko Kawai, and Takahiro Mochizuki. control for multimodal classification. In Pattern Recognition: 5th Asian Conference, ACPR 2019, Auckland, New Zealand, November 26-29, 2019, Revised Selected Papers, Part II 5, pages 412-422. Springer, 2020.
learning rate
[9] Ian Goodfellow, David Warde-Farley, Mchdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. Proceedings of the 30th International Conference on Machine Learning, pages 1319-1327 2013.
[10] Zirun Guo, Tao Jin, Jingyuan Chen, and Zhou Zhao. Classifier Ctassifier-guided gradient modulation for enhanced multimodal learning. Advances in Neural Information Processing Systems, 17:113328-13322 2024.
[11] Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Philippe Morency, and Mohammed (Ehsan) Hoque. UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2046 2056, Hong Kong, China, 2019, Association for Computational Linguistics.
[12] Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural computation, 773 1997.
[13] Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modality reconcilement. In The Forty-first International Conference on Machine Learning, 2024.
[14] Yu Huang, Chenzhuang Du Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi modal learning better than single (provably). Advances in Neural Information Processing Systems. 34:10944 1110944-10956, 2021.
10

---

[15] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang. and Longbo Huang. Modality compecition: What makes joint training of multi-modal network fail in deep (epraing.(prvablly).. In International Conference on Machine Learning, pages 9226-9259. PMLR, 2022.
[16] Aya Abdelsalam Ismail, Mahmudul Hasan, and Faisal Ishtiaq. Improving multimodal accuracy through modality pre-training and attention. arXiv preprint S.Ki:20111.66000 2020.
[17] Vishrut Jetly and Hikaru Ibayashi. Splash in a flash: Sharpness-aware minimization for efficient liquid splash simulation. In Annual Conference of the European Association for Computer Graphics, Eurograph- ics, 2022.
[18] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch batch training for deep learning: Generalization gap and sharp minima. arXiv preprint a60iv: 04836, 2016.
[19] Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pages PMLR, 2022.
[20] Konstantinos Kontras, Christos Chatzichristos, Matthew Blaschko, and Maarten De Vos. Improving multimodal learning with multi-loss gradient modulation, arXiv preprint arXiv:2405.07953 2024.
[21] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi, Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914. PMLR. 2021.
[22] Binh Le and Simon S Woo. Gradient alignment for cross-domain face anti apooiis...ing.... Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 188-199 199,1999,66 2024.
[23] Hong Li, Xingyu Li, Pengho Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting multi-modal model performance with adaptive gradient modulation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22214 222244, 2023.
[24] Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimiza- tion. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5631 1514-564,222 2024.
[25] Paul Pu Liang. Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng. Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In Fiitty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[26] Clara Na. Sanket Vaibhav Mehta, and Emma Strubell. Train flat, then compress: Sharpness-aware minimization learns more compressible models. EMNLP, 2022.
[27] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-1I), pages 689-699 (89-666,220 2011.
[28] Xiaokang Peng. Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-ily gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8238-8247, 2022.
[29] Benedek Rozemberczki, Rik Sarkar, and Andrew McCallum. The shapley value in machine learning. arXiv preprint 2XXi:22002.05594, 2022.
[30] Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 28:1650-166 1654, 2021.
[31] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on computer vision (ECCV), pages 247-263, 2018
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
[33] Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: a multilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. pages 0-0, 2018
11

---

[34] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695-12705, 2020.
[35] Yake Wei, Weiran Shen, and Di Hu. MMPareto: Innocent uni-modal assistance for enhanced multi modal learning. 2024.
[36] Thomas Winterbottom, Sarah Xiao, Alistair McLean, and Noura Al Moubayed. On modality bias in the tvqa dataset. arXiv preprint arXiv:2012.10211 2020.
[37] Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
[38] Yiqun Yao and Rada Mihalcea. -pdality-peecffie learning rates for effective multimodal additive late- fusion. In Findings of the Association for Computational Linguisties: ACL 2022, pages 1824-1834, 2022.
[39] BT Thomas Yeo, Mert R Sabuncu, Rahul Desikan, Bruce Fischl, Polina Golland. Effects registration regularization and atlas sharpness on segmentation accuracy. Medical image analysis, 12(5):603-615, 2008.
[40] Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pages 20247-20257, 2023.
[41] Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Jubua Liu, Bo Du, and Dacheng Tao. Improving sharpness- minimization with fisher mask for better generalization language models. arXiv preprint x/X1v:2210005497, 2022.
[42] Yixuan Zhou, Yi Qu, Xing Xu, and Hengtao Shen. Imbsam: closer look at sharpness- minimization in ilas--imbllnnced recognition. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11345 11355, 2023.
[43] Hao Zhu, Man-Di Di Luo. Wang, Ai-Hua Zheng, and Ran He. Deep visual learning: survey. International Journal of Automation and Computing, 18(3):311337,, 2021.
[44] Juntang Zhuang, Boqing Gong. Liangzhe Yuan, Cui. Hartwig Adam, Nicha Dvorek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness aware training. ICLR, 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately state the contributions of the paper and align with both the theoretical and experimental results presented, effectively setting the scope and expectations for the reader.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
the paper discusses the limitations of the work, including thorough examina- tion of assumptions, potential robustness issues, and the scope of empirical validity. This discussion ensures transparency and aligns with academic norms that encourage honesty about the limitations of research methodologies and findings.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
the paper provides complete set of assumptions and correct proofs for each theoretical result, ensuring that all theorems and lemmas are well-documented and cross-referenced, with detailed proofs accessible either in the main text or supplemental material.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All of our results can be reproduced and we will release our code after the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used the paper is publicly accessible, We will also release our code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
mublic//guides/CoeStttmi cessoonnolccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ctipps.c/puulic//uiiiee guwdes/ CoodeSubeeieeeee wwtsSSuuiisiiioooo...oooo ssi ssionnolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
training and test details, including data splits, hyperparameters, and optimizer types, well as how these were chosen. These details presented the core of the paper, with additional information available in the appendix or supplemental material, ensuring that the experimental results can be fully understood and appreciated.
Justification: the paper specifies
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
15

---

Justification:
the paper reports error bars suitably and correctly defined, along with appro- priate information about the statistical significance of the experiments. It clearly states the factors of variability captured by the error bars, the method for calculating them, and the assumptions made. Additionally, the paper explains the preprocessing of data in detail, providing a level of transparency not found in some other literature.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: the paper provides sufficient information on the computer resources needed to reproduce the experiments by mentioning the hardware used for exccution, such as the type of CPU or GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
16

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: the research is foundational and applicable broad area, making it difficult to define specific societal applications or impacts. The paper does not address societal impacts because the nature of the work does not lend itself to immediate, direct societal implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our method doesn't have high risk of misuse. We only use publicly available data and models, which poses no significant risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We credit every referenced asset with citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasste has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces new code for sharpness-awan aware minimization designed for multi modaiity learning, and we will release the code after the paper accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper doesn't perform crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
18

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper doesn't perform research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
19

---

A Dataset
To evaluate the effectiveness of our approach, conducted experiments on three widely used multimodal datasets from the domains of affective computing and multimedia. From the affective computing domain, we utilized the CREMA-D and UR-Funny datasets.
Table 3: The datasets specifications.
Field of Research
Size
Dataset
Modality
Samples
content
UR-Famry
[a.1.1)
16,114
hamox
Affective Computing
CREMA-D
AV-NNIST 33
(a.v]
70,000
digit
Multimedia
AVE
(a.v) 4,143
event
detection
The CREMA-D dataset, curated for speech emotion recognition tasks, includes six emotional labels spanning various speech recordings. The UR-Funny dataset, designed for humor detection, incorpo- rates multimodal cues, including text, visual gestures, and ccoustic-prosodif features, providing rich benchmark for affective computing. In the multimedia domain, we employed the AV-MNIST dataset, which focuses on multimedia classification. This dataset features disturbed images paired with audio signals, offering a challenging setting for evaluating cross-modal learning.
Further details about these datasets, including modality types and task descriptions, are provided in Table. 3
B Mono-Modal Contribution in Total Performance
To split the total loss function into the modality-specifee loss function what Eq. shows, adopted the Shapley metric proposed by [23]. Consider:
4 (x) (01, ØM: 8*)
(16)
where x = (x1...., represents all corresponding M modalities the input and om = f (x) ( 2hi i 8 8 1....... m im........ 8 (m is defined the feature representation pro- duced by the m-th encoder taking into account other modalities' effect through the fusion network's parameters, Omj- Lets M {
M denote the set of all modalities. Zero-padding Om indicates the absence of modality m features. If S is subset of M, then ((S) indicates that for m E S, Fm is replaced with Om The mono-modal response for modality m is then defined
IS (k
|S 1)!
+ pm(r) (x)
Σ
V (S:
(17)
k!
SCM. {m) 3+0
where (S; 4(S {m]) - ((S).
The empty subset is excluded from the summation to
ensure the relation:
(xx) = Σ q (c).
(18)
T
For illustration, in the case of two modalities, it would be follows;
Pmn (x)
| [ (((01, + #([m), Omz))-
(19)
B.1 From Mono-Modal Attribute to Loss Landscape Decomposition
The Shapley values in Eq. 17 serve as decomposing weight to compute mono-modal contributions into the overall accuracy. They inherently operate in the domain of performance metries, not loss
20

---

Audio Accuracy A2 VRIST
Vinual Accurnsy AU-BIETT Hanlen)
Cveral AMAN
Epachs
Ecodts
Sucido ACCUCRIZ on CHEMA Date falion)
Youual SeCouEcce on CAEMAUI CIAA Borkac
Sestral Aicornca on CHENG Ausiont
Epocth
Spocta
Figure 4: Performance comparisons of late fusion settings: Joint-Trained, AGM, OGM-GE, SAM, and our proposed Modality-Awart SAM, on the AV-MNIST and CREMA-D datasets. Viewing this in color is recommended.
ACCUMNCT
Eoodis
Ecochs
Esocha
Audio Accurasy on CREMAUD Jearle Iasion
Visua Accurssy on CREMA lsar lusion)
Over Accurasy on CHENA rearly Bailet
Figure Performance comparisons of early fusion settings: Joint-Trained, AGM, OGM-GE. SAM, and our proposed Modality-Aware SAM, on the AV-MNIST and CREMA-D datasets. Viewing this in color recommended.
values. In contrast, SAM is explicitly designed to operate on the optimization landscape by directly utilizing loss values. This fundamental difference raises critical challenge: how can we bridge the gap between modality-specific performance contributions and the loss landscape decomposition during the training process?
To address this, we propose leveraging the Shapley value in Eq. 17 By computing Shapley values for each modality during training, dynamically adjust the weights associated with their losses, thereby linking the concept of modality performance to loss decomposition. This dynamic adjustment ensures that modalities contributing more performance are appropriately emphasized in the loss computation. Specifically, the weight associated with each modality's 's loss derived from its normalized Shapley value. Mathematically, this is expressed as:
m
Vm
M
(20)
Din
where M the total number of modalities, om is the Shapley value of the mith modality, and Pm represents the normalized weight in Eq.
21

---

ACCUTRCC MiAlee Falint
SOWIN DCSEETNC
Figure 6: Performance comparisons of the AGM, SAM, and our M-SAM UR-Funny datasets' validation set using late fusion (first row) and early fusion (second row) architecture. It is recommended to view this color.
C Learning Curves
Fig- and Fig. 6 present the performance comparisons of our proposed Modality-Aware SAM (M-SAM) with SAM and other state-of-the-art methods, including AGM, OMG-GE, and Joint-Train, under late fusion settings for the AV-MNIST and CREMA-D datasets and both late and early fusion scenarios of URFunny dataset, respectively. The results consistently demonstrate the superiority of M-SAM across both datasets and all metries (audio, visual, and overall accuracy).
For the AV-MNIST dataset, M-SAM achieves the highest accuracy in all metrics. In the audio modal- ity, M-SAM converges quickly and achieves final accuracy of approximately 0.40, outperforming SAM (0.35), AGM, and OMG-GE, which lag significantly. In the visual modality. M-SAM demon- strates smooth and steady improvement, achieving the best accuracy of around 0.65. In contrast, SAM achieves slightly lower accuracy, and AGM and OMG-GE show notable instability. Regarding overall accuracy, M-SAM reaches approximately 0.74, clearly surpassing SAM (0.71) and significantly outperforming other methods, which fail to approach competitive levels. On the CREMA-D dataset, M-SAM maintains its superiority across all metrics. In the audio modality, M-SAM consistently achieves higher accuracy and improved stability compared to SAM, which shows oscillations during training. AGM and OMG-GE perform poorly, failing converge effectively.
the visual modality, M-SAM achieves the highest accuracy, with SAM trailing behind and other methods struggling to maintain stability. Finally, overall accuracy, M-SAM again emerges as the best pest-performing method, with the smoothest convergence and highest final accuracy.
D Margin of Superiority over Joint-Train baseline
To evaluate the effectiveness of each method, we report the normalized marginal improvement in accuracy over the Joint Training (JT) baseline. Specifically, we compute the percentage increase in overall accuracy (Accmm) for each method relative to the JT baseline using the following formulation:
KAPRENT
(a)
(d)
Figure 7: Relative improvement in multi-modal accuracy (AcCmmm) over the Joint- Shnt-Training baseline on (a) AV-MNIST, (b) CREMA-D, (c) UR-Funny, and (d) AVE datasets. Our M-SAM consistently achieves the highest normalized gain, outperforming all methods.
This normalized metric allows for fair comparison across datasets of varying base difficulty and scales, highlighting how much each method improves over standard joint optimization.
22

---

A (Method) Ac (Method) Acc (JT) x 100
(21)
mm (JT)
As shown in Figure 7 our proposed M-SAM consistently achieves the largest relative gain across all
datasets, outperforming strong baselines such as Recon-Boos Boost and CGGM.
23


Reviews:
[{"review_id": "pj4VVMSjcd", "reviewer": "Reviewer_oFnA", "summary": "This paper proposes Modality-Aware Sharpness-Aware Minimization (M-SAM), a novel optimizer-level approach for harmonizing multimodal learning by extending Sharpness-Aware Minimization (SAM). M-SAM focuses on guiding optimization to flatter minima by dynamically favoring the dominant modality per mini-batch using Shapley value decomposition and per-modality gradient modulation. The method is evaluated across four benchmark multimodal datasets under both early and late fusion setups and achieves superior accuracy and generalization compared to a wide range of recent state-of-the-art methods.", "questions": "* It would be helpful if the authors could clarify whether the proposed approach is supported by theoretical analysis or formal guarantees.\n* What are the implications of dominant modality shifts across training phases or mini-batches?\n* The writting issue should be fixed.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n* By using Shapley values to identify and prioritize the dominant modality's generalization, the paper proposes a unique solution to the pervasive issue of modality imbalance and uncoordinated convergence in multimodal learning.\n* M-SAM outperforms strong baselines such as CGGM, AGM, Recon-Boost, and MM-Pareto in almost all cases across all benchmark datasets.\n* The proposed method performs well in both early and late fusion settings, showing its robustness and broad applicability.\n\n**Weaknesses:**\n* While intuitively and empirically strong, the paper lacks theoretical analysis or convergence guarantees under the M-SAM optimization dynamics, especially with selective gradient application.\n* The paper states M-SAM \"detects and prioritizes the dominant modality at each iteration and mini-batch\". However, the implications of a dominant modality potentially shifting or oscillating across different training phases or mini-batches are not thoroughly discussed.\n* There are still some minor issues in the writing that should be addressed for improved clarity and professionalism. For example, neither Table 1 nor Table 2 is referenced in the main text. The subsection titles in Section 4 use an unusual numbering format (e.g., 4.0.x), which is inconsistent with standard academic formatting. Figure 2 is cited before Figure 1, which disrupts the logical flow.\n* Algorithm 1 is poorly typeset and hard to read. A cleaner format or pseudocode would help comprehension.", "was_revised": true, "final_justification": "The paper is now well-developed and supported by solid theoretical contributions.Based on the overall quality of the work, I recommend that this paper be accepted for publication.", "rebuttal": "**Response to Comment 1 (Theoretical Convergence Guarantees under the M-SAM optimization dynamics):** \n\nWe would like to sincerely thank the reviewer for raising the concern about the convergence behavior of M‑SAM under its dynamic updates.\n\nThrough the following proof we will show that the convergence wont be affected by the choice of the dominant modality gradient at each step. Even if there is approximation error in Shapley-based modality selection, the updates remain stable and converge.\n\nUnder the assumptions of:\n\n1. $\\\\mathcal{L}(\\\\theta_t) = \\\\sum_{m=1}^{M} \\\\nu_m \\\\mathcal{L}_m(\\\\theta_t)$ and its gradient is bounded $K$-smooth ($K$-Lipschitz), i.e., $\\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t) - \\\\nabla\\\\mathcal{L} (\\\\hat \\\\theta_t) \\\\| \\\\leq K \\\\| \\\\theta_t - \\\\hat \\\\theta_t\\\\|$, \n\n$\\\\forall (\\\\theta_t, \\\\hat \\\\theta_{t})$, that gradient is bounded:\n    $\n    \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\leq G_{\\\\text{max}}.\n    $\n\n2. \n    Learning rate \n    $\n    \\\\eta_t = \\\\frac{\\\\eta_0}{\\\\sqrt{t}}\n    $\n    and perturbation \n    $\n    \\\\rho_t = \\\\frac{\\\\rho_0}{\\\\sqrt{t}}.\n    $\n\nThen when $T$ is large enough the M-SAM updates satisfy:\n\n\\\\[\n\\\\frac{1}{T} \\\\sum_{t=1}^{T} \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2\n\\\\leq \\\\mathcal{O}\\\\left(\\\\frac{\\\\log T}{\\\\sqrt{T}}\\\\right).\n\\\\]\n\n(Before going through the proof, we would like to mention that, for tractability in the theoretical analysis, we assumed a smooth decay schedule $\\\\eta_t = \\\\eta_0/ \\\\sqrt{t}$. In practice, our implementation uses a step-wise decay (multiplying the learning rate by $0.1$ every $70$ iterations), which decreases $ \\\\eta_t $ even faster. This practical schedule still satisfies the conditions required for convergence and does not weaken the theoretical guarantee.)\n\nConsidering $d_t = \\\\theta_{t+1} - \\\\theta_t = -\\\\eta_t \\\\nabla \\\\mathcal{L} (\\\\hat{\\\\theta_t})$ and $\\\\hat{\\\\theta_t} = \\\\theta_t + \\\\rho_t \\\\frac{\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)}{\\\\|\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)\\\\|}$; where $\\\\nabla_{m^\\\\ast}$ denotes the gradient with respect to the dominant modality, generated by backpropagation of loss value corresponding to the dominant modality, $m^\\\\ast$ (Eq. 4 in the paper). \n\nUsing descent Lemma for a $k$-smooth function we can write:\n\\\\begin{equation}\n     \\\\begin{aligned}\n\\\\mathcal{L}(\\\\theta_{t+1}) - \\\\mathcal{L}(\\\\theta_t) \\\\leq \n& -\\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\theta_{t+1} - \\\\theta_t \\\\rangle + \\\\frac{K\\\\eta_t^2}{2} \\\\|\\\\theta_{t+1} - \\\\theta_t\\\\|^2 = \\\\\\\\ \n& -\\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_t) \\\\rangle + \\\\frac{K\\\\eta_t^2}{2} \\\\|\\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_{t})\\\\|^2 = \\\\\\\\ \n&-\\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\nabla \\\\mathcal{L} (\\\\theta_{t}) -\\\\nabla \\\\mathcal{L}(\\\\theta_t) + \\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_t) \\\\rangle + \\\\frac{K \\\\eta_t^2}{2} \\\\|\\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_t)\\\\|^2 = \\\\\\\\\n& - \\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 - \\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_{t}) - \\\\nabla \\\\mathcal{L}(\\\\theta_t) \\\\rangle  + \\\\frac{K \\\\eta_t^2}{2} (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) \\\\leq \\\\\\\\ \n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + \\\\eta_t \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\|\\\\nabla \\\\mathcal{L} (\\\\theta_{t}) - \\\\nabla \\\\mathcal{L}(\\\\hat\\\\theta_t) \\\\| + \\\\frac{K \\\\eta_t^2}{2}  (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) \\\\leq \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\|\\hat\\\\theta_{t} - \\\\theta_t \\\\| + \\\\frac{K \\\\eta_t^2}{2}  (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) = \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\| \\\\rho_t \\\\frac{\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)}{\\\\|\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)\\\\|}\\\\|^2) + \\\\frac{K \\\\eta_t^2}{2}  (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) = \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\rho_t^2 \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|  + \\\\frac{K \\\\eta_t^2}{2}  \\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2 \\\\leq \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\rho_t^2 G_{max}  + \\\\frac{K \\\\eta_t^2}{2}  G_{max}^2 \n     \\\\end{aligned}\n\\\\end{equation}\n\n_Here we would like to highlight that the step of finding the upper bound on the right-hand side of the inequality is the only place where the dynamic nature of modality selection could potentially affect convergence. However, it is clearly shown that the convergence is independent of_ $\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)$. *This is because the upper bound,* $\\\\| \\\\rho_t \\\\frac{\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)}{\\\\|\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)\\\\|}\\\\|^2$, *depends only on* $\\\\rho_t$_, *and not on the direction of the selected modality’s gradient.* Hence, the convergence is independent of modality selection at each iteration. \n       \nBy rearranging the inequality, it will be as:\n\n\\\\begin{equation}\n    \\\\begin{aligned}\n        &\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 \\\\leq \\\\mathcal{L}(\\\\theta_{t}) - \\\\mathcal{L}(\\\\theta_{t+1})+K\\\\eta_t \\\\rho_t^2 G_{max}  + \\\\frac{K \\\\eta_t^2}{2}  G_{max}^2\n    \\\\end{aligned}\n\\\\end{equation}\n\nconsidering definition of $\\\\eta_{t} = \\\\frac{\\\\eta_{0}}{\\\\sqrt{t}}$ and taking summation over all iterations on the left side of the above inequality, we can define a lower bound as follows:\n\n\\begin{equation}\n\\frac{\\eta_{0}}{\\sqrt{T}}\\sum_{t=1}^{T} \\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2 \\leq\n\\sum_{t=1}^{T} \\eta_t \\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2 \\leq\n\\sum_{t=1}^{T} \\big(\\mathcal{L}(\\theta_{t}) - \\mathcal{L}(\\theta_{t+1}) \\big) + K \\eta_0 \\rho_0^2 G_{max} \\sum_{t=1}^{T} \\frac{1}{t\\sqrt{t}} + \\frac{k \\eta_0^2 G_{max}^2}{2} \\sum_{t=1}^{T} \\frac{1}{t}\n\\end{equation}\n\nusing telescope sum properties and considering $0 \\\\leq \\\\mathcal{L}(\\\\theta_t)  \\\\forall t :$\n\n\\begin{equation}\n        \\sum_{t=1}^{T} \\big(\\mathcal{L}(\\theta_{t}) - \\mathcal{L}(\\theta_{t+1}) \\big) = \\mathcal{L}(\\\\theta_1) - \\mathcal{L}(\\\\theta_{T+1})\n        \\leq \\mathcal{L}(\\\\theta_1)\n\\end{equation}\n\n\nSo we can rewrite the main inequality as follows:\n\n\\begin{equation}\n\\frac{1}{T}\\sum_{t=1}^{T} \\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2 \\leq\n \\frac{\\mathcal{L}(\\\\theta_1)}{\\sqrt{T}} + \\frac{K \\rho_0^2 G_{max}}{\\sqrt{T}} \\sum_{t=1}^{T} \\frac{1}{t\\sqrt{t}} + \\frac{K \\eta_0 G_{max}^2}{2\\sqrt{T}} \\sum_{t=1}^{T} \\frac{1}{t}\n\\end{equation}\n\n\nConsidering $\\sum_{t=1}^{T} \\frac{1}{t} \\leq 1+\\log T$ and the fact that $\\sum_{t=1}^{T} \\frac{1}{t^{3/2}}$ is a $p$‑series with $p=\\tfrac{3}{2}(>1)$, its partial sum remains bounded, so the only term on the right-hand side that confine the asymptotic convergence rate is the harmonic term, which results in an overall rate of $\\mathcal{O}\\big(\\tfrac{\\log T}{\\sqrt{T}}\\big)$.\n\n\n\\\\[\n\\\\frac{1}{T} \\\\sum_{t=1}^{T} \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2\n\\\\leq \\\\mathcal{O}\\\\left(\\\\frac{\\\\log T}{\\\\sqrt{T}}\\\\right).\n\\\\]\n\nSince the average gradient norm $\\frac{1}{T}\\sum_{t=1}^{T}\\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2$ decreases at a rate of $\\tfrac{\\log T}{\\sqrt{T}}$ and tends to $0$ as $T$ grows, the updates approach a stationary point, which by definition establishes convergence (as in standard SGD and traditional SAM).\n\nThis bound, $\\mathcal{O}\\big(\\tfrac{\\log T}{\\sqrt{T}}\\big)$, matches the standard non‑convex convergence rate of SGD with a decaying learning rate schedule (where the logarithmic factor comes from $\\sum 1/t$). Thus, the proposed M-SAM method preserves the same theoretical convergence guarantees as classical SGD and traditional SAM while offering dynamic modality-based perturbations.\n\n**Response to Comment 2 (Implications of Dominant Modality Shifts across Training Phases and Mini‑Batches):**\n\nWe thank the reviewer for raising this thoughtful point about shifts in the dominant modality during training and their effect on the learning dynamics of M‑SAM.\n\nM‑SAM is designed to benefit from these shifts rather than suppress them. By re‑evaluating the dominant modality at every iteration using Shapley‑based contributions, the method dynamically allocates attention to the modality that is most informative at that stage of training.\n\nAs discussed in our response to Comment 1, such alternations do not compromise stability or convergence. Instead, they guide the currently dominant modality toward flatter minima, giving other modalities freedom to contribute. When dominance oscillates between modalities, this competition prevents any single modality from monopolizing learning, leading to a more balanced and robust multimodal representation.\n\n**Response to Comment 3 (Writing Issues):**\n\nWe appreciate the reviewer’s detailed feedback on clarity and presentation. We acknowledge these issues and will address them in the camera‑ready version by correcting table references, subsection numbering, figure order, and improving the formatting of Algorithm 1 for better readability."}, {"review_id": "cTJXm0lm0o", "reviewer": "Reviewer_sAY6", "summary": "This paper addresses the important problem of modality imbalance in multi-modal learning. In current joint learning frameworks, the gradients of different modality encoders are coupled, which often leads to the dominant modality overpowering the fusion process and ultimately limiting overall performance. The authors incorporate SAM to address this issue. Comprehensive experiments demonstrate the effectiveness of the proposed method.", "questions": "Please see the weakness above.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work in the appendix section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The strengths are listed as follows:\n\n- The paper offers a novel perspective on mitigating modality imbalance by introducing SAM into the multi-modal learning community.\n\n- M-SAM achieves consistent improvements over state-of-the-art methods across several benchmarks (e.g., AV-MNIST, CREMA-D, UR-Funny, AVE) in both single- and multi-modal settings. Its focus on flatter minima leads to better generalization and reduced overfitting, as shown by the normalized overfitting gap and loss landscape visualizations.\n\nHowever, I have the following concerns:\n\n- From my perspective, SAM-based methods such as ImbSAM and CCSAM face a trade-off between computational efficiency and control over the loss landscape. I am curious how the proposed M-SAM balances these two aspects.\n\n- Although the method shows strong empirical performance, it lacks formal guarantees or deeper theoretical analysis on convergence or generalization, such as those provided by FocalSAM, especially under varied modality dominance patterns or real-world noisy scenarios.", "was_revised": true, "final_justification": "This paper explores an important problem and conduct various experiments. However, my primary concern is the computational efficiency as mentioned in W.1. Nevertheless, I lean towards accept.", "rebuttal": "**Response to Comment 1 (computational efficiency):**\n\nWe thank the reviewer for raising this point.\nUnlike Imb‑SAM and CCSAM (which address class imbalance), M‑SAM targets modality imbalance. Its complexity depends only on the number of input modalities $k$, not on the number of classes.\n\nIn each iteration, M‑SAM computes Shapley weights using all non‑empty subsets of modalities, excluding only the trivial all‑zero subset.\nThis introduces $2^k-2$ additional forward passes over SAM, with no additional backward passes or memory overhead, since these extra passes are used solely for weight estimation and do not store activations and gradients.\n\nIn the paper we evaluated M-SAM using $4$ famous and widely used datasets in the field:\n\n$k=2$ (e.g., AV‑MNIST, CREMA‑D): extra forward passes = $2^2-2=2$\n\n$k=3$ (e.g., UR‑Funny): extra forward passes = $2^3-2=6$\n\n| Method          | Forward | Backward |\n| --------------- | ------- | -------- |\n| SAM             | 2       | 2        |\n| M‑SAM (\\$k=2\\$) | 4       | 2        |\n| M‑SAM (\\$k=3\\$) | 8       | 2        |\n\nM‑SAM introduces only a modest additional cost:\nfor $k=2$ modalities, the forward‑to‑backward ratio changes from $(2+2)/(2+2)=1$ in SAM to $(4+2)/(2+2)=1.5$,\nand for $k=3$ modalities it becomes $(8+2)/(2+2)=2.5$,\nwhile the number of backward passes and memory requirements remain exactly the same as SAM.\n\nI would like to emphasize that, this overhead is entirely independent of the number of dataset classes,\nand in practice the number of modalities rarely grows beyond $2$ or $3$, so the additional cost remains small.\n\n**Response to Comment 2 (Convergence and Theoretical Guarantees):** \n\nWe thank the reviewer for this valuable comment highlighting the need for a deeper theoretical analysis.  \nTo address this concern, we provide a convergence proof for the proposed M‑SAM method.  \nUnder standard smoothness and bounded gradient assumptions, we show that the average gradient norm decreases at a rate of $\\mathcal{O}\\big(\\tfrac{\\log T}{\\sqrt{T}}\\big)$, which matches the classical convergence behavior of SGD and SAM.  \nThis result formally guarantees the stability and convergence of M‑SAM even under varying modality dominance patterns.\n\n\nUnder the assumptions of:\n\n1. $\\\\mathcal{L}(\\\\theta_t) = \\\\sum_{m=1}^{M} \\\\nu_m \\\\mathcal{L}_m(\\\\theta_t)$ and its gradient is bounded $K$-smooth ($K$-Lipschitz), i.e., $\\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t) - \\\\nabla\\\\mathcal{L} (\\\\hat \\\\theta_t) \\\\| \\\\leq K \\\\| \\\\theta_t - \\\\hat \\\\theta_t\\\\|$, \n\n$\\\\forall (\\\\theta_t, \\\\hat \\\\theta_{t})$, that \n    $\n    \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\leq G_{\\\\text{max}}.\n    $\n\n2. \n    Learning rate \n    $\n    \\\\eta_t = \\\\frac{\\\\eta_0}{\\\\sqrt{t}}\n    $\n    and perturbation \n    $\n    \\\\rho_t = \\\\frac{\\\\rho_0}{\\\\sqrt{t}}.\n    $\n\nThen when $T$ is large enough the M-SAM updates satisfy:\n\n\\\\[\n\\\\frac{1}{T} \\\\sum_{t=1}^{T} \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2\n\\\\leq \\\\mathcal{O}\\\\left(\\\\frac{\\\\log T}{\\\\sqrt{T}}\\\\right).\n\\\\]\n\nBefore going through the proof, we would like to mention that, for tractability in the theoretical analysis, we assumed a smooth decay schedule \\( \\eta_t = \\eta_0/\\sqrt{t} \\). In practice, our implementation uses a step-wise decay (multiplying the learning rate by $0.1$ every $70$ iterations), which decreases $ \\\\eta_t $ even faster. This practical schedule still satisfies the conditions required for convergence and does not weaken the theoretical guarantee.\n\nconsidering $d_t = \\\\theta_{t+1} - \\\\theta_t = -\\\\eta_t \\\\nabla \\\\mathcal{L} (\\\\hat{\\\\theta_t})$ and $\\\\hat{\\\\theta_t} = \\\\theta_t + \\\\rho_t \\\\frac{\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)}{\\\\|\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)\\\\|}$; where $\\\\nabla_{m^\\\\ast}$ denotes the gradient with respect to the dominant modality that generated by backpropagation of loss value corresponding to the dominant modality (Eq. 4 in the paper). \n\nUsing descent Lemma for a $k$-smooth function we can write:\n\\\\begin{equation}\n     \\\\begin{aligned}\n\\\\mathcal{L}(\\\\theta_{t+1}) - \\\\mathcal{L}(\\\\theta_t) \\\\leq \n& -\\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\theta_{t+1} - \\\\theta_t \\\\rangle + \\\\frac{K\\\\eta_t^2}{2} \\\\|\\\\theta_{t+1} - \\\\theta_t\\\\|^2 = \\\\\\\\ \n& -\\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_t) \\\\rangle + \\\\frac{K\\\\eta_t^2}{2} \\\\|\\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_{t})\\\\|^2 = \\\\\\\\ \n&-\\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\nabla \\\\mathcal{L} (\\\\theta_{t}) -\\\\nabla \\\\mathcal{L}(\\\\theta_t) + \\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_t) \\\\rangle + \\\\frac{K \\\\eta_t^2}{2} \\\\|\\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_t)\\\\|^2 = \\\\\\\\\n& - \\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 - \\\\eta_t \\\\langle \\\\nabla \\\\mathcal{L}(\\\\theta_t), \\\\nabla \\\\mathcal{L} (\\\\hat \\\\theta_{t}) - \\\\nabla \\\\mathcal{L}(\\\\theta_t) \\\\rangle  + \\\\frac{K \\\\eta_t^2}{2} (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) \\\\leq \\\\\\\\ \n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + \\\\eta_t \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\|\\\\nabla \\\\mathcal{L} (\\\\theta_{t}) - \\\\nabla \\\\mathcal{L}(\\\\hat\\\\theta_t) \\\\| + \\\\frac{K \\\\eta_t^2}{2}  (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) \\\\leq \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\|\\hat\\\\theta_{t} - \\\\theta_t \\\\| + \\\\frac{K \\\\eta_t^2}{2}  (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) = \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\| \\\\| \\\\rho_t \\\\frac{\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)}{\\\\|\\\\nabla_{m^\\\\ast} \\\\mathcal{L}(\\\\theta_t)\\\\|}\\\\|^2) + \\\\frac{K \\\\eta_t^2}{2}  (\\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2) = \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\rho_t^2 \\\\| \\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|  + \\\\frac{K \\\\eta_t^2}{2}  \\\\|\\\\nabla \\\\mathcal{L} (\\\\hat\\\\theta_{t})\\\\|^2 \\\\leq \\\\\\\\\n&-\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 + K\\\\eta_t \\\\rho_t^2 G_{max}  + \\\\frac{K \\\\eta_t^2}{2}  G_{max}^2 \n     \\\\end{aligned}\n\\\\end{equation}\n\n\n       \nby rearranging the inequality, it will be as:\n\n\\\\begin{equation}\n    \\\\begin{aligned}\n        &\\\\eta_t \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2 \\\\leq \\\\mathcal{L}(\\\\theta_{t}) - \\\\mathcal{L}(\\\\theta_{t+1})+K\\\\eta_t \\\\rho_t^2 G_{max}  + \\\\frac{K \\\\eta_t^2}{2}  G_{max}^2\n    \\\\end{aligned}\n\\\\end{equation}\n\nconsidering definition of $\\\\eta_{t} = \\\\frac{\\\\eta_{0}}{\\\\sqrt{t}}$ and taking summation over all iterations on the left side of the above inequality, we can define a lower bound as follows:\n\n\\begin{equation}\n\\frac{\\eta_{0}}{\\sqrt{T}}\\sum_{t=1}^{T} \\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2 \\leq\n\\sum_{t=1}^{T} \\eta_t \\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2 \\leq\n\\sum_{t=1}^{T} \\big(\\mathcal{L}(\\theta_{t}) - \\mathcal{L}(\\theta_{t+1}) \\big) + K \\eta_0 \\rho_0^2 G_{max} \\sum_{t=1}^{T} \\frac{1}{t\\sqrt{t}} + \\frac{k \\eta_0^2 G_{max}^2}{2} \\sum_{t=1}^{T} \\frac{1}{t}\n\\end{equation}\n\nusing telescope sum properties and considering $0 \\\\leq \\\\mathcal{L}(\\\\theta_t)  \\\\forall t :$\n\n\\begin{equation}\n        \\sum_{t=1}^{T} \\big(\\mathcal{L}(\\theta_{t}) - \\mathcal{L}(\\theta_{t+1}) \\big) = \\mathcal{L}(\\\\theta_1) - \\mathcal{L}(\\\\theta_{T+1})\n        \\leq \\mathcal{L}(\\\\theta_1)\n\\end{equation}\n\n\nSo we can rewrite the main inequality as follows:\n\n\\begin{equation}\n\\frac{1}{T}\\sum_{t=1}^{T} \\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2 \\leq\n \\frac{\\mathcal{L}(\\\\theta_1)}{\\sqrt{T}} + \\frac{K \\rho_0^2 G_{max}}{\\sqrt{T}} \\sum_{t=1}^{T} \\frac{1}{t\\sqrt{t}} + \\frac{K \\eta_0 G_{max}^2}{2\\sqrt{T}} \\sum_{t=1}^{T} \\frac{1}{t}\n\\end{equation}\n\n\nConsidering $\\sum_{t=1}^{T} \\frac{1}{t} \\leq 1+\\log T$ and the fact that $\\sum_{t=1}^{T} \\frac{1}{t^{3/2}}$ is a $p$‑series with $p=\\tfrac{3}{2}(>1)$, its partial sum remains bounded, so the only term on the right-hand side that confine the asymptotic convergence rate is the harmonic term, which results in an overall rate of $\\mathcal{O}\\big(\\tfrac{\\log T}{\\sqrt{T}}\\big)$.\n\n\n\\\\[\n\\\\frac{1}{T} \\\\sum_{t=1}^{T} \\\\|\\\\nabla \\\\mathcal{L}(\\\\theta_t)\\\\|^2\n\\\\leq \\\\mathcal{O}\\\\left(\\\\frac{\\\\log T}{\\\\sqrt{T}}\\\\right).\n\\\\]\n\nSince the average gradient norm $\\frac{1}{T}\\sum_{t=1}^{T}\\\\|\\nabla \\mathcal{L}(\\theta_t)\\\\|^2$ decreases at a rate of $\\tfrac{\\log T}{\\sqrt{T}}$ and tends to $0$ as $T$ grows, the updates approach a stationary point, which by definition establishes convergence (as in standard SGD and traditional SAM).\n\nThis bound, $\\mathcal{O}\\big(\\tfrac{\\log T}{\\sqrt{T}}\\big)$, matches the standard non‑convex convergence rate of SGD with a decaying learning rate schedule (where the logarithmic factor comes from $\\sum 1/t$), so the proposed M‑SAM method has the same order of convergence as classical SGD and traditional SAM."}, {"review_id": "IBoZ8FJxwS", "reviewer": "Reviewer_VvWF", "summary": "This paper focuses on the challenge of ​​modal imbalance​​ in multimodal learning.That is, existing joint training paradigms ​​couple gradient updates​​ across modality-specific encoders where the dominant modality​​ suppresses weaker modalities during training. As a result, the generalization capability of multimodal models are degraded. To this end, the authors leverage the ​​Sharpness-Aware Minimization (SAM) framework​​ to dynamically balance gradient contributions. ​​Experiments​​ on four benchmarks are carried out to validate the method’s efficacy in harmonizing modal optimization.", "questions": "See above.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n- The research problem is important and practical.\n\n**Weakness**\n\n- The paper lacks both rigious theoretical analysis or empirical evidence to demonstrate the effectiveness of M-SAM in addressing the modality imbalance problem. The motivation of M-SAM seems only supported by intuitive explaination without solid analysis. Given this point, the claim are unsubstantiated for the reviewer.\n\n- The authors argue that the consistency between validation accuracy and training accuracy indicates better generalization. However, this is not supported by extensive evidence and seems conflict with existing works[1].\n\n- Standard deviation or error bars are missing, which makes it difficult for the reviewer to figure out if the improvement is significant.\n\n- The core part of the proposed method is almost identical to ImbSAM, which makes the originality of this paper not compelling enough for the reviewer.\n\n[1] Do we need zero training loss after achieving zero training error? ICML'20", "was_revised": true, "final_justification": "The authors correctly replied my concerns. My major concerns about the soundness of theoretical analysis and reproducibility have been addressed.", "rebuttal": "**Response to Comment 1 (Lack of Theoretical Analysis or Empirical Evidence):**\n\nWe sincerely thank the reviewer for emphasizing the importance of both empirical and theoretical validation of our method.\nTo address the empirical aspect, we would like to highlight that the paper provides a broad set of experiments designed to demonstrate the effectiveness of M‑SAM in mitigating modality imbalance. \n\nFor example, Figure 1 presents a qualitative comparison of the loss landscapes of M‑SAM, traditional SAM, and a strong multimodal baseline (AGM) on CREMA‑D. This figure illustrates that M‑SAM consistently converges to flatter minima, a property that is widely associated with better generalization. Complementing this, Figure 2 reports results based on the generalization‑gap metric on AV‑MNIST, where M‑SAM demonstrates a clear and consistent advantage over all benchmarks. Beyond these visualizations, Tables 1 and 2 provide an extensive quantitative comparison across four diverse and challenging datasets (AV‑MNIST, CREMA‑D, UR‑Funny, and AVE) under both early‑fusion and late‑fusion architectures for single-modal and multi-modal accuracy. \n\nThese results, together with the training‑curve analyses presented in the Appendix, show that the benefits of M‑SAM hold across datasets and training regimes. Collectively, these results offer a comprehensive empirical evaluation spanning multiple settings and confirm the robustness of the proposed approach.\n\nIn addition to empirical validation, the paper offers a rigorous theoretical treatment. Specifically, we derive a closed‑form mathematical representation for the M‑SAM optimization target (Eq. 12), providing a principled foundation for the method.\n\nFurthermore, our detailed analyses presented in the responses to Comments 1 and 2 of Reviewers **oFnA** and **sAY6**. Those responses address concerns about convergence proof, computation analysis of MSAM, and proof of the fact that the proposed update rule remains stable and effective even in the presence of oscillating dominant modality selection. \n\nWe would like to mention that these analyses complement the empirical findings and reinforce that the contributions of M‑SAM are both theoretically sound and practically impactful.\n\n\n**Response to Comment 2 (Validation-Training Accuracy Consistency and Generalization):**\n\nWe would like to thank you for your thoughtful comment. Previous studies, including \\cite{Ishida}, the paper that you mentioned in your comment,  have firmly established that the gap between training and testing performance is an important indicator of the geometry of the solution found during training. For example, \\cite{Keskar} demonstrates that when a training process results in a significant gap between training and testing accuracies, it signals that the network has converged to a sharp minimum, characterized by a significant number of large positive eigenvalues in $\\\\nabla^2 f$. Such a geometry indicates a higher sensitivity to perturbations and typically results in lower inference accuracy on unseen data.\n\nMoreover, the relation between sharp minima and reduced generalization ability is a well-established topic that has been extensively investigated, from the pioneering work of Hochreiter and Schmidhuber \\cite{Hochreiter_Schmidhuber} to more recent studies on robust optimization and Sharpness-Aware Minimization \\cite{SAM_paper}. \n\nWe would like to highlight that in line 58, it clearly says \"... huge gap shows it tends to overfit..\" which is a highly compatible to what the paper you mentioned, \\cite{Ishida}, elaborate on its Fig.1, when it discussed the generalization gap in phase A and phase B.\n\n\n[Ishida]: \"Do we need zero training loss after achieving zero training error?\" Ishida, et al. (ICML2020)\n\n[Keskar]: \"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. \" Keskar, et al. (ICLR 2017).\n\n[Hochreiter_Schmidhuber]: \"Flat minima.\", Hochreiter, Sepp, and Jürgen Schmidhuber. Neural computation (1997).\n\n[SAM_paper]: \"Sharpness-aware minimization for efficiently improving generalization.\" , Foret, et al. (ICLR 2021).\n\n\n**Response to Comment 2 (Missing standard deviations or error bars in the results):**\n\nWe thank the reviewer for highlighting the importance of reporting variability.\nWe would like to mention that the values reported in the tables for M‑SAM correspond to the mean accuracy over five runs with seeds 1, 2, 3, 4, and 5. Also, we would like to clarify that our decision to report results based on just mean accuracy (without SD) was made to align with the result presentation approach established in widely adopted baseline methods such as Reacon-Boost (Classifier-guided Gradient Modulation for Enhanced Multimodal Learning- ICML2024), CGGM (Classifier-guided Gradient Modulation for Enhanced Multimodal Learning- NIPS2024), MM-Pareto (MMPareto: Innocent Uni-modal Assistance for Enhanced Multi-modal Learning -ICLR 2024), AGM (Boosting Multi-modal Model Performance with Adaptive Gradient Modulation-CVPR2023). \n\nTo address your concern, in the following tables we have included SD in our results (in Table.1, and Table.2) to more comprehensively demonstrate the reliability of our method.\n\n\nstandard deviations for Table.1 (late fusion):\n\n| Model  | AV-MNIST Acc_a | AV-MNIST Acc_v | AV-MNIST Acc_mm | CREMA-D Acc_a | CREMA-D Acc_v | CREMA-D Acc_mm | UR-Funny Acc_a | UR-Funny Acc_v | UR-Funny Acc_t | UR-Funny Acc_mm | AVE Acc_a | AVE Acc_v | AVE Acc_mm |\n|--------|----------------|----------------|-----------------|----------------|----------------|-----------------|----------------|----------------|----------------|-----------------|-----------|-----------|-------------|\n| **M-SAM (SD)** | 0.36 | 0.44 | 0.36 | 0.66 | 0.65 | 0.46 | 0.99 | 1.01 | 1.13 | 1.00 | 0.44 | 0.26 | 0.34 |\n\n\n\nstandard deviations for Table.2 (early fusion):\n\n| Model  | AV-MNIST Acc_a | AV-MNIST Acc_v | AV-MNIST Acc_mm | CREMA-D Acc_a | CREMA-D Acc_v | CREMA-D Acc_mm | UR-Funny Acc_a | UR-Funny Acc_v | UR-Funny Acc_t | UR-Funny Acc_mm | AVE Acc_a | AVE Acc_v | AVE Acc_mm |\n|--------|----------------|----------------|-----------------|----------------|----------------|-----------------|----------------|----------------|----------------|-----------------|-----------|-----------|-------------|\n| **M-SAM (SD)** | 0.49 | 0.32 | 0.31 | 0.52 | 0.53 | 0.55 | 1.12 | 0.92 | 0.89 | 1.04 | 0.40 | 0.42 | 0.49 |\n\n**Response to Comment 4 (Originality and Similarity to ImbSAM):**\n\nWe would like to thank the reviewer for this thoughtful comment. As the authors transparently acknowledged in Sec. 3.5, the idea of this paper was partly inspired by (Zhou et al., 2023), i.e., the Imb-SAM framework. This, by no means, signifies that we forcibly applied Imb-SAM to our approach. The high-level differences between our approach and Imb-SAM are two-fold: (i) loss computation and decomposition (explained in Sec. 3.2), (ii) considering the dynamics of the learning process, i.e., where and how to apply the SAM-based approach (explained in Sec. 3.3). Using the intuition from Sec 3.2 and Sec. 3.3, we arrive at Eq. (12), whose final mathematical form happens to resemble that of Imb‑SAM, but the derivation, underlying objectives, and dynamics that lead to it are fundamentally different, as explained above."}]

Author rebuttal:

Meta-review: This paper tackles the challenge of modality imbalance in multimodal learning by proposing M-SAM, a modality-aware extension of Sharpness-Aware Minimization. M-SAM identifies the dominant modality in each iteration using Shapley-based contribution analysis and modulates the gradient updates accordingly, guiding optimization toward flatter minima while mitigating the dominance effect.
The method demonstrates consistent improvements over strong baselines across multiple datasets and both early- and late-fusion settings. In the rebuttal, the authors further strengthened the work with convergence analysis and standard deviation reporting, effectively addressing concerns about efficiency, stability, and reproducibility. Overall, the paper makes a solid contribution in method design, theoretical support, and empirical validation, and I recommend acceptance.
Decision: Accept (poster)