MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
Authors: Seong-Hyeon Hwang
Soyoung Choi
Steven Euijong Whang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal learning
Data augmentation
Imbalanced Multimodal Learning
Primary area: deep_learning

Abstract: Multimodal models often over-rely on dominant modalities, failing to achieve optimal performance. While prior work focuses on modifying training objectives or optimization procedures, data-centric solutions remain underexplored. We propose MIDAS, a novel data augmentation strategy that generates misaligned samples with semantically inconsistent cross-modal information, labeled using unimodal confidence scores to compel learning from contradictory signals. However, this confidence-based labeling can still favor the more confident modality. To address this within our misaligned samples, we introduce weak-modality weighting, which dynamically increases the loss weight of the least confident modality, thereby helping the model fully utilize weaker modality. Furthermore, when misaligned features exhibit greater similarity to the aligned features, these misaligned samples pose a greater challenge, thereby enabling the model to better distinguish between classes. To leverage this, we propose hard-sample weighting, which prioritizes such semantically ambiguous misaligned samples. Experiments on multiple multimodal classification benchmarks demonstrate that MIDAS significantly outperforms related baselines in addressing modality imbalance.

MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
Seong-Hyeon Hwang"
Soyoung Choi" Steven Euijong Whang
KAIST
{sh. hwang, wer07081, swhang) ac.kr
Abstract
Multimodal models often over-rely on dominant modalities, failing to achieve opti- mal performance. While prior work focuses on modifying training objectives or optimization procedures, data-centric solutions remain underexplored. We propose MIDAS, a novel data augmentation strategy that generates misaligned samples with semantically inconsistent cross-modal information, labeled using unimodal confidence scores compel learning from contradictory signals. However, this confidence-basse labeling can still favor the more confident modality. To address this within our misaligned samples, we introduce weak-modality weighting. which dynamically increases the loss weight the least confident modality, thereby helping the model fully utilize weaker modality. Furthermore, when misaligned features exhibit greater similarity to the aligned features, these misaligned sam- ples pose greater challenge, thereby enabling the model better distinguish between classes. To leverage this, we propose hard-sample weighting, which prioritizes such semantically ambiguous misaligned samples. Experiments mul- tiple multimodal classification benchmarks demonstrate that MIDAS significantly outperforms related baselines in addressing modality imbalance.
1
Introduction
The human ability to perceive and interpret the world is inherently multimodal, integrating information from visual, auditory, and textual cues to form complete understanding. Motivated by this human perception, multimodal learning has gained significant attention in artificial intelligence research [ This approach has led to breakthroughs across domains, including rision-language modeling (3] medical diagnosis [4], and autonomous systems (5]. In spite of its success in various areas, one of the fundamental challenges is imbalanced multimodal learning (6). where models tend to rely
the more informative modality while neglecting the weaker ones. Modality imbalance leads models toward degraded overall performance, even worse than unimodal models (7).
While extensive recent studies
10 have addressed the multimodal imbalance problem, they have largely overlooked the importance of data and feature processing. Most studies solve the problem through designing new training objectives or optimization strategies using only aligned samples, such as re-weighting each modality importance 7 adjusting the direction and magnitude of gradients [TT. Even if a few studies propose data or feature-Ievel solutions, these approaches typically rely on masking [12] or zero-vector substitution [13]. Such approaches intentionally limit information use and fail to fully exploit the information embedded in multimodal data.
To overcome this limitation, we introduce a different data-centric perspective: leveraging misaligned samples as informative data that can reveal and help address modality imbalance, rather than noise or outliers [4]. A misaligned sample is formed by combining modalities from different original
'Equal contribution.
"oorespooniig author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

100
0.8
Joint training Ours
Joint training Audio
Ours Audio Ours Video
Confidence Score
80-
+9.3%
0.6
Joint training - Video
Accuracy (%)
60
0.4
40-
+19.6%
0.2
20
0.0
0
10
20
30
40
50
60
70
0
Aligned data
Misaligned data
Epoch
(b)
Figure 1: (a) Accuracy comparison between Joint training and our method on aligned (original) and misaligned validation data. (b) Comparison of modality confidence scores between Joint training and our method when predicting misaligned validation data on the Kinetics-Sounds dataset.
samples (e.g., an image of dog paired with text describing a cat), with its original labels. This structure preserves the features of the original samples, enabling full exploitation of multimodal representations. More importantly, because aligned samples associated with single label shared across all modalities, it is difficult to determine whether the model is truly using all modalities or relying on just one. In contrast, misaligned samples contain modality to different labels, making it possible to diagnose modality imbalance by analyzing the model's s prediction. Ideally, a model trained to truly leverage information across modalities should able to identify content related to all modalities, even when the input is synthetically misaligned.
information pointing
To investigate whether a standard multimodal model has this capability, we evaluate its accuracy on aligned and misaligned samples from the Kinetics-Sounds dataset [15]. For aligned samples, accuracy measured using the top-1 predicted class, while for misaligned samples, it is measured based on whether both ground-truth labels appear within the top-2 predictions. As shown in Figure Ta the multimodal model exhibits low accuracy on misaligned samples (6.3%) compared to aligned data (65.5%), revealing its biased reliance on a specific modality. Furthermore, Figure [ shows that the model consistently predicts the class associated with the dominant modality (audio, in this case) with high confidence when facing misaligned inputs.
Thus, we propose MIDAS, novel modality to mitigate modality imbalance (see Figure MIDAS systematically generates misaligned samples by pairing modalities from different source instances, where each modality has different labels. The key idea is to enforce the model to recognize conflicting signals within a single input. To assign meaningful label for misaligned samples, we take weighted average of labels based the confidence scores of each unimodal classifier, since the modalities are not equally informative (10).
agnostic modality-agnostic
multimodal data augmentation strategy designed
However, the model still relies heavily the stronger modality, more confident modalities contribute more to the target label. To supplement this, we first propose weak-modality weighting. which increases the loss contribution of the least-confident modality. This adjustment counteracts the labeling and allows the model to better attend to underutilized equally useful. Higher similarity between the
weakness of unimodal confidence-basee sources. Moreover, not all misaligned samples swapped and original embeddings makes the semantic conflict more subtle, causing these samples to be more challenging and informative. Thus, we employ hard-sumple weighting, which emphasizes such difficult samples to improve class discrimination. Together, these techniques enhance the model's 's ability learn balanced representations from misaligned data.
We conduct comprehensive evaluations of MIDAS on multiple real-world multimodal datasets for classification. The results demonstrate that MIDAS effectively enhances modality utilization and outperforms existing approaches for addressing imbalanced multimodal learning. Our findings highlight that MIDAS is the first data augmentation method to construct misaligned pairs with distinct labels and advanced weighting techniques.
Summary of contributions: (1) We propose new modality-agnostic data augmentation method based on misalignment for multimodal imbalance learning; (2) We introduce novel labeling strategy and two weighting mechanisms to maximize learning from misaligned data; (3) We show that MIDAS outperforms related baselines via extensive experiments.

---

Input
Encoder
Feature
Classifier
Output
Loss
Confidence-bused Label yi
Modality|
Modalityy2
Aligned sample 24
Uni
p.= LCE (pt...)))
Vi Vj Mk
Weak-Moddality Weight (+)
[ f2
pi Caliga (at, Mb)
p;
Multi
Misaligged sample
p. Cais (Z Dii
Vi
V3
yo
f3
Pj
Hard-Sample Weight 8
Uni
pt LCC
Figure 2: MIDAS trains multimodal model on both aligned and misaligned samples with conflicting semantics simultaneously. MIDAS consists of three main components: 1) We label misaligned samples with confidence-based labeling strategy using unimodal classifiers. 2) Weak Weak-modality weighting increases the loss weight of the least confident modality. 3) Hard-sample weighting assigns higher loss weight more confusing misaligned samples containing similar semantics.
2 Related work
Imbalanced Multimodal Learning The disparity in learning effectiveness between modalities
key challenge in multimodal learning. Many studies 77 [6 17
[8]
6
[
19
20
21 [2] have
highlighted this problem with two representative directions using only aligned samples: optimization- based strategies that adjust training dynamics, and data/feature- Aptimization-base techniques such as AGM [8] and OGM [16] aim to
approaches that manipulate
inputs or representations. reduce the gradient strength for dominant modalities based on performance or Shapley value estima- tions. However, they often introduce additional overhead through complex steps, including repeating gradient computation. In parallel, data and feature-centric approaches like selective resampling [13] and adaptive masking 12 aim to support weak modality by generating weak modality-specific data or masking dominant modality features, respectively. While effective, they often perturb data or features and fail to fully leverage information from the original data. Recently, few studies have used misaligned pairs to inject cross- mrosll information. LFM [1] leverages misaligned data points negative samples in contrastive learning, and MCR (3 employs within-batch permutations to estimate conditional mutual information. However, these methods remain unsupervised learning schemes, limiting their applicability to downstream tasks. In contrast, approach treats misaligned data as supervised training signals, enabling us to leverage the original features in addition to the misaligned relationship between modalities for imbalanced multimodal learning.
Multimodal Data Augmentation Although data augmentation has greatly improved unimodal learning, its application to multimodal settings remains relatively underdeveloped. Recently, a few studies 24 25 26 27 have focused on augmentation techniques for multimodal data. Yet many of these methods are either modality (25] 28 29] or apply uniform transformation to all modalities [27]. failing to account for the varying influences that different modalities exert learning. For instance, MixGen (25) only augments image-tex pairs by interpolating images and concatenating the corresponding texts. In contrast, PowMix (24] introduces a regularization strategy that interpolates latent representations with anisotropic mixing and adjusts their contributions. However, it does not explicitly address modality imbalance. On the other hand, our method focuses on imbalanced multimodal learning, while considering diverse and complementary cross-modal information 11.
Method
In this section, we detail our approach for solving the multimodal imbalance problem. We first introduce the basic setup and notation in Sec. 3.1 Then, we describe our core mechanisms generating misaligned samples (Sec. 3.2p. unimodal confidence based sample-level labeling (Sec. (.3), weak-
modality weighting (Sec. 3.4 and hard-sample weighting (Sec. 3.5). Finally, we present the overall

---

training objective combining these elements in Sec. 3.6 For simplicity, we consider a two-modality setting (M=2) in this paper, such as image-text pairs. We briefly discuss how these concepts generalize to scenarios with more than two modalities in the Appendix (Sec. A.2).
3.1 Preliminaries
We consider dataset D = [2 w:)]/= where each input 2; = (x),... .r(') consists of M modal-
ities and y/ E {I....,Cl C} is the class label. Each modality x;" is mapped to a feature f,"h = Pm (I.7 by dedicated encoder fm the set of encoders is y {pmm
These features (f)
S. M)
are processed by a multimodal classification layer 9s produce logits 24 = gr ................................ ).M.
yielding predicted probabilities p; = softmax(z.). In parallel, M unimodal classification layers {gm)m=1 process individual features f!" to yield unimodal logits z!"h = 9m (f?") and probabilities P."h = softmax(2""" Crucially, the encoders yp are shared across both multimodal and unimodal
pathways. Unless stated otherwise, we use the standard cross-entropy loss, LCE, for training.
3.2 Generating Misaligned Samples
To reduce the tendency of the model to over-rely on dominant modality, we systemically generate misaligned samples that contain conflicting semantic information across modalities. Multimodal models trained only on aligned data with hard labels often learn "shortcuts", focusing excessively on the most informative modality while ignoring the others 7 211. To address this imbalance, we explicitly leverage such misaligned samples, which have generally been treated as noise or outliers 14). By requiring the model to interpret information from all modalities within these misaligned samples, we encourage the model to develop a more balanced reliance on each modality.
Consider an aligned sample =(z x?) with label yr- Within the same mini-batch, we randomly select another sample æy = (x)) x such that yj * yi- We adopt random replacement due to its two advantages: computational efficiency and improved model generalizability. To substantiate this choice, we provide comparisons with alternative diisaligned-sampp generation strategies in Appendix (Sec. A.4. A misaligned sample i; is then constructed by swapping one modality as follows:
(5), 23) = 23)
(1)
This 2e combines the first modality from sample with the second modality from sample j. For
example, if x; is an image of a "cat" (y/- "cat"" and 2² is text description of a "dog" (y, "dog"), )) the misaligned sample ã. (xt,={) a²) pairs the "cat" image with the "dog" text. Symmetrically, we also generate and utilize (a). x?))
3.3 Unimodal Confidence based Sample-level Labeling
Supervising the model with the generated misaligned sample á, presents a challenge: determining an appropriate target label for ž. Thus, we generate a label E Eº for each ž, by using a sample-level labeling strategy. Since Σ; combines two modalities from different source classes, using a single hard label for either class is inappropriate. On the other hand, naïve average of source labels ignores valuable supervision or inadvertently overweights a less informative modality [10.
Our key idea is to compute ÿ by evaluating how confidently each unimodal classifier predicts the original label associated with its respective modality. Specifically, we use the unimodal classifiers (g) and g2) to estimate the confidence scores for their corresponding source labels, based on the individual components (x: and u²") of the misaligned sample. This approach draws inspiration from prior work that utilizes unimodal output probabilities to estimate modality importance or reliability [I 30 Furthermore, sample- labeling is crucial because even within the same modality, the amount of discriminative information can vary across samples [3].
Given a misaligned sample ă; (x), x3)) we obtain the unimodal output probabilities p softmax(91 (11 (z)))) and p) = softmax (g2 (x)(()))). Using the probabilities, we calculate the confidence scores for the original class labels of each modality: (p)),, (confidence of modality on label y:) and (p)),, (confidence of modality on label 8j), where (+)x indicates k-th components of the input. These confidences are normalized as:
(p!)
(p))
ē =pt) +p3)),
(2)
(p:).. p. +V)))

---

Then, the target label ÿ is weighted average of the one-hot encoded source labels as follows:
M= Ys.m = 2 + e
=
where ; E {0, 13 are the one-hot vectors for labels y, and yj, respectively. For example, if (p)),, 0.9 and (p²),,, 0.3, then the normalized confidences are दे 0.75 and è 0.25. In this case, the target label is YJ = 0.75y; + 0.25y,, The ü; reflects the relative contribution of each modality in the misaligned sample, as estimated by the unimodal classifiers. Then, the loss for misaligned sample é; with its label Ù is defined as:
mis (3 D.) LC = (e (yg)c) log((())))
(4)
where ir = softmax (})())) is the output probability of the multimodal classifier for T.. In practice, we implement warm-up phase train encoders and unimodal classifiers for each modality before the labeling process. During this phase, encoders and unimodal classifiers are trained with aligned data ensure that confidence scores reflect meaningful modality reliability. Without this step, unreliable confidence estimates could mislead the labeling and downstream weighting process during the early stages of training.
3.4 Weak-Modalit Weighting
To further prevent the underrepresented modality from being overshadowed in misaligned samples, we introduce a weak-modality weight = ( While the target label 5t provides supervisory signal for the misaligned sample T,, a standard cross-entropy loss (CF De) may still be insufficient to suppress the model's s tendency to rely on the dominant modality. Even when optimizing toward y = y + o,y,, the multimodal model g could primarily focus on the component of the modality with higher confidence (i.e., max(?!, (()) potentially undervaluing the signal of the other modality. Thus, the weak modality weight dynamically increases the loss weight for the least confident modality when the multimodal model's 'ss normalized confidence for the class associated with that modality falls below the corresponding unimodal confidence used to generate the target label.
We first identify the least confident modality in whose unimodal classifier shows the lowest average confidence across batch of misaligned samples B follows:
m arg min E, (3...))~~88 [2
(5)
(1(1)2)
We initialize weights a (4) to 1, and only update the weight of the identified least confident modality, a at each batch iteration t. This update mechanism compares the target label associated with modality in (i.e., 2rh) within the misaligned label against the multimodal model's predicted confidence for the corresponding source class from Σ. Let p. = softmax(g) (((tma) be the
multimodal prediction for T. and y; ((h) be the target label for the modality in in the sample ã, (e.g..
if ã; (x] x²)), then //"h) y; and 57.2) = ,2). To accurately assess the model's confidence for the specific source class associated with the modality in in the misaligned sample, we calculate the normalized multimodal confidence for class 81"h as (2))gm g.²) / ((pt). + (Pt) ). The update signal A, is computed as the batch-averaged difference between the target label and this normalized predicted confidence:
D sign
E B
[cp]]
Eg ((C))
(6)
where the expectations are over the current batch of misaligned samples B. The weight o updated as follows, while the weight for the other modality remains 1:
(t+1)
is
(t+1) max a A..)) (1+1) =1
an
where 1 > the step size. the model under-predices the contribution of modality in (i.e., Ao >> 0), ama increases, thereby amplifying its influence in the misaligned sample loss. For example, the normalized unimodal confidence of the least confident modality ńn is 2 0.25, and the normalized multimodal confidence for its corresponding class is only (4) 0.10. the multimodal model underestimates the importance of the modality m. In this case, as becomes larger than 1. The loss for misaligned samples with (!) will be presented in Sec. 3.6
5

---

3.5 Hard-Sample Weighting
We also propose a hard-sample weight š, which modulates the influence of a misaligned sample i; The intuition is that not all misaligned samples equally informative. If the swapped-in feature is highly similar to the original feature despite originating from a different class, the semantic conflict might be difficult for the model to discern. Thus, encourage the model to focus more the misaligned samples composed of more similar semantics, thereby improving its capacity to capture fine-grained feature representations. The hard-sample weight is based on how similar the feature embedding of the swapped modality is to that of its original counterpart from the same source sample.
Consider the misaligned sample é; (z) 22)), where æ3 (from sample j. label yj) replaces the original xi (from sample label y: # yy). We compare the feature vector of the original modality, f? = (x()). ), with the feature vector of the swapped-in modality, f3 (2(x)) The &; is calculated as the cosine similarity between these two features:
Sf 11 12
8;
(8)
This weight 8; E [-1, quantifies how similar the swapped-in modality feature (f}) is to the feature it replaced (f??). A higher & suggests a more subtle semantic difference between the components involved in the swap for the second modality. As detailed later in Sec. (3.6] this weight modulates the loss contribution of the misaligned samples. (If the other type misaligned sample I; = (z). x?)
is generated, the similarity would be calculated between f and ʃ} ( (x))). The harder the misaligned sample, the higher its loss weight.
3.6 Overall Training Objective
Our final training objective combines the standard supervised loss from aligned samples with the weighted supervised loss from misaligned samples. For aligned data (I4.3s), the loss Latign for multimodal model and Luni for unimodal models are represented as:
Calign (T,, M4) yc)(p..y);
(ami y:) = (ce(p). + (CE(p?
(9)
As mentioned in Sec. 3.3 we train encoders and unimodal classifiers prior to training multimodal classifiers using Luni during the warm-up phase.
The final objective Lmis for a misaligned sample ã; (i.e., (x],,))) is calculated using the confidence- based label ūt the weak-modality weight (!),, and the hard-sample weight 5, follows:
5; + 1
Cmis (ā,, (m, Dri (1) (,,))) ) = (1 +
(C (()))))))))
2
(10)
3; +
19 + ())(()))c) log((())))
(1
2
The total loss Liotal is averaged over mini-batch B:
Liotal
Σ
Catign (Ie, Bi) + (ani vi) + ALmis Fe (0,a09),))) 8:)
(11)
B
(BL.)))
The generation of ã., the computation of J. and 8,, and the update (t) occur dynamically within the training loop. The model parameters (1 3.39 ty. {9m}) are updated by minimizing Crotal. The overall algorithm of our method is given the Appendix (Sec. A.Ip
In addition, we analyze the computational complexity of MIDAS. For instance, in a two-modality setting (M = 2). MIDAS generates two misaligned samples per original sample x;, As this adds only constant number of additional samples per I;, the overall computational complexity remains O(N), where is the number of training samples. While increasing the number of modalities M would lead to generating more samples, M is typically small (2 3) in real-world settings.

---

4 Experiments
We provide experimental results for MIDAS, evaluating its performance on multimodal classification tasks in the presence of the imbalance modality problem. We report the mean and standard deviation (+) across three independent runs with different random seeds for all experiments. All experiments are conducted using NVIDIA GeForce RTX A6000 and Quadro RTX 8000 GPUs.
4.1 Experimental Settings
Datasets We evaluate our method and basclines on four widely used benchmarks for imbalanced multimodal learning, each exhibiting varying degrees and types of modality characteristics: Kinetics- Sounds [1] a dataset linking audio and video clips for action recognition with 31 classes. CREMA-
D is an audiovisual dataset for emotion recognition featuring actors speaking sentences with 6 classes. UCF-101 [22 an action recognition dataset consisting of RGB frames and optical flows with 101 classes. F0od-101 33 is dataset of food images paired with their corresponding textual recipes with 101 classes. Additional dataset statistics are summarized in the Appendix (Sec. A.3
Metrics We report the Top-1 Accuracy (Ace) and F1-Score (F1) as our primary evaluation metrics in percentages following [6] Accuracy measures the overall classification correctness, while F1-Score provides a balanced measure between precision and recall, which is particularly relevant in cases of class imbalance or varying difficulty across classes. For both metrics, higher is better.
Implementation Details We conduct experiments following the configurations in ( For Kineties- Sounds and CREMA-D, we use ResNet-18 88 [34] encoders for both audio and video, training from scratch. For UCF-101, we also use ResNet- 18 as encoders. For the Food-101 dataset, we use a pre-trained ResNet-18 and a pre-trained ELECTRA 35 as image and text encoders, respectively. More detailed configurations are provided the Appendix (Sec. A.3).
Baselines We compare MIDAS against the following related baselines, which manipulate features or generate new types of data for imbalanced multimodal learning: 1) Joint training is a vanilla multimodal learning technique; 2) SMV [13] re-samples data from low-contributing modalities based on sample-level modality valuation; 3) OPM (36) modulates features by weight multiplication to adjust the contribution of each modality by predicting per-sample modulation weights during training: 4) AMCo [12) applies adaptive masking on the features of the dominant modality to adjust the learning difficulty: 5) LFM [21] combines multimodal learning with contrastive learning through dynamic integration; 6) MCR 23 leverages misaligned features for unsupervised learning and uses game-theoretical regularization to balance the contributions of modalities.
4.2 Comparison with Baselines
We compare the performance of MIDAS with existing related baselines across four benchmark datasets. As shown in Table MIDAS consistently outperforms the baselines across all datasets in both accuracy and F1 Fl-score. Specifically, MIDAS achieves significant improvements in accuracy the Sinetics-Sounde (3. (3%))) and CRÉMA-D (4.08%p1) compared to the best- cest-performing baselines trained solely aligned samples. Furthermore, MIDAS is effective not only on audio-video pairs (representative examples of imbalanced modality pairs) but also on other modalities, including image, text, and optical flow, demonstrating its versatility. While MIDAS shows comparable performance to SMV on the UCF-101, it is worth noting that SMV has more epochs and opportunity to learn due approximately 3x more generated data. In addition, compared to LFM and MCR, which utilize misaligned samples in unsupervised or contrastive learning frameworks, MIDAS achieves superior performance through supervised learning. By providing explicit labels derived from unimodal confidences, MIDAS offers direct supervision to the model, enabling more effective representation learning compared these less direct (unsupervised or contrastive) approaches.
4.3 Ablation Study
To understand the contributions of each component in MIDAS, we evaluate the impact of three key elements: the warm-up phase (W) for unimodal classifiers, weak-modality weighting (WM), and hard-sample weighting (HS) on all datasets. As shown in Table 2 each component individually
7

---

Table I: Performance results comparing MIDAS against related baselines four multimodal datasets. The best and second-bes -bet results are highlighted in bold and underlined, respectively.
Kinetics-Sounds
CREMA-D
UCF-101 Acc (t)
Food-101
Method
Acc (1)
Fl (f)
Acc (1) FI (t)
FI (1)
Ace (t)
Fl (1)
Joint training
63.9210.23
55.54.0..0
1. 1.16 58.60
90.07
83.8 1.55
91.3511.17
05.75
SMV 13
65.7611.48 57. 1.56
6.79
66,87 86.8311.89
95.24
91.8010.86 10.88
91 01.644...28
86,2610.54
OPM 36
67.3520..7
59.29 19.20=00100 70
63.97=1.72
62.71 68
91.73:20.51
86.4220.66
92.40
97.41=0.50
AMCo
12
67.04+0.68 58.41=1.05
69.91=322
68.85+4.25
93.77=0.51
89.46+0.79
92.00+0.22
86.79=0047
LFM
64.8810.61 56.3011.00
64.02_ 1.82
62.50.2.24
91.86100.4
86,4 86.4810.61
92. 12.034 86,48.0.49
MCR
23
71.75+0.58 64.23=0.00 4a
70.21
10-19-1117
91.8420.27
86.2740.46
90.58+0.26 84.62=0.38
MIDAS
74.88
67.18=1:225
10.9
73.82.10.55
95.20 18 91.6610.57
93.4620.330 89.02.0.62
Table 2: Ablation study on four multimodal datasets.
Kinetics-Sounds
CREMA-D Acc (1)
UCF-101 Acc (1)
Food-101
W WM HS
(t)
F1 (?)
FI (f)
FI (t)
Acc (1)
F1 (t)
X
X
71.70=0.61 63.4740.36
72.3220.52 70.94
94.1620.08 90.1 720032
93.3940.19
88.91
X
72.32=0.52 64.04+0.81
71.25+1.87
70.28+1.79
94.97 #0.41 91. 40
93.46+4..2
89.02 59.02=#0.4
X
72.84=0.41 64.76+0..3
12.28+1.14
71.03+1.14
94.66
91.03=0.224
93.424+.17
89. $9.00=0.30
V
71.86=0.69 63.41+1.10
72.35#1.45 71.07
94.19=0.46
90.0420.64
#0.51 #9.21:00.47
V
V
x
73.83=0.71 65.54411.6
72.68+1.42 71.43++142
95.11 #0.29
91.63++0.43
93.45-
89.02 46
V
X
V
73.17
64.7710.68
73.7611144
72.60.11.7
94.86110.67
91.12:10.1
93.48.00.2
89.00.0..00
X
V
V
73.90
66. 10.1..0
73.98.1.62
72.93.1166
93.95.10.16
89.81
74.41116.4
64. 64.244221.4
V 74.88=0.48 67.18
74.99:0.51 33.82:0033
95.20 91.66=0.57
93.446+0.56
89.02=a
contributes performance improvements, but the gains relatively small when applied in isolation. In contrast, combining all three components consistently leads to the best result, demonstrating a clear synergistic effect. For Food-101, the individual components (W, WM, HS) do not yield additional performance gains, possibly because MIDAS leveraging misaligned samples already achieves strong performance. Overall, MIDAS clearly improves or is at least as good as the related baselines. These findings confirm that the components complement each other and that their integration is crucial to maximizing the effectiveness of learning from misaligned samples.
4.4 Comparison with Existing Data Augmentation Methods
In addition to imbalance-specific techniques, we also compare our method with modality- multimodal data augmentation strategies, including Mixup (37].
Table 3: Performance evaluation of MIDAS against existing multimodal data augmentation strategies.
CREMA-D
Food-101
Method
Acc (1)
Fl (1)
Acc (1)
Fl (t)
PowMix (24), and LeMDA 27
Joint training
60.28
58.6011.14
91.35
85.75
on the CREMA-D and 101
Mixup 37
61.8443.39
60.38+3.59
91.36+0.28
85.80=0.63
datasets. While Mixup is origi- nally designed for unimodal data, we extend to the multimodal set- ting by interpolating each modal- ity's embedding independently and computing the label as weighted average of the original labels, following the standard Mixup formulation. As shown in Table MIDAS consistently outperforms data augmentation baselines in both accuracy and F1 Fl-score. These results indicate that typical multimodal data augmentation strategies are insufficient for handling modality imbalance, and they sometimes even show worse ac- curacy than Joint training. In contrast, MIDAS leverages misaligned samples with targeted weighting schemes, allowing the model to learn better from underutilized modalities. This demonstrates the importance of incorporating imbalance-aware design into multimodal data augmentation.
PowMix 22
63.6641125
62.3241.21
89.59+2.73
83.2144.07
LcMDA 27
58.13:2274
56.75+2.45 45
91.19:0.21
85.56=0-43
MIDAS
74.99.0.31
73.82
93.11
88.48:0.60
4.5 Additional Analysis
Efficacy of weak-modality weighting To validate the efficacy of the modk-ity weighting, present the averaged normalized confidence scores of training pairs as training progresses on
8

---

Audio Uni
Audio Multi
Audio Uni
Audio Multi
Normalized Confidence Score
Video Uni
Video Multi
Normalized Confidence Score
Video Uni
Video Multi
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.0
10
20
30
40
60
70
20
30
40
50
60
70
Epoch
Epoch
(a)
(b)
Figure 3: Normalized confidence score comparison between (a) MIDAS without the weak-modality
weighting and (b) MIDAS with the weak-modality weighting on the CREMA-D dataset.
Table 4: Performance comparison between cosine similarity and L2 distance across three datasets.
Kinetics-Sounds
CREMA-D
Food-101
Method
Acc (1) F1 (1)
Acc (1) Fl (1)
Acc (f) F1 (f)
L2 distance
74.69
67.00
74.26
73.36
93.79
89.54
Cosine similarity (Ours)
75.26
68.34
75.00
74.05
93.82
89.56
the CREMA-D dataset. In Figure 3 "Uni" represents the confidence scores from the unimodal classifiers (L ش B (eT")) and "Multi" represents the confidence scores from the multimodal classifier (Bg, (c: g.n D) The normalized confidence scores are measured when classifiers predict the misaligned training samples. As shown in Figure 3al without the beak-modality weighting, the normalized confidence score gap of the multimodal classifier between audio (dominant modality) and video becomes larger than the target normalized confidence scores. The reason the larger gap is that confidence-based labeling still assigns the larger value to the loss regarding the dominant modality, as we point out in Sec. Sec.3.4 In contrast, with the weak-modality weighting, the normalized confidence scores of the unimodal classifiers and the multimodal classifier become consistent within each modality, as shown in Figure 3b This means that the weak-modality weighting leads the multimodal classifier predict the misaligned samples closer to the target labels.
Trend of weak-modalin weight We analyze the tra- jectory of the reak-modality weight a throughout the training process on all datasets. As illustrated in Fig- ure after the warm-up phase, a increases to allocate greater weight to the undervalued modality. Notably, a converges as training progresses, indicating that it effec- tively captures the relative importance of each modal- ity while maintaining an appropriate balance between them, without diverging.
4.0
Kinetics-Sounn CREMA-D
UCF-101
3.5
Food-101
3.0
t
2.5
2.0
1.5
1.0
10
20
30
40
50
60
70
Similarity metric study for hard-sample weight- ing We compare cosine similarity for hard-sample weighting with L.2 distance, a widely used distance metric. Cosine similarity generally preferable for high-dimensional features due to its scale invariance and stronger discriminative properties To empirically validate this choice, we conduct experiments where L2 distance replaces cosine simi- larity in our hard-sample weighting. As shown in Table cosine similarity consistently outperforms L2 distance terms of accuracy and Fl-score. These results confirm that cosine similarity is more suitable for our hard-sample weighting based on high-dimensional feature similarity.
Epoch
Figure 4: Trends of weak-modality weight during training on four datasets.
38
39
Modality confidences during training We also provide modality confidence of Joint training and MIDAS when predicting misaligned validation data training progresses on the CREMA-D dataset

---

0.8
1.0
Joint training A
MIDAS A
Joint training OF
MIDAS OF MIDAS
Confidence Score
0.6
Joint training
MIDAS
Confidence Score
0.8
Joint training
RF
0.6
0.4
0.4
0.2
0.2
0.0
10
20
30
40 Epoch
50
60
70
0
10
20
30
40
50
60
70
Epoch
(a)
(b)
Figure 5: Model confidence curves of Joint training and MIDAS for each modality on (a) CREMA-D (audio, A; video, V) and (b) UCF-101 (optical flow, OF: RGB frame, RF) datasets.
MIDAS predicts the misaligned samples based on two modalities in balanced manner while Joint training solely relies on the audio. The additional confidence curves for the Sinetics-Sounds and food-101 datasets are in the Appendix (Sec. (Sec.A..4).
and UCF-101 dataset. As shown in Figure 5
Accuracy for misaligned samples We show the accu- racy of MIDAS on misaligned validation data across all datasets, compared Joint training, F For all datasets, our method yields substantial improvements over Joint training, demonstrating its robustness under semantic inconsistency. This suggests that method effectively captures modality-invariin representations, enabling the model to maintain reliable predictions even when semantic cues are partially misaligned.
100
Misaligned Accuracy (%)
Joint training
Ours
80
60
+48.6%
+29.8%
40
+19.6%
+13.4%
20
Kinetics CREMA-D UCF-101 Food-101
Experiments on trimodal dataset To demonstrate the scalability of our method, we further conduct exper- iments on the CMU-MOSI dataset [40]. which contains three modalities: text, audio, and video. This dataset is widely used for multimodal sentiment analysis, consist-
Figure 6: Accuracy for misaligned vali- dation samples comparison between Joint training and MIDAS.
ing of two classes: positive and negative. We use 1,284, 229, and 686 pairs for training, validation, and testing, respectively. We follow the experimental settings introduced by [6] and [41]. MIDAS achieves 74.00% accuracy and 73.64 Fl-score, outperforming Joint training (71.13% accuracy and 70.86 Fi-score). These results indicate that our method is not limited to simpler bimodal tasks but can effectively scale to more complex multimodal environments.
Conclusion
In this paper, first identify misaligned samples key signal for exposing and addressing modality imbalance in multimodal learning. Based on this insight, we propose MIDAS, a novel data augmentation approach that constructs misaligned samples by pairing modalities from semantically different data points and labels them based on the confidence of unimodal classifiers. To strengthen the model's s ability to extract useful information from these difficult samples, MIDAS incorporates two weighting mechanisms. First, weak modalttyyy weighting amplifies the contribution of the least confident modality. thereby encouraging the model utilize underrepresented signals during training. Second, hard-sample weighting prioritizes semantically ambiguous samples, making the model learn more effectively from misaligned samples. Extensive experimental results on various multimodal datasets demonstrate that MIDAS outperforms the baselines using both weighting strategies, con- firming its effectiveness as data-driven solution for mitigating the multimodal imbalance problem.
Limitations: While MIDAS demonstrates significant improvements in imbalanced multimodal learning, the current framework focuses primarily on classification tasks only. Extending applicability to other multimodal tasks, such as generation retrieval, could require further adaptations.
10

---

Acknowledgement
This material is based on work that is partially funded by an unrestricted gift from Google. This work supported by the Institute of Information & Communications Technology Planning & Robust,
Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. Fair, Extensible Mata-Centric Continual Learning). This work was supported by the Institute of Information Communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (No. RS 24244-000044462, Non-invasive near-infrared based Al technology for the diagnosis and treatment of brain diseases).
References
[1] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.0344 2022.
[2] Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE signal processing magazine, 34(6))96-108, 2017.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[4] Tzu-Ming Harry Hsu, Wei-Hung Weng, Willie Boag, Matthew McDermott, and Peter Szolovits. Unsupervised multimodal representation learning across medical images and reports. arXiv preprint arXiv:1811.08113 2018.
[5] Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, and Antonio M López. Multimodal end-to-end autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23(1):537-547, 2020.
[6] Shaoxuan Xu, Menglu Cui, Chengxiang Huang, Hongfa Wang, and Di Hu. Balancebenchmark: survey for multimodal imbalance learning. arXiv preprint 2rXiv:2502.10816, 2025.
[7] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22695-1275,, 2020.
[8] Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting mudti-modal model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 22214-22224, 2023.
[9] Yunfeng Fan, Wenchao Xu, Haozhao Wang. Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20029-20038, 2023.
[10] Yake Wei, Siwei Li, Ruoxuan Feng. and Di Hu. Diagnosing and re-learning for balanced multimodal learning. In European Conference Computer Vision, 2024.
[11] Yake Wei and Di Hu. Mmpareto: boosting multimodal learning with innocent unimodal assistance. International Conference on Machine Learning, 2024.
[12] Ying Zhou, Xuefeng Liang, Shiquan Zheng, Huijun Xuan, and Takatsune Kumada. Adaptive mask co-optimization for modal dependence in multimodal learning. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1-5. IEEE, 2023.
[13] Yake Wei, Ruoxuan Feng, Zihe Wang, and Di Hu. Enhancing multimodal cooperation via sample-level modality valuation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 27338-273A4 2024.

---

[14] Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen. Di Hu, et al. Multimodal fusion on low-quality data: A comprehensive survey. arXiv preprini arXiv:2404. 18947, 2024.
[15] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, pages 609-617, 2017.
[16] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly fly gradient modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8238 8247, 2022.
[17] Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof Geras. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference Machine Learning, pages 24043-24055. PMLR, 2022.
[18] Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 18:1650-16544 2021.
[19] Cong Hua, Qiangian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modality reconcilement. arXiv preprint arXiv:2405.093211 2024.
[20] Xiaohui Zhang, Jachong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2024.
[21] Yang Yang, Fenggiang Wan, Qing- Yuan Jiang, and Yi Xu. Facilitating multimodal classification via dynamically learning modality gap. Advances in Neural Information Processing Systems, 01661088621222 2024.
[22] Zequn Yang, Yake Wei, Ce Liang, and Di Hu. Quantifying and enhancing multi-modal robustness with modality preference. In International Conference on Learning Representations, 2024.
[23] Konstantinos Kontras, Thomas Strypsteen, Christos Chatzichristos, Paul Liang, Matthew Blaschko, and Maarten De Vos. Multimodal fusion balancing through game- game-theoretic regular- ization. arXiv preprint arXiv:2411.07335, 2024.
[24] Efthymios Georgiou, Yannis Avrithis, and Alexandros Potamianos. Powmix: A versatile regularizer for multimodal sentiment analysis. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:5010-5023, 2024.
[25] Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 379-389, 2023.
[26] Teng Wang, Wenhao Jiang, Zhichao Lu, Feng Zheng, Ran Cheng, Chengguo Yin, and Ping Luo. Vlmixer: Unpaired vision-language pre-training via mross cutmix. International Conference on Machine Learning, pages 22680-22690. PMLRR 2022.
[27] Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, and Andrew Gordon Wilson. Learning multimodal data augmentation in feature space. In International Conference on Learning Representations, 2023.
[28) Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, and Austin Re- iter. Cross-modal retrieval augmentation for multi-moda classification. arXiv preprint arXiv:2104.08108 2021.
[29] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 6023-6032, 2019.
[30] Chengxuan Qian, Kai Han, Jingchao Wang. Zhenlong Yuan, Rui Qian, Chongwen Lyu, Jun Chen, and Zhe Liu. Dyncim: Dynamic curriculum for imbalanced multimodal learning. arXiv preprint
2025.
12

---

[31] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd- sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
[32]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv: 2012.
[33] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference Multimedia & Expo Workshops (ICMEW), pages 1-6, IEEE, 2015.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[35] Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. Electra: Pre- training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020.
[36] Yake Wei, Di Hu, Henghui Du and Ji-Rong Wen. On-the-fly fly modulation for balanced multi- modal learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[37] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8766 8763. PmLR, 2021.
[39] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in neural information processing systems, 33:21798 21809, 2020.
[40] Amir Zadeh, Rowan Zellers, Eli Pineus, and Louis- Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis online opinion videos. arXiv preprint arXiv:1606.0622 06259, 2016.
[41] Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning modality-speciffc representations with self-supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10790-10797, 2021.
[42] Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 2515,2515, 2019.
[43] Jianfci Yu and Jing Jiang. Adapting bert for target-oriented multimodal sentiment classification. In International Joint Conference on Artificial Intelligence, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide the abstract and introduction reflecting and supporting the paper's contribution and scope with detailed explanations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We explain that our method is modality-agnostte general with extensive empirical results. We also discuss the computational efficiency at See 3.6 and provide the future limitations after Sec
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
Our paper do not claim theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed configurations for experiments in Sec. 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We use open access datasets for experiments and provide the codes for conducting experiments in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccccuclccuuueeeeeeeeeeeee Coaee//ddeeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed configurations for experiments in Sec. 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide experimental results with means and stand deviations Sec We also plot the figures with error bars in Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the environment for experiments including the type of CPU and GPUs in Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttps:/nneurips.. httpr//sss.ccbbccchhccccccdddddd
Answer: [Yes]
Justification:
We adhere to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We believe that our work deals with the fundamental problems of machine learning in multimodal settings, it have no critical societal impact, which affects to our lives.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not generate any data or models potentially having a high risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite original papers or datasets accurately in Sec
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide assets in the paper and supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We have not used any works related to crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We have not used any works related crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

A
Appendix
A.1 The Algorithm of MIDAS
Algorithm 1: The algorithm of MIDAS
Input: Training dataset D; Number of total epochs E;; Warm-up epochs Ens Batch size Learning rate "y; Hyperparameters X, 17, Model parameters
0 ......... 9 91....19MM............... Initialize modaltmy weights am f 1 for m -,...,,
2 for epoch
do
/ //Warm-up phase for training unimodal classifiers epoch F.w then for each mini-batch Buligned (11,, vi)) D do
P. = softmax (gm (pm (x:"
Sua -
M LCe yt);
Update parameters 1 + 0 7
else
//Main training
Initialize 1, Om
(1) for
M.
10
for B = {(x; M1) -)) D do
//1. Compute losses for aligned samples
11
Compute Luni 8:):
12
Compute Calign (Iz, 34) using Eq.
1/2. Generate misaligned samples
13
{ by random shuffling B
B = =i bi using Eq. 2
14
//3. Compute the loss for misaligned samples
Compute Cmis (ã,, 3e: (1) 34) using Eq. 10:
15
//4. Compute the total loss and update parameters
16
Compute Crotal using Eq. 11
17
Update parameters
VoLuai
//5. Update weak weak-modality weight Identify batch-wise least confident modality mn using Eq.
18
19
Compute average update signal Aa using Eq,
(t+1) + max(1, a + 17 A.) using Eq. Update ag t+
20
21
Output Trained model parameters
A.2
Generalization to M > 2 Modalities
Continuing from Sec. 3 while we focus M=2 for clarity in the main paper, our approach readily generalizes 2 modalities. The core mechanisms remain identical, with summations and operations extending naturally over M modalities.
Generating misaligned samples Generation involves selecting of M source samples K; = (i.ji,......a - with at least two different labels. A random permutation E SM determines the assignment ã; ( ....
..M (immayy) ) where k(.) indexes into K;
Confidence based sample-level labeling Confidences (77 ( are computed for each modality m. Normalization uses ΣM ( Dmmaaa the denominator. The soft label is
J1 Σn. - "kit(m)))"
20

---

Table 5: Summary of datasets used in experiments.
Dataset
#Train
#Val
#Test
#Class
Modality 1
Modality 2
Kinetics-Souud
16,890
2,461
4,778
31
Audio
Video
CREMA-D
5,209
744
1,489
6
Audio
Video
UCF-101
9,159
1.308
2.618
101
Optical Flow
RGB frame
Food-101
63,481
9,069
18,138
101
Text
Image
Weak-modality weighting The least confident modality mn is found using arg mi...e and normalization involves E in the denominator, The up- date signal A. and rule remain conceptually the same, targeting On.-
Hard-sample weighting The set of replaced modalities R; = =mm ((((m))) # i} is identified. The weight š; is the average similarity (if R2 + 0).
1 [R: EmER, sim(fi". .f.)) where y7 # y:
A.3 Experimental Details
Dataset details Table summarizes the number of samples in each split, class counts, and the types of modalities used for each dataset.
we provide additional configurations for imple- menting experiments. We the SGD optimizer with momentum of 0.9 and initial learning rate of le-3 for Kinetics-Sounds, CREMA-D, and Food-101, and le-2 for UCF-101. We use a batch size of 64 across all datasets. Models are trained for 30 epochs for Food-101 and 70 epochs for the other three datasets. We combine the features from different modalities by concatenation. The step size of the StepLR scheduler 15 for Food-10) 101 and 50 for the others. We apply weight decay of lc-4 and a StepLR learning rate schedule for all datasets. We use five workers for all experiments. For MIDAS, we use the hyperparameter of for the Kinetics-Sounae datasets, and for others. We also provide an analysis of the hyperparameter À in Sec. A.4 We use 7 of 5e-2 for all datasets. The best model is selected based on validation accuracy.
Implementation details
Continuing from Sec, 4.1
In practice, generating a misaligned sample at the input level can be computationally expensive it requires passing each input component through its respective encoder for each modality. For efficiency, we implement this process at the feature level. Given batch of samples B = { 9:)} we first compute the features f. = (m(2"") for all and m {1, Then, we construct misaligned feature vector f by combining features from different samples within the batch:
f. (fi.f))
(12)
where and yy are not identical. = (?) is also possible. This feature-level combination reuses the already computed features, significantly reducing the computational overhead compared to processing misaligned samples from scratch. Quantitative comparisons of the efficiency of feature- level augmentation are provided in Sec. A.4
For the CMU-MOSI dataset, a trimodal dataset, we use Transformer encoders for all three modalities, trained from scratch. We employ the SGD optimizer with momentum of 0.9 and weight decay of le-4. The batch size is set to 64, and the models are trained for 70 epochs. A StepLR scheduler with decay of 0.1 and step size of 50 is used. The initial learning is le-2.
A.4 Additional Experiments
during training Continuing from Sec. 4.5 we additionally present the modality confidence curve on the Kinetics-Sounds and Food-101 datasets in Figure 7 Similar to Figure Sa MIDAS utilizes all modalities a much more balanced way when predicting multimodal data compared to the Joint training.
Modality confidence
Hyperparameter analysis As introduced Sec. A.3 to further study the impact the loss weight hyperparameter X, which controls the contribution of the midaligned sample loss term Lmis,
21

---

0.8
1.0
Joint training A Joint training
MIDAS A
Joint training T
MIDAS T
MIDAS V
0.8
Joint training
MIDAS I
Confidence Score
0.6
Confidence Score
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10
20
30
40 Epoch
50
60
70
10
15 Epoch
20
25
30
(a)
(b)
Figure 7:
Model confidence curves of Joint training and MIDAS for each modality on the (a)
Kinetics-Sounds (audio, A; video, V) and (b) Food-101 (text, T; image, I) datasets.
Table 6: Performance evaluation with varying À values across three datasets.
Kinetics-Sounds
CREMA-D
Food-101
A
Acc (1)
F1 (t)
Acc (1) F1 (1)
Acc (f) F1 (t)
0.5
70.43
62.06
73.52
72.26
93.65
89.32
1
71.35
62.87
75.00
74.05
93.82
89.56
73.23
65.05
74.60
73.31
93.57
89.21
5
75.26
68.34
73.25
72.32
65.20
52.11
10
75.34
68.10
63.17
62.58
n/a
n/a
20
74.19
66.12
57.46
57.12
n/a
n/a
conduct an additional analysis on the Kinetics-Sounds, CREMA-D, and Food-101 datasets. As summarized in Table the optimal À value varies depending on the dataset. When À is too small, the model under-utilizes the information from misaligned samples. Conversely, À is too large, the model overly prioritizes misaligned samples and fails to adequately learn shared multimodal representations from aligned data.
Evaluation on additional datasets To further validate effectiveness MIDAS, we additionally evaluate our method on the Sarcasm 42] and Twitter2015 43 datasets, beyond the four widely used datasets included Sec. Both datasets consist of image and text modalities. The Sareasm dataset contains 17.316 samples for training, 2,463 samples for validation, and 4,936 samples for test, while the Twitter2015 dataset includes of 3,736 pairs for training. 534 pairs for validation, and 1,068 pairs for test. As provided in Table 7 the results are consistent with original findings, where MIDAS consistently outperforms competing methods across these datasets, confirming the same performance trend. These findings demonstrate that MIDAS is effective not only standard benchmarks but also on diverse domains such as sarcasm detection.
Efficiency of feature-level augmentation Feature-level augmentation leverages precomputed features after processing aligned samples. In contrast, input-level augmentation requires processing twice as many raw inputs (aligned and misaligned samples), thereby significantly increasing compu- tational cost. To quantify this, measure the training times on the CREMA-D, Kinetics-Sounds, and Food-10 101 datasets. As shown in Table | the feature-level method achieves substantial reductions in training time compared the input-level method. These results confirm the efficiency advantages of our approach.
Study of the misaligned sample generation methods To examine the effectiveness of the random replacement method in terms of model generalizability, we compare our method against a confusion- based replacement strategy, which each sample paired with another whose label among its top-2 predicted classes. This alternative strategy makes misaligned samples inherently challenging or informative for better training. On the CREMA-D dataset, the confusion-hanl based replacement yields an accuracy of 67.67% and an Fl score of 66.79. In contrast, random replacement method achieves
22

---

Table 7: Performance comparison the Sarcasm and Twitter2015 datasets.
Sarcasm
Twitter2015
Method
Acc (f) Fl
Acc (f) F1 (t)
Joint training
86.89
86.38
71.82
64.35
AMCo 12
85.92
85.39
71.44
58.59
MCR
85.33
84.68
72.19
65.17
MIDAS (ours)
87.16
86.58
73.60
65.89
Table Training time comparison (in minutes) between feature-level and input-level methods.
Dataset
Feature-Ievel (Ours)
Input-level
CREMA-D
52
77
SSnenicc-Sounds
311
445
Food-101
278
693
higher performance, with an accuracy of 75.00% and an score 74.05. These results indicate that while confusion-based replacement enforces harder negative pairs, limits generalization and ultimately reduces model performance. In comparison, random replacement provides a better balance between efficiency and generalizability, leading stronger overall results.
Role of modality balance in performance improvement To further investigate whether the performance gains of MIDAS stem from improved modality balancing, we evaluate the contributions of each modality using Shapley value-based modality valuation [13] on the CREMA-D dataset, which consists of audio and video modalities. Specifically, we compare predictions from models trained with joint training against those from MIDAS, while also considering unimodal baselines. For modality-specifif analysis, we replace the features of the non-target modality with zero vectors and then measure performance in the usual way. For example, in an audio-only evaluation, the video features are replaced with zero vectors before computing the results.
The unimodal audio classifier achieves 60.0% accuracy, and the unimodal video classifier achieves 53.7%. When using joint training, the audio-only accuracy drops 42.1 %, and the video-only accuracy further drops to 19.2%, indicating severe under-utilization of the video modality. In contrast, MIDAS achieves much more balanced accuracy contributions with 54.3% for audio and 49.9% for video, leading to an overall accuracy of 73.7%. These results suggest that the substantial performance gain (from 60.2% with joint training 73.7% with MIDAS) largely stems from improved modality balancing rather than a general regularization effect. Regularization alone cannot explain the drastic improvement video feature utilization, which increases from 19.2% to 49.9%, while maintaining strong audio performance. This result confirms that MIDAS explicitly enhances modality balance, which is the key factor behind the observed performance improvements.
23


Reviews:
[{"review_id": "S6CKv8XGKU", "reviewer": "Reviewer_W7Kn", "summary": "This paper introduces MIDAS (Misalignment-based Data Augmentation Strategy), an approach addressing modality imbalance in multimodal learning. The method generates semantically inconsistent cross-modal samples (e.g., pairing a \"cat\" image with \"dog\" text) labeled using unimodal confidence scores, compelling models to learn from conflicting signals. MIDAS incorporates two weighting mechanisms: weak-modality weighting enhances learning of underrepresented modalities through dynamic loss adjustment, while hard-sample weighting prioritizes semantically ambiguous samples to improve discriminative power.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The use of misaligned samples as supervised training signals, along with dynamic weighting mechanisms, introduces a new perspective.\n2. The paper conducts thorough experiments on four multimodal datasets, with results showing improvements over existing methods.\n3. The methodology is described systematically, with clear mathematical formulations and supporting diagrams.\n\nWeaknesses\n1. The impact of key hyperparameters (e.g., loss weight λ) is not systematically studied.\n2. The experimental validation remains limited in scope, failing to include several key benchmark datasets such as Sarcasm, Twitter2015, NVGesture, and VGGSound.\n3. Although empirically effective, the paper lacks a theoretical analysis for the proposed weighting schemes, which could deepen understanding of their behavior.\n4. While feature-level augmentation is mentioned as an efficiency measure, no quantitative comparison (e.g., training time, memory usage) is provided to support this claim.", "was_revised": true, "final_justification": "Most of my concerns are solved except for the theoretical foundation. But considering the methodology, I'd like to raise my score.", "rebuttal": "We appreciate your helpful comments and would like to address your concerns.\n\n> W1: Hyperparameter study\n\nFollowing your great suggestion, we conduct an additional analysis to study the impact of the loss weight hyperparameter ($\\lambda$) on the Kinetics-Sounds, CREMA-D, and Food-101 datasets. As summarized below, the optimal $\\lambda$ value varies depending on the dataset. When $\\lambda$ is too small, the model under-utilizes the information from misaligned samples. Conversely, if $\\lambda$ is too large, the model overly prioritizes misaligned samples and fails to adequately learn shared multimodal representations from aligned data. We will discuss the hyperparameter study in our revision.\n\nMetric | Dataset         | 0.5   | 1     | 2     | 5     | 10    | 20\n--------|------------------|-------|-------|-------|-------|-------|-------\nAcc     | Kinetics-Sounds  | 70.43 | 71.35 | 73.23 | 75.26 | **75.34** | 74.19\nF1      | Kinetics-Sounds  | 62.06 | 62.87 | 65.05 | **68.34** | 68.10 | 66.12\nAcc     | CREMA-D          | 73.52 | **75.00** | 74.60 | 73.25 | 63.17 | 57.46\nF1      | CREMA-D          | 72.26 | **74.05** | 73.31 | 72.32 | 62.58 | 57.12\nAcc     | Food-101         | 93.65 | **93.82** | 93.57 | 65.20 |  n/a  |  n/a\nF1      | Food-101         | 89.32 | **89.56** | 89.21 | 52.11 |  n/a  |  n/a\n\n\n> W2: Additional benchmarks\n\nIn addition to the four widely used benchmarks [1] already included in our paper, we appreciate the suggestion and will further evaluate MIDAS against baselines on the Sarcasm and Twitter2015 datasets. The results are consistent with our original findings, with MIDAS outperforming competing methods and confirming the same performance trend, demonstrating that MIDAS is effective on diverse benchmarks.  We will include these additional results in our revision.\n\nMetric | Dataset     | Joint training | AMCo   | MCR    | MIDAS (ours)\n--------|-------------|----------------|--------|--------|---------------\nAcc    | Sarcasm     | 86.89          | 85.92  | 85.33  | **87.16**\nF1     | Sarcasm     | 86.38          | 85.39  | 84.68  | **86.58**\nAcc    | Twitter2015 | 71.82          | 71.44  | 72.19  | **73.60**\nF1     | Twitter2015 | 64.35          | 58.59  | 65.17  | **65.89**\n\n\n[1] Xu et al., BalanceBenchmark: A Survey for Multimodal Imbalance Learning, ArXiv 2025\n\n> W3: Theoretical analysis\n\nWe acknowledge that our weighting schemes primarily rely on empirical justification, but the weak-modality weighting strategy is theoretically motivated by the modality competition problem analyzed by [1]. This strategy also conceptually aligns with gradient-based [2], confidence-based [2], and loss-based [3] balancing methods, which predominantly depend on empirical validation. Our hard-sample weighting draws upon robust empirical evidence showing that challenging samples consistently improve model training across diverse domains [4, 5]. While the theoretical characterization of this phenomenon remains an active research area, our systematic empirical validation provides practical insights that advance the field's understanding. We appreciate the reviewer’s suggestion and will clarify these points in our revision.\n\n[1] Huang et al., Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably), ICML 2022  \n[2] Peng et al., Balanced Multimodal Learning via On‑the‑Fly Gradient Modulation, CVPR 2021  \n[3] Wang et al., What Makes Training Multi‑Modal Classification Networks Hard?, CVPR 2020  \n[4] Robinson et al., Contrastive Learning with Hard Negative Samples, ICLR 2021  \n[5] Kalantidis et al., Hard Negative Mixing for Contrastive Learning, NeurIPS 2020  \n\n> W4: Efficiency of feature-level augmentation\n\nWe agree that quantitative comparisons would strengthen our claims about efficiency. Feature-level augmentation leverages precomputed features after processing aligned samples. In contrast, input-level augmentation requires processing twice as many raw inputs (aligned and misaligned samples), thereby significantly increasing the computational cost. To quantify this, we measure the training times on the CREMA-D, Kinetics-Sounds, and Food-101 datasets as shown in the table below. We appreciate your suggestion and will explicitly include detailed training-time comparisons in our revision.\n\nDataset         | Feature-level (Ours) | Input-level\n----------------|----------------------|-------------\nCREMA-D         | 52 min                   | 77 min\nKinetics-Sounds | 311 min                  | 445 min\nFood-101        | 278 min                  | 693 min"}, {"review_id": "hIx7BurAtU", "reviewer": "Reviewer_Ny82", "summary": "This paper proposes MIDAS, a novel data augmentation strategy for imbalanced multimodal learning that leverages semantically misaligned samples instead of aligned ones. Specifically, MIDAS incorporates two weighting mechanisms to address existing challenges: (1) weak-modality weighting, which aims to better exploit underrepresented modalities, and (2) hard-sample weighting, which focuses on handling semantically ambiguous samples. Extensive experiments on multimodal classification tasks demonstrate the effectiveness of the proposed method.", "questions": "While the experimental evaluation is generally solid, this paper lacks comparisons with several advanced and competitive baselines, which limits the ability to fully assess the advantages of the proposed method. Moreover, the paper would benefit from visualizations illustrating how MIDAS’s two weighting mechanisms assign weights across modalities and samples, providing deeper insights into the model’s internal behavior. Additionally, it remains unclear whether the authors have explored the impact of alternative strategies—beyond random replacement—for generating misaligned samples on the method’s performance.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n(1)\tThis paper proposes MIDAS, a novel approach that leverages semantically misaligned samples to achieve superior performance in imbalanced multimodal learning.\n\n(2)\tThe paper is well-written with a clear and logically structured presentation. The results are supported by well-designed figures and tables, which enhance the overall clarity of the work.\n\n(3)\tThe experimental evaluation is relatively comprehensive, including comparisons with baselines, detailed ablation studies, and additional insightful discussions.\n\nWeaknesses:\n\n(1)\tWhile the experimental evaluation is generally solid, it lacks comparisons with several advanced and competitive baselines (e.g., CGGM [1], Dyncim [2]), which limits a full assessment of the proposed method's advantages.\n\n(2)\tThe paper would benefit from visualizations illustrating how MIDAS's two weighting mechanisms assign weights across modalities and samples, which could offer deeper insights into the model’s internal behavior.\n\n[1] Guo Z, Jin T, Chen J, et al. Classifier-guided gradient modulation for enhanced multimodal learning[J]. Advances in Neural Information Processing Systems, 2024, 37: 133328-133344.\n\n[2] Qian C, Han K, Wang J, et al. Dyncim: Dynamic curriculum for imbalanced multimodal learning[J]. arXiv preprint arXiv:2503.06456, 2025.", "was_revised": true, "final_justification": "Thanks for the detailed rebuttal. I will keep my positive rating.", "rebuttal": "We appreciate your thoughtful comments and would like to address your concerns.\n\n> W1/Q: Comparison with recent baselines\n\nWe agree that a comparison with advanced baselines would further support the effectiveness of MIDAS. Therefore, we compare MIDAS to the advanced baselines (CGGM [1] and DynCIM [2]) on the CREMA-D dataset with three random seeds, as shown in the table below. As a result, MIDAS outperforms CGGM and DynCIM on average, even though both methods show strong results compared to the other baselines in our paper. For a specific seed, DynCIM shows better performance compared to MIDAS, but exhibits worse performance stability. We believe this is because both baselines rely on complex balancing mechanisms that combine signals across modalities. These mechanisms require careful tuning of gradient magnitudes or loss scaling factors during optimization. In contrast, MIDAS operates through sample-level labeling and weighting in the data space, providing more stable training dynamics without requiring sensitive gradient computations. We will discuss these points in our revision.\n\nMetric | CGGM             | DynCIM             | MIDAS (ours)\n--------|------------------|--------------------|-------------------\nAcc     | 68.99 ± 3.10     | 67.06 ± 8.51       | **74.99 ± 0.31**\nF1      | 67.21 ± 3.15     | 60.95 ± 11.05      | **73.82 ± 0.33**\n\n[1] Guo et al., Classifier-guided Gradient Modulation for Enhanced Multimodal Learning, NeurIPS 2024  \n[2] Qian et al., DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning, ArXiv 2025\n\n> W2/Q: Visualization of two weighting mechanisms\n\nThank you for this valuable suggestion. For weak-modality weighting, Fig. 3 illustrates how MIDAS dynamically adjusts the weak-modality weights during training across four datasets. The weights increase to allocate greater weight to the undervalued modality and converge as training progresses. For hard-sample weighting, we generate new histograms showing the distribution of similarity scores. For example, on the CREMA-D dataset, the histograms reveal that video samples exhibit a highly concentrated distribution (mean=0.875, std=0.093) with most samples being easy positives, while audio samples show a more dispersed distribution (mean=0.748, std=0.126) with greater variance, clearly demonstrating how MIDAS assigns higher weights to video samples with higher similarity scores. We will add the additional visualized analysis in our revision.\n\n> Q: Additional study of the misaligned sample generation method\n\nThank you for this insightful idea. We use random replacement because of two key advantages: its computational efficiency and model generalizability. To illustrate how the model generalizes better, we compare random replacement against a “confusion-based” replacement strategy (pairing each sample with another whose label is among its top-2 predicted classes). This alternative strategy makes misaligned samples inherently challenging or informative for better training. As summarized in the table below, the results on the CREMA-D dataset indicate that random replacement outperforms confusion-based replacement. That is, training a model exclusively on hard samples limits its generalizability, leading to lower accuracy. We will include this comparative analysis and a brief discussion on alternative strategies in our revision.\n\nMetric | Confusion-based replacement | Random (Ours)\n--------|------------------------------|------------------\nAcc     | 67.67                        | **75.00**\nF1      | 66.79                        | **74.05**"}, {"review_id": "RzYon0oCjF", "reviewer": "Reviewer_TS6B", "summary": "The paper presents MIDAS, a data augmentation strategy to artificially generate multimodal misaligned samples, i.e., samples with semantically inconsistent information across different modalities, for example, a picture of a car paired with the caption of a different object. MIDAS uses unimodal classification confidence to combine modalities from different samples. The network is then trained to match the unimodal classification performance with the multimodal classifier, encouraging the model to learn equally from both modalities, despite the misalignment.", "questions": "- Sec. 4.4 presents an analysis on the performance of MIDAS on misaligned data but few details are provided on this evaluation. How are the misaligned data obtained in this case? Given the focus of the paper on misaligned data, I think that the evaluation on misaligned data would require more attention. \n- Are the baselines in the Experiments section also trained on the misaligned samples? If not, how would these baselines compare to MIDAS when trained on the same misaligned data? By looking at the first lines in Table 1 and Table 2, it seems the largest improvement is brought by training on misaligned data alone. It would also be useful to compare the same baselines in Table 1 in this setting.\n- As the model is trained with a cross entropy loss, is it optimal to use cosine similarity for hard-sample weighting? I think it would be more appropriate to use cosine similarity if the classifier was a cosine classifier, and L2 distance when using cross entropy.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper focuses on semantic misalignment across different modalities in multimodal models and frames this scenario as an “opportunity” to encourage balanced learning across modalities. This approach is very intuitive but effective and proposes an interesting take on what might otherwise be seen as a problem.\n- The experiments show consistent improvements compared to previous works, without requiring any modification to the architecture or using external data. Experiments focus on audio/video and image/text dataset, showing the effectiveness of the approach on different combinations of the input modalities.\n\n\nWeaknesses:\n- With MIDAS, the misalignment is created artificially by combining samples with different labels. However, semantic misalignment across modalities is a more broad issue that may arise naturally in data, for example due to annotation errors or intrinsic differences in the semantic information that each modality can encode [1]. In this sense, the proposed approach appears limited as it assumes that samples are normally aligned. Could it be extended to a more realistic scenario in which we don’t know if and how much the individual modalities composing a sample are misaligned? In other words, could MIDAS be applied to already misaligned samples?\n- The labeling strategy and the weak-modality weighting mechanism in MIDAS encourage the multimodal classifier to match the confidence of the individual unimodal classifiers. However, different modalities may have different levels of information and convergence rates, making some modalities inherently more “informative” and therefore their respective unimodal classifiers more confident. This also makes the confidence scores not comparable across modalities. Why is the multimodal model expected to match the performance of their unimodal counterparts? Is this optimal? How would this compare with other approaches that “hardcode” the balance between the two modalities (for example giving the two modalities the same weight)?\n\n\n[1] Von Kügelgen, Julius, et al. \"Self-supervised learning with data augmentations provably isolates content from style.\" NeurIPS 2021", "was_revised": true, "final_justification": "The paper's take on misaligned samples as an opportunity for multimodal learning is both interesting and novel, with potential for future extensions in realistic scenarios. The method is sound and validated on many datasets. The rebuttal and follow-up discussion clarified my doubts, mostly related to the misaligned samples generation pipeline and the evaluation setting. I am now confident in increasing the paper's score.", "rebuttal": "We appreciate your constructive comments and would like to address your concerns.\n\n> W1: MIDAS in a realistic scenario\n\nMIDAS can be reasonably extended to handle cases where the degree of modality misalignment is unknown because we rely on the assumption that each unimodal classifier’s confidence score provides an appropriate proxy for alignment quality. If a modality is associated with incorrect or semantically confusing information, MIDAS will assign it a smaller weight, based on its lower confidence score obtained from the unimodal classifier. However, natural misalignment imposes fundamental limitations on all approaches for imbalanced multimodal learning, and MIDAS cannot completely resolve these unknown misalignments either. For example, a sample might contain an image of a cat and accompanying text “a kitten”, yet the label is mistakenly set to “dog” due to an annotation error. In this case, neither the vision nor the language classifier can confidently predict “dog”, so MIDAS assigns both modalities very low weights. Nevertheless, by combining confidence-based labeling with misaligned sample generation, MIDAS provides a practical mechanism for reducing their impact. We will explicitly clarify and discuss this potential extension in our revision.\n\n> W2: Optimality of unimodal classifiers\n\nWe acknowledge that different modalities indeed have varying information levels and convergence rates, making direct confidence matching potentially suboptimal. However, in imbalanced multimodal learning, it is standard practice to leverage each modality's information signals across modalities simultaneously [1, 2, 3]. Accordingly, we employ a confidence-based approach that reflects each modality's actual information contribution, rather than enforcing an artificial balance. \nIn addition, we clarify that our warm-up phase is specifically designed to mitigate the convergence rate differences by pre-training each unimodal classifier individually prior to joint multimodal training. This helps ensure that confidence scores are more comparable across modalities by allowing each to reach a stable state.\nRegarding hard-coded weighting approaches, we experiment on CREMA-D comparing our confidence-based method against equal (5:5) weighting:\n\nMethod | Confidence-based (Ours) | hard coding (5:5)\n-------|-------------------------|-------------------\nAcc    |        **75.00**            |      70.23\nF1     |        **74.05**            |      69.02\n\nThe results demonstrate that artificially balancing modalities regardless of their actual information content significantly harms performance. We will include these additional experimental comparisons in our revision to further validate this design choice.\n\n[1] Wei et al., Mmpareto: Boosting multimodal learning with innocent unimodal assistance, ICML 2024  \n[2] Qian et al., Dyncim: Dynamic curriculum for imbalanced multimodal learning, ArXiv 2025  \n[3] Peng et al., Balanced Multimodal Learning via On‑the‑Fly Gradient Modulation, CVPR 2021\n\n> Q1: Misaligned accuracy measurement\n\nTo measure accuracy for misaligned samples, we generate misaligned data by randomly pairing modalities from different samples within the validation set, following the same procedure as for the training data, ensuring they contain conflicting semantic information. This synthetic misalignment strategy allows us to evaluate which modality the model relies on when faced with semantic inconsistencies. We consider the prediction correct if both labels from the sources of the misaligned samples are among the top-2 classes of the model's output.\nWe acknowledge that this evaluation deserves a more detailed explanation, given its importance to our main contribution. In our revision, we will provide more comprehensive details about the data generation method, including specific examples of the misaligned pairs (e.g., pairing a \"cat\" image with \"dog\" text, where the prediction is considered correct if the model's top-2 output classes are \"cat\" and \"dog\").\n\n> Q2: Experimental setting details\n\nWe clarify that the baselines are trained exclusively on aligned samples, as they do not consider misaligned samples for training. In contrast, MIDAS generates misaligned samples and leverages both aligned and misaligned samples during training, which is our main contribution. This represents a fair comparison since we evaluate each method under its optimal training conditions - baselines with aligned data, and MIDAS with its ability to benefit from additional misaligned samples. We will clarify this experimental setup more explicitly in our revision to avoid confusion.\n\n> Q3: Similarity metric for hard-sample weighting\n\nCompared to L2 distance, cosine similarity is generally preferable for high-dimensional features due to its scale invariance and better discriminative properties [1, 2, 3]. Our hard-sample weighting is also based on high-dimensional feature similarity, so we use cosine similarity. To further validate this choice, we perform new experiments below where we replace cosine similarity with L2 distance when using our method. As a result, using cosine similarity leads to better accuracy as expected. We will incorporate these experimental results in our revision to substantiate our choice.\n\nMetric | Dataset         | Cosine similarity (Ours) | L2 distance\n--------|------------------|---------------------------|-------------\nAcc     | Kinetics-Sounds  | **75.26**                     | 74.69\nF1      | Kinetics-Sounds  | **68.34**                     | 67.00\nAcc     | CREMA-D          | **75.00**                     | 74.26\nF1      | CREMA-D          | **74.05**                     | 73.36\nAcc     | Food-101         | **93.82**                     | 93.79\nF1      | Food-101         | **89.56**                     | 89.54\n\n[1] Radford et al., Learning Transferable Visual Models From Natural Language Supervision, ICML 2021  \n[2] Robinson et al., Contrastive Learning with Hard Negative Samples, ICLR 2021  \n[3] Kalantidis et al., Hard Negative Mixing for Contrastive Learning, NeurIPS 2020"}, {"review_id": "RNuP7UVfQ6", "reviewer": "Reviewer_rnN8", "summary": "The paper presents a novel approach to addressing modality imbalance in multimodal learning by leveraging misaligned samples (synthetically generated examples that contain conflicting signals from different modalities). They posit that such samples force the model to use information from the non-dominant modality. They also propose two re-weighting mechanisms to further modality balance. One mechanism, weak-modality weighting, adjusts the contribution of the non-dominant modality in the loss computation based on the multi-model’s confidence. The second, hard-sample weighting increased the weight of samples where the substituted modality in the misaligned samples is similar to the original. Such samples are inherently harder for the model to learn, and hence provide a stronger signal for learning.\n\nThe proposed method is tested across several different multimodal datasets and shown to outperform other proposed methods for addressing modality imbalance. Ablations are performed to confirm that each each the three proposed components contributes meaningfully to the method’s efficacy.", "questions": "1. Do specific modality combinations or tasks make the proposed method more or less effective? This would provide greater insight into the generalizability of the approach.\n2. Does the proposed method still offer performance benefits when the modalities are naturally balanced in their informational content? At the moment, it is unclear whether model performance could be degraded in such a scenario by training on misaligned samples.\n3. Did you perform any experiments to specifically verify that the models actually do utilize information in both modalities more evenly after being trained with the proposed method? This seems important in validating the claim that MIDAS specifically addresses mobility imbalance.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and easy to follow;  the mathematical details of the proposed method are clearly outlined.\n\nThe results are well presented, and plots are easy to follow. The proposed method shows moderate performance gains over baselines.\n\nThe presented work is on a highly relevant topic given the rise of multimodal learning across various settings. It addresses an important gap in existing work (i.e., the lack of data-level approaches for handling modality imbalance).\n\nThe use of misaligned samples as a training signal in this context is a novel and interesting approach\n\nWeaknesses:\n\nNo analysis is presented on how well the proposed method performs on different modality combinations.  Fig. 2 is a bit overly complex, difficult to understand. The performance results are only presented for bimodal datasets\n\nNo analysis is presented to demonstrate that improved modality balance is the actual mechanism behind the observed performance gains.", "was_revised": true, "final_justification": "I find this paper original, presented well, with findings backed by good experiments that were well explained/clarified including in the rebuttal. I hence have kept my original recommended score.", "rebuttal": "We appreciate your insightful comments and suggestions. We would like to address your points.\n\n> W1: Analysis for diverse modality combinations\n\nWe clarify that the results in Table 1 contain not only image-text but also other bimodal pairs (audio-video and RGB frame-optical flow), showing that our method consistently improves performance across diverse modality combinations. Moreover, in the Supplemental Material (Section B.1), we present additional experiments on a tri-modal dataset (text, audio, and video), further demonstrating that our approach generalizes effectively to multi-modality settings.\n\n> W2: Link between modality balance and performance gains\n\nWe note that modality imbalance has been widely recognized as a primary bottleneck in multimodal learning. Numerous studies have shown that when multimodal models over-rely on dominant modalities, overall performance suffers [1, 2, 3]. Methods that explicitly rebalance modality contributions consistently yield substantial accuracy gains. MIDAS follows the same principle, and our experimental results align with the well-established connection between modality balance and multimodal performance.\n\n[1] Wang et al., What Makes Training Multi‑Modal Classification Networks Hard?, CVPR 2020  \n[2] Huang et al., Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably), ICML 2022  \n[3] Xu et al., BalanceBenchmark: A Survey for Multimodal Imbalance Learning, ArXiv 2025\n\n> Q1: Modality-specific effectiveness\n\nThe level of imbalance between modalities plays a major role in how much MIDAS improves performance. In our experiments (see Table 1), the audio-video modalities (Kinetics-Sounds and CREMA-D datasets) have significant imbalance, and MIDAS is thus very effective. On the other hand, the UCF-101 dataset uses RGB frame and optical flow modalities, which have lesser imbalance, so MIDAS is less effective, although it is still comparable to baselines. The imbalance varies across modality combinations as each modality has distinct characteristics [1].  We will discuss and cover these aspects in our revision.\n\n[1] Liang et al., Foundations and trends in multimodal machine learning: Principles, challenges, and open questions, ArXiv 2022\n\n> Q2: Naturally balanced modality setting \n\nWhen modalities are naturally balanced in their informational content, MIDAS should maintain performance without degradation for two key reasons. First, when modalities are naturally balanced, the unimodal confidence scores would be similar, leading to balanced weighting without artificial bias toward either modality. Second, the soft labels assigned to the misaligned samples (e.g., (0.5, 0.5)) are effectively true labels that help with regularization instead of being noisy. We will elaborate on these points in our revision.\n\n> Q3: Experiment for verifying improved modality balance\n\nYes, we provide direct experimental validation through confidence tracking on the Kinetics-Sounds and CREMA-D datasets (Figures 1(b) and 4(b)). These experiments show that MIDAS produces more balanced confidence distributions across modalities compared to joint training, especially for misaligned samples. Under the assumption that a modality’s confidence reflects its actual contribution to the final decision, this balance in confidence implies that MIDAS is effectively leveraging information from each modality."}]

Author rebuttal:

Meta-review: The paper proposes to generate misaligned samples as a data augmentation strategy to handle modality imbalance problem in multimodal learning. Unimodal confidence scores are leveraged to achieve this generation and further a weighting mechanism is introduced to support weaker modalities during learning.

Some notable strengths of the submission include:
- a novel data augmentation strategy to generate misaligned samples 
- thorough experimental results and the paper is well-written

Among weaknesses, the concerning were:
- missing impact on hyperparameters 
- evaluation scope is limited, as some important benchmarks are missing (e.g., Sarcasm, Twitter2015, NVGesture)
- some visualizations are required to better understand the two weighting mechanisms
- grounding of the method that proposed modality balanced actually provides the claimed gains
- assumption of data being normally aligned and what if the data is by default misaligned

After the post-rebuttal discussion phase, majority of the reviewers remained positive about the novelty of the submission and the experimental validation. Some new results such as hyperparameter study, more dataset results, comparison with more recent baselines were provided during the rebuttal and post-rebuttal discussion period. Furthermore, the clarifications on normal aligned data assumption and stability of experimental results were acknowledged by the reviewers. Overall, nothing major or concerning remained after the closing of author-reviewer discussion. Therefore, the AC decides to recommend acceptance.
Decision: Accept (poster)