Improving Multimodal Learning Balance and Sufficiency through Data Remixing
Authors: Xiaoyu Ma
Hao Chen
Yongjian Deng
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Learning
Machine Learning
Representation Learning
Primary area: general_machine_learning->representation_learning

Abstract: Different modalities hold considerable gaps in optimization trajectories, including speeds and paths, which lead to *modality laziness* and *modality clash* when jointly training multimodal models, resulting in insufficient and imbalanced multimodal learning.
Existing methods focus on enforcing the weak modality by adding modality-specific optimization objectives, aligning their optimization speeds, or decomposing multimodal learning to enhance unimodal learning. These methods fail to achieve both unimodal sufficiency and multimodal balance.
In this paper, we, for the first time, address both concerns by proposing multimodal Data Remixing, including decoupling multimodal data and filtering hard samples for each modality to mitigate modality imbalance; and then batch-level reassembling to align the gradient directions and avoid cross-modal interference, thus enhancing unimodal learning sufficiency. 
Experimental results demonstrate that our method can be seamlessly integrated with existing approaches, improving accuracy by approximately **6.50\%$\uparrow$** on CREMAD and **3.41\%$\uparrow$** on Kinetic-Sounds, without training set expansion or additional computational overhead during inference. The source code is available at Data Remixing.

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
Xiaoyu Ma
Hao Chen 12 Yongjian Deng²
Abstract
Different modalities hold considerable gaps in optimization trajectories, including speeds and paths, which lead to modality laziness and modal- ity clash when jointly training multimodal mod- els, resulting in insufficient and imbalanced mul- timodal learning. Existing methods focus on en- foreing the weak modality by adding modality- specific optimization objectives, aligning their op- timization speeds, decomposing multimodal learning to enhance unimodal learning. These methods fail to achieve both unimodal sufficiency and multimodal balance. In this paper, we, for the first time, address both concerns by proposing multimodal Data Remixing, including decoupling multimodal data and filtering hard samples for each modality to mitigate modality imbalance; and then batch-level reassembling to align the gradient directions and avoid cross-modal inter- ference, thus enhancing unimodal learning suf- ficiency. Experimental results demonstrate that our method can be seamlessly integrated with existing approaches, improving accuracy by ap- proximately 6.50%t t CREMAD and 3.41 % Kinetic-Sounds, without training set expan- sion or additional computational overhead during inference. The source code is available at Data Remixing.
Distribution with other reaially.
Decouple Multimodal Data
Reasermble Unimodal Data
being interoredd vat.
Diretkin NOT uxeeffedd
Batch
Mack medakly ofoech sample pakr.
NEW Batch
AlMaMa 1 conaccall I Hoidlly 2 mesk canallig 0:0 of
Seened (8
Figure I. We decouple the multimodal data to assign samples to each modality's 's training and then reassemble the inputs control the consistency of modalities within the batch. By regulating the number of samples, we mitigate modality laziness, and by adjusting the hatch composition, we alleviate modality clash.
perception and decision decision-making capabilities of models by integrating data from diverse modalities, including vision, sound, and text (Zhu et al., 2024). However, existing multi- modal learning methods often face challenges in fully inte- grating rich multimodal knowledge across different modali- ties. Due to the inherent differences in data representation and distribution across modalities, their optimization trajec- tories differ significantly, resulting in imbalanced learning when multiple modalities are jointly trained under a unified objective. Specifically, multimodal models often prioritize learning the most discriminative features from the strong modality, suppressing the training of the weak modality and causing it to become lazy, known as modality laziness (Wang et al., 2020: Huang al., 2022; Du al., 2021).
1. Introduction
Multimodal learning (Ngiam et al., 2011) is rapidly evolv- ing field in artificial intelligence, aimed at enhancing the
"School of Computer Seience and Engineering, Southeast University, Nanjing, China Key Laboratory of New Genera- tion Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University). Ministry of Education, China Colleee of Computer Science, Beijing University of Technology, Beijing, China. Correspondence to: Hao Chen <haochen303 @ scu.edu.en>.
Several methods have been proposed to address these is- sues. Some focus on alleviating modality laziness by enforc- ing the weak modality through modality-specific optimiza- tion objectives, such as adjusting aseciic supervision (Wang et al., 2020; Xu et al., 2023; Du et al., 2023), in- troducing prototype learning (Fan al., 2023), or using
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
knowledge distillation (Du et al., 2021). Other methods aim to align optimization speeds by modifying learning rates (Sun et al., 2021) and gradients (Peng et al., 2022; Li et al., 2023; Sun et al., 2023; Fu et al., 2023; Kontras et al.. 2024; Wei & Hu, 2024) based on unimodal performance. In con- trast, some approaches attempt to regulate modality inputs to more explicitly enhance the learning of weak modalities, such as by augmenting weak modality samples (Wu al., 2022; Wei et al., 2024) and masking strong ones (Zhou et al., 2023), or by decoupling multimodal learning into unimodal tasks to avoid modality laziness (Zhang et al., 2024b). How- ever, most of the existing methods align the optimization speeds of different modalities to alleviate modality laziness, without addressing the fact that even when modality balance achieved, differing optimization paths can still cause inter- ference during modality optimization. In multimodal learn- ing, the gradient update directions of different modalities are inconsistent (Fan et al., 2023), leading to cross-modal inter- ference during batch gradient descent (Figure 3(c)), which can be called modality clash. This inconsistency causes the modalities to deviate from their expected optimization paths, as shown in Figure 1, resulting in insufficient learning across all modalities and limiting the effectiveness of multi- modal learning. Moreover, some methods are constrained by specific model architectures (He et al., 2022; Lin et al., 2023; Zhang et al., 2024b) or hinder training efficiency (Wei et al., 2024; Zhang et al., 2024b), which further limits their applicability.
We test our method on different datasets and achieve excel- lent results. When combined with conventional fusion meth- ods, the model's performance shows notable improvement across multiple datasets. Importantly, our strategy does not expand the dataset or introduce additional overhead during inference. We summarize our key contributions as follows:
We introduce the Data Remixing method, a training strategy that combines decoupling multimodal data and reassembling unimodal data to address modality laziness and clash. Our method does not require dataset expansion and is not constrained by specific model architectures.
We analyze the optimization process and propose that modality laziness and clash originate at the batch level. To the best of our knowledge, we are the first to analyze the challenges of multimodal learning at the batch level and propose solution.
We perform experiments and demonstrate that (1) Data Remixing achieves excellent results in multimodal learning tasks; (2) Data Remixing can be easily and effectively integrated with other methods, leading to significant improvements of approximately 6.50 % f on CREMAD and 3.41 % on Kinetic-Sounds.
2. Related Work
Recognizing these limitations, we propose the Data Remix- ing method, illustrated in Figure 2. Through dynamic sam- ple allocation and batch-level alignment mechanisms, we si- multaneously address modality laziness and modality clash without hindering training efficiency or being constrained by specific model architectures.
In this section, we introduce several strategies for multi- modal balance learning. We categorize these methods based on the form of data input. The methods described in Section 2.1 1 retain multimodal inputs throughout the learning pro- cess, whereas those in Section 2.2 operate with unimodal inputs (though not necessarily for the entire duration of training).
Concretely, our Data Remixing method consists of two key steps: sample-level decoupling of multimodal data and batch-level reassembling of unimodal data. First, based unimodal separability, we evaluate the representational ca- pability of each modality at the sample level, retaining only the input from the weak modality while masking the others to zero. By decoupling the multimodal data, multimodal models can leverage specific samples to train each modality effectively. However, even after decoupling, unimodal gra- dient update directions still lead interference. We further analyze the optimization process and propose that modal- ity clash originates at the batch level. To address this, we reassemble unimodal data based on the decoupling assign- ments, ensuring that each batch contains data from only one modality, free from interference from other modalities, as shown in column 3 of Figure 1. Such sample-leve decou- pling and batch-level reassembling allow each modality to be learned sufficiently and balanced, all without expanding the dataset.
2.1. Balance with Multimodal Joint Input
Wang et al. (2020); Du et al. (2021) discover that the opti- mization speed varies across different modalities. Therefore, when optimizing a multimodal model with unified objec- tive, the weak modality fails to learn adequately, leading to modality laziness. Most methods maintain multimodal data inputs during training, achieving modality balance by adjusting optimization objectives and speeds.
Some try alleviate modality laziness by enforcing the weak modality through modality-speciffi optimization ob- jectives Wang et al. (2020); Du et al. (2021): Xu et al. (2023); Du et al. (2023); Fan et al. (2023). Wang et al. (2020) pro- pose
-linnding which calculates the optimal mix- ing mode of modality losses by determining the overfitting situation for each modality. Du et al. (2021) attempt to distill knowledge from well-trained unimodal models to enhance
2

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
(a) Warm-up training framework
(c) Specifie Training (visual as example)
Law
Ando male
badwoard
Miskkd Audio
Audio
Inted Dotaett
Chew hampp
Loe
Viad
Vied
Cue Taxen
Visual
Alternating steps
(b) Decoupling and Reassembling of Modality Inputs
KL Audio
Audio Dataset
IF KL Audio < KL Visunl
Audio
Eaital Bataod
Unieoda Prodecion
Litton Durthasat
Audio
ELSE:
add to Visual dataset
Visual Dutuset
Viaii
KL. Visual
Visual
Unireddal Prediet
Figure 2. The pipeline of Data Remixing method. In step (a), the complete dataset is used for training to ensure the model develops basic representational capability. In step (b), the multimodal data decoupled based on unimodal separability (calculated using KL-divergence), and the original dataset is reassembled into non-overlapping subsets. In step (c), the subsets obtained in step (b) are used to train on the specific modality by masking other modalities to zero.
the unimodal encoders. Fan et al. (2023) introduce the pro- totype cross-entropy loss for each modality to accelerate the slow-learning modality. Others try to align optimiza- tion speed by changing learning rates (Sun et al., 2021) or modifying gradients (Peng al., 2022: Li al., 2023; Sun al., 2023; Fu et al., 2023; Kontras al., 2024) according to unimodal performance. Sun et al. (2021) dynamically adjust the learning rates of different modalities based the unimodal predictive loss. Peng et al. (2022) adaptively modulate the gradients of each modality by monitoring the discrepancy in their contribution to the learning objective. These methods promote modality balance and enhance the expressive power of multimodal models. However, they fail to recognize that even when modality laziness is addressed, modality clash persists, meaning that multimodal capability remains limited.
et al. (2022) measure the relative speed of each modality and train unimodal branches to fully utilize them. Zhang et al. (2024b) decompose multimodal learning into alternating unimodal learning with gradient modification to preserve cross-modal interactions. Wei al. (2024) design sample- level modality contribution evaluation based on Shapley values and perform resampling of the weak modality to hance specific modality training. These methods, which rely on unimodal inputs, seem to avoid modality clash. How- ever, upon closer inspection, we find that modality clash introduced at the batch level. Like the methods Scction 2.1, they have not analyzed or addressed the root cause of modality clash.
Overall, current work focuses addressing modality lazi- ness but fails to recognize that modality clash still exists in balanced multimodal learning. Meanwhile, some meth- ods are constrained by specific model architectures (Wu et al., 2022; Zhang et al., 2024b) or hinder training effi- ciency (Zhang et al.. 2024b; Wei al., 2024), limiting their applicability. In this paper, we aim simultaneously mit- igate modality laziness and modality clash while ensuring compatibility with various fusion methods and model archi- tectures.
2.2. Balance with Unimodal Single Input
Some methods attempt to use alternating or selective uni- modal inputs to promote multimodal learning, thereby avoid- ing interference from other modalities and enhancing the expressive power multimodal models (Wu et al., 2022: Zhang al., 2024b; Wei al., 2024: Zhou al., 2023). Wu
3

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
3. Method
However, modality-level control over the optimization pro- cess objectives overlooks the imbalance at the sample level between modalities (Wei et al., 2024), which limits the effectivvness of the balancing. Based on this premise, we consider promoting balance by assigning specific samples (in terms of difficulty and quantity) to different modalities, achieving a more precise and convenient modality balance.
3.1. Model formulation
In this paper, we focus on the multimodal discrimination task, following related works (Peng et al., 2022; Fan al., 2023; Wei al., 2024; Zhang et al., 2024b). For conve- nience, we consider two input modalities: ma and my- The training dataset is D- = { yo 1.... Each consists of multimodal inputs, i.e., 2, = (z), V/ E {1,2..... M). where M is the number of classes. The mul- timodal model is trained using batch gradient descent, with data x - (x x²) sampled from a batch B.
We first define a sample-level unimodal capability evalu- ation method to achieve either-or data allocation, instead of using Shapley values as proposed by Wei et al. (2024)', to avoid dataset expansion. For each sample 24, we obtain unimodal prediction probabilities pf. k E {a, v}. The rep- resentational ability of each modality is then assessed by calculating the KI. divergence between pe¹ and a uniform distribution as
We use a multimodal model consisting of two unimodal branches for prediction. Each branch has unimodal coder, denoted as 6º and Φ², used to extract features from the corresponding modality of x. The encoder outputs are represented as n" = (0" x0" and z" = (0'" (0" x"), where H and 6" are the parameters of the encoders. The results of the two unimodal encoders are fused in some way (Owens& & Efros, 2018; Gunes& & Piccardi, 2005) to obtain the multimodal output. We use cross-entrops (CE) loss the loss function. To achieve better representation ability in the warm-up stage, we add a separate classification head for each modality, as Wang et al. (2020) do, and modify the loss function as follows:
M Σ
Pi.j Uj
DKL(PE
pi, log j=1
(2)
where ph, represents the predicted probability for class j. and U = M corresponds to the uniform distribution. KL divergence measures the difference between two probability distributions. In the current application scenario, smaller KL divergence indicates that the output is closer to a uniform distribution, meaning that the separability of the correspond- ing unimodal output is worse. This can be interpreted as insufficient training for the current modality on this sam- ple. Therefore, we decouple the multimodal inputs for each sample and only retain the modality that performs the worst, masking input from other modalities to achieve accurate unimodal training (Zhang et al., 2024a; Wei et al., 2024). That is, the new training set is as follows:
K
L = CEE
w
ma CE
(1)
k=1
where Leg represents the CE loss for multimodal predic- tion, while Ceg represents the CE loss for the individual prediction of the k-th modality. We provide experiments to demonstrate that the improvements brought by our method are not dependent on the design of this loss function.
D' = x
x EEE D.D.m. m. = arg min PKc(me) m.EK
(3)
3.2. Data Remixing Method
Overview of Method. The complete pipeline of our method is presented in Figure 2. In step (a), we first use the com- plete dataset and multimodal inputs for warm-up training to ensure the model has the basic representational capability. Then, the model is optimized through alternating steps (b) and (c). In step (b), we decouple the multimodal inputs based on the KI. divergence of unimodal prediction proba- bilities and reassemble the data at the batch level according to the remaining modality. In step (c), we perform specific training for each modality using the reassembled dataset.
By decoupling the multimodal inputs, the strong modality is exposed to fewer samples, while the weak modality is exposed to more, shown in Figure 3(a). This forces the model to focus more on learning the weak modality during the optimization process and prevents the strong modality from suppressing the weak modality's 's learning, thereby promoting modality balance, as shown in Figure 3(b), and ensuring more comprchensive multimodal learning. More importantly, our sample-level unimodal evaluation method provides the basis for reassembling unimodal inputs avoid modality clash.
Decouple Multimodal Data. In multimodal learning, the model tends to fit the modality that optimizes faster, often converging before the other modality has been sufficiently learned (Du et al., 2021). Therefore, appropriately reduc- ing the training speed of the strong modality may be an effective strategy for balancing learning (Peng et al., 2022).
Reassemble Unimodal Data. After decoupling the mul- timodal inputs, we address modality laziness but find that
For x; if the unimodal predictions are correct with probabili- ties of 0.9 and 0.6, the Shapley values for both modalities would be the same, hut the representational abilities are different.
4

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
modality clash remains, shown in Figure 3(c). For more sufficient multimodal learning, we further analyze the phe- nomenon and identify that the interference is introduced at the batch level.
To control the composition of each batch, we reassemble the unimodal inputs based the decouple assignments and ensure that each batch B; contains data from only one subset, as shown in Equation 9.
Without loss of generality, we assume that the multimodal fusion method selected for the model is Concatenation, as done by others (Fan et al., 2023; Peng et al., 2022; Wei et al., 2024). Let W [M, and b E IR represent the parameters of the linear classifier that produces the logits output. We can then express the output of the multimodal model (without softmax) as
D
D & E (1,2...... K]
(9)
When sampling from specific data subset, Equation 6 can be further simplified. For example, when sampling from the data subset corresponding to the video modality, Equation simplifies
B
W" *
((x) W (0" xx") (0 x") b.
(4)
LCE
log
(10)
Lj=1 == ef(ondy
i=
To analyze the optimization process of each modality, we represent W a block matrix composed of two parts:
where the phenomenon of cross-modal optimization inter- ference has been effectively addressed.
[W", ". We can then rewrite Equation 4 as
((x) = w "00 x²)] + W (0" (0" x bb
In general, the pscudo-code for our method provided in Algorithm We bridge modality decoupling and reassem- bling through unimodal evaluation and implement the Data Remixing method to simultancously tackle modality imbal- ance and insufficiency.
(5)
= W =" + W' z" b.
Assuming that no additional classification heads are added for each modality branch, the loss for each batch is
Algorithm Method of Data Remixing
1
e((i))s
LCe
log
input data D = (xx x xf, ..., ...,.... of y4 )= number modalities K, model parameters 0, training epoch E, warm-up epoch E, for 0,... do
Input:
[B|
(6)
[
1
log
B
En (x(s)s
if < E., then
Update model parameters 0 with dataset D; else
Observing Equation 6, we see that when using batch gradi- ent descent, even if we decouple the multimodal inputs and mask the dominant modality for each sample, multimodal inputs still coexist within batch. The inconsistency in optimization directions causes interference between their gradient update directions, ultimately leading to insufficient model training. Therefore, we propose that cross cross-modal optimization interference originates at the batch level and can be addressed by controlling the composition of each batch, referring to this as reassembling unimodal inputs.
for 1, K do Initialize Dm D = 0; end for
for each sample r in D do
Calculate Uni-modal Ability (sp "p.,..."""""" with Equation 2;
Identify & corresponding to the minimum sp:: Mask x², .j * k to zero;
Add x into Dm
We assume that the dataset D with K modalities can be transformed into D' through decoupling multimodal inputs. Based on the retained modality, the dataset be divided into K subsets, denoted as
end for for h: 1, K do
Update model parameters 0 with dataset D end for
end if
m = Dz
x
0, Vj
(7)
end for
The divided subsets satisfy Equation 8, ensuring that our training set does not expand.
4. Experiments
K
4.1. Dataset and Experimental settings
U
Dm = =D',
(8)
k=1
CREMA-D (Cao et al., 2014) an audiovisual dataset for emotion recognition, consisting of 7,442 video clips from
MD" = Vi
5

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
91 actors. The dataset includes six of the most common emotions: anger, disgust, fear, happy, neutral and sad. A total of 2,443 participants rated each clip for emotion and emotional intensity using three modalities: audiovisual, video only, and audio only. The entire dataset is randomly divided into a training and validation set of 6,698 samples and a test set of 744 samples, with a ratio of approximately 9:1.
nation as the representative fusion method for our approach. It is evident that our method significantly improves model performance across different datasets, with an increase of 8.20% on CREMAD and 5,40% on Kinetic-Sounds, out- performing other conventional fusion strategies. To further demonstrate the generalizability of our method, we com- bine Data Remixing with Summation and Decision Fusion, achieving substantial improvements across both datasets.
Kinetic-Sounds (Arandjelovic & Zisserman, 2017) is dataset derived from the Kinetics dataset (Kay et al., 2017), which includes 400 action classes based on YouTube videos. Kinetic-Sounds consists of 31 action categories, selected for their potential to be represented both visually and aurally, such as playing various instruments. Each video is manually annotated for human actions using Mechanical Turk and is cropped to 10 seconds, focusing on the action itself. The dataset comprises 19k 10-second video clips, split into 15k for training, 1.9k for validation, and 1.9k for testing.
4.3. Comparison with imbalanced multimodal learning methods
Our method is primarily designed to address the issues of modality imbalance and insufficiency in multimodal learn- ing. To evaluate the improvements, we compare our ap- proach with several representative methods that target mul- timodal balance and sufficiency: G-Blend (Wang et al., 2020), OGM-GE (Peng et al., 2022), Greedy (Wu et al., 2022), PMR (Fan et al., 2023), MLA (Zhang et al., 2024b) and Resample (Wei et al., 2024). For a fair comparison, we adopt Concatenation the fusion strategy for the baseline, following the approach used in the above methods. The re- sults, including accuracy (Acc) and dataset expansion factor (Factor), are presented for all methods.
Experimental settings. Unless otherwise specified, all feature extraction networks used in the experiments are ResNet-18 18 (He et al., 2016), trained from scratch. During training. we use the Adam (Kingma, 2014) optimizer with 3 (0.9, 0.999) and set the learning rate to 5 x 10-5, We obtain unimodal prediction results from the unimodal clas- sification heads and also present the results of our method by masking specific modalities to zero, as in Hinton (2012) and Wei et al. (2024). All reported results are averages from three random seeds, with all models trained on two NVIDIA RTX 3090 GPUs using a batch size of 64.
Table 2. Comparison with other imbalanced multimodal lcarning methods. All modulation strategies are applied to the baseline, using Concatenation as the fusion method. We also include results of applying Data Remixing to Resample and MI.A for further comparison.
Table Combination and comparison with conventional fusion methods. "+ Remix" indicates that our Data Remixing method is applied. Bold indicates that our method brings improvement.
CREMAD
Kinetic-Sounds
Method
Acc(%)
Factor
Acc(%) Factor
Concatenation
64.52
1
50.23
1
Method
CREMAD
Kinetic-Sounds
OGM-GE
68.15
1
52.66
1
G-Blend
69.89
1
53.55
1
Concatenation
64.52%
50.23%
+ Greedy
68.28
1
$1.20
Summation
63 63.44%
51.66%
+ PMR
68.95
1
51.93
1
Decision fusion
67. .7.%
52.47%
+ MLA
68.01
2
54.66
2
FiLM
62.77%
49.61%
Resample
67.61
1.85
55.17
1.97
Bi-Gated
63.04%
49.92%
+ Remix (Ours)
72.72
1
55.63
1
Concat Remix
72.72%
55.63%
MLA Remix
74.19
1
57.75
1
Sum Remix
71.51 %
54.09%
+ Resample Remix
73.25
1.92
58.40
2.01
Decision + Remix
70.70%
54.86%
Based on the results in Table 2, we observe that, due to the differing optimization trajectories of modalities, these im- balanced multimodal learning methods show performance improvements over traditional fusion methods. However, our method addresses both modality laziness and modality clash simultaneously, leading to even greater improvements.
4.2. Comparison with conventional fusion methods
We first compare our method with several representative multimodal fusion methods commonly used in deep learn- ing frameworks: Concatenation (Concat) (Owens & Efros, 2018), Summation (Sum), Decision Fusion (Decision) (Gunes& & Piccardi, 2005), FiLM (Perez et al., 2018), and Gi-Ga (Kiela et al., 2018). The results are shown in Table We apply Data Remixing in combination with Concate-
Among these, the MLA and Resample strategies also per- form relatively well, but hinder training efficiency. Un- der the same training conditions (especially GPUs and
6

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
batch size), MLA employs multi-stage training strategy and requires gradient modification based on previous fea- tures, making its training approximately 3x slower than our method?. The Resample method enhances the model's 's train- ing on weak modality samples through resampling, which expands the training set by nearly twice, but it hinders train- ing efficiency. To more intuitively reflect the differences in efficiency, we measure the training time of four methods under the same conditions, as shown in Table 3. We observe that balancing methods using unimodal single input tend to increase the training time, whereas Remix does not expand the training set, making it more efficient. Unlike these meth- ods, our approach neither expands the dataset nor reduces the model's s training efficiency.
Table 4. Results on CREMAD and Kinetic-Sounds with two types of complex modal architectures. "+ Remix" indicates the application of our Data Remixing method.
Method
CREMAD
Kinetic-Sounds
Concatenation
64.52%
50.23%
Concat Remix
72.72%
55.63%
MMTM
66.40%
52.27%
MMTM + Remix
68.82%
54.78%
CentralNet
65.46%
54.09%
CentralNet + Remix
67.61 %
55.94%
As shown in Table 4, when multimodal fusion occurs during the encoding process, applying the Data Remixing method leads to significant improvements, further demonstrating the applicability of our method in complex cross-modal architectures.
Table 3. Training time convergence (seconds) on CREMAD and Kinetic-Sounde datasets.
Method
Baseline
Remix
Resample
MLA
CREMAD
1536
2537
4525
6128
4.5. Analysis of Methods
Kinetic-Sounds
3849
4946
10362
12868
In this section, we provide a detailed analysis of the per- formance improvements introduced by our method. This includes comprehensive ablation experiments and evalua- tion of the effectiveness of each design choice. Additionally, we explore different unimodal prediction methods to further demonstrate the broad applicability of the Data Remixing approach.
Considering the similarities between our approach and the Resample and MLA methods- -uch as MLA's decoupling of unimodal training, which indirectly achieves batch con- trol, and Resample's selective data training-we we further combine Data Remixing with these methods to demon- strate its advancements and generalizability. We integrate sample-level evaluation process into MLA, promoting model balance without increasing the number of input sam- ples (which is why the Factor in Table 2 is updated to I), and reassemble the unimodal inputs after applying Resample. With our method applied, both approaches show significant improvements in their final results.
1.5.1. ABLATION STUDY
We conduct ablation studies to demonstrate the effectiveness of our two main designs, with the specific results shown in Table 5. Our experiments consist of two parts: (1) De- couple, which involves decoupling the multimodal data and masking specific modality without reassembling; (2) Re- assemble, which involves reassembling the modality inputs based on sample-level evaluation without specific training (i.e., without modality masking).
4.4. Combination with complex cross-modal architectures
The above methods and experiments are conducted using simple fusion methods, where multimodal fusion occurs after the unimodal encoders classifiers. To validate the applicability of the Data Remixing method in more complex multimodal scenarios, we combine it with two intermediate fusion methods: MMTM (Joze et al., 2020) and CentralNet ( (Viclzeuf et al., 2018). MMTM recalibrates the channel fea- tures of different CNN streams through squeezing and mul- timodal excitation steps, while CentralNet uses unimodal hidden representationn alongside central joint representa- tion at each layer, performing fusion through weighted summation learned during training.
Observing the experimental results, demonstrate the ef- fectiveness of our method. In Experiment (1), we decouple the multimodal data and select specific samples for train- ing. This approach facilitates modality balance, yielding some performance improvements. In Experiment (2), we reassemble the inputs based on sample-level evaluation, controlling the data composition within each batch. This strategy alleviates modality clash and leads corresponding improvements. Finally, when both strategies are combined, the model's 's performance is further enhanced.
4.5.2. STUDY OF DECOUPLE MULTIMODAL DATA
? Athhouhh MLA does not increase the number of samples, decoupling the inputs without selecting for unimodal training is similar to expanding the dataset, which slows down training.
In the process of conducting sample- level evaluation and decoupling multimodal data, we choose to mask the better-
7

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
Lidi sital mach Ios boouph
Fporh
(a)
(b)
(C)
Figure Proof of effectiveness of Decouple and Reassemble Methods. The results are obtained on CREMAD. (a) Statistics of the number of samples used for training specific modalities. (b) The change of the imbalance ratio (c) Comparison of gradient direction discrepancies.
CREMAD at different training stages. We calculate the angle between the actual gradient update direction and the ideal guidance direction for the audio modality. The results show that after reassembling the unimodal inputs, the gra- dient update direction aligns more closely with the ideal direction, providing further evidence of the effectiveness of our approach.
Table 5. Results of ablation studies on CREMAD and Kinetic- Sounds. The second and third rows correspond to Experiment (1) and Experiment (2), respectively.
Decouple
Reassemble
CREMAD
Kinetic-Sounds
X
64.52%
50.23%
x
69.89%
52.31%
68.68%
51.70%
72.72%
55.63%
Table 6. Results of accuracy(55) CREMAD and Kinetic- -ounds using diffcrent methods for unimodal predictions.
performing modality and train with specific samples to alle- viate modality imbalance. Figure 3(a) shows the distribution of samples assigned to different modalities during training. is clear that the strong modality consistently receives fewer samples compared to weak modality. Figure 3(b) illustrates the change in imbalance ratio p (Peng al., 2022) before and after decoupling the modality inputs. The results indicate that our method alleviates the modality imbalance, supporting our hypothesis that balancing modality repre- sentation via sample quantity and difficulty can improve model performance. Note that our method is applied after a 10-epoch warm-up stage.
CREMAD
Kinetic-Soundi
Method
Dropout
Head
Dropout
Head
Concatenation
61.56
64.52
49.58
50.23
Summation
59.68
63.44
48.00
51.66
Decision fusion
62.23
67.47
50.08
52.47
Concat + Remix
70.03
72.72
53.89
55.63
Sum + Remix
68.68
71.51
53.12
54.09
Decision + Remix
68.55
70.70
53.55
54.86
4.5.4. STUDY OF UNIMODAL PREDICTION METHODS
4.5.3. STUDY OF REASSEMBLE UNIMODAL DATA
Since method requires accurate unimodal predictions, we add a classification head to each modality branch and synchronize the updates of the classification head parame- ters by modifying the loss function. This approach has been shown to improve model performance (Wang et al., 2020). further validate our method, we compare the results with the dropout method (Hinton, 2012; Wei et al., 2024) for uni- modal prediction evaluation across three traditional fusion strategies. The results are presented in Table 6. From the table, we observe that both methods of obtaining unimodal predictions show effective improvements when applying our strategy. Notably, regardless the chosen method, our approach does not introduce any additional overhead during inference.
To mitigate cross-modal optimization interference, assemble unimodal inputs control the data composition within each batch. Our theoretical analysis of batch control supports the effectivvenss of this strategy, and the improve- ments unimodal performance are evident. Specifically, audio accuracy on CREMAD increased by 1.75%, while video accuracy improved by 2.96%, since the strong modal- ity is less affected compared to the weak modality.
To further validate the effectiveness of our method in allevi- ating gradient interference, we evaluate the gradient update discrepancies with and without data reassembling, as shown in Figure 3(c). Specifically, we assess the gradient update discrepancies the audio modality (the strong modality)
8

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
5. Conclusion
Du, C., Teng, J., Li, T., Liu, Y., Yuan, T., Wang, Y., Yuan, Y., and Zhao, H. On uni-modal feature learning in supervised multi-modal learning. In International Conference on Machine Learning, pp. 8632-8651 PMLR, 2023.
Multimodal models often face challenges due to the dif- ferences in optimization trajectories between modalities, leading to issues of insufficient and imbalanced learning dur- ing joint training. We propose the Data Remixing method, which decouples multimodal data based on unimodal separa- bility and reassembles unimodal data to ensure consistency between modalities at the batch level. Our method effec- tively mitigates both modality laziness and modality clash, achieving significant performance improvements across var- ious datasets, fusion methods, and model structures. Addi- tionally, it does not require dataset expansion introduce extra computational overhead during inference. Moreover, to the best of our knowledge, we are the first to propose that the challenges of multimodal learning originate at the batch level and to offer a solution for them. However, a limitation of our approach arises when one modality serves primarily as an auxiliary modality with limited information (Wei et al., 2025). In such cases, the unimodal evaluation and allocation strategy may require further refinement.
Fan, Y., Xu, W., Wang, H., Wang, J., and Guo, S. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20029-20038, 2023.
Fu, J., Gao, J., Bao, B.-K., and Xu, C. Multi- modal imbalance-awarr gradient modulation for weakly- supervised audio-visual video parsing. IEEE Transac- tions on Circuits and Systems for Video Technology, 2023.
Ghorbani, A. and Zou, J. Y. Neuron shapley: Discovering the responsible neurons. Advances in neural information processing systems, 33:5922-5933 5932, 2020.
Gunes, H. and Piccardi, M. Affect recognition from face and body: early fusion vs. late fusion. In 2005 IEEE in- ternational conference on systems, man and cybernetics, volume 4, pp. 3437-3443. IEEE, 2005.
Acknowledgments
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778.70 2016.
The work is jointly supported by the National Natu- ral Science Foundation of China (NSFC) under Grant 62261160576 and Grant 62203024, the Beijing Natural Science Foundation (4252026), the Research and Develop- ment Program of Beijing Municipal Education Commission [KM20231000502] and the Fundamental Research Funds for the Central Universities of China. This research work is also supported by the Big Data Computing Center of Southeast University.
He, Y., Sun, L., Lian, Z., Liu, B., Tao, J., Wang, M., and Cheng, Y. Multimodal temporal attention in sentiment analysis. In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pp. 61-66, 2022.
Hinton, G. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: met::02070000 2012.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Huang, Y., Lin, J., Zhou, C., Yang, H., and Huang, L. Modality competition: What makes joint training of multi- modal network fail in deep leamingg((roovbblll national conference on machine learning, pp. 9226-9259. PMLR, 2022.
In Inter-
References
Joze, H. R. V., Shaban, A., Juzzolino, M. I.,, and Koishida, K. Mmtm: Multimodal transfer module for enn fusion, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp.
Arandjclovic, R. and Zisserman, A. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp. 609-617, 2017.
2020.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijaynnarasimhan, S., Viola, F,, Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint 1705.v:7005.6660 2017.
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 344)3377-390, 2014.
Kiela, D.,, Grave, E., Joulin, A., and Mikolov, T. Efficient large-scale multi-modal classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Du, C., Li, T., Liu, Y., Wen, Z., Hua, T., Wang, Y., and Zhao, H. Improving molti-modal learning with uni-modal teachers. arXiv preprint arXiv:2 206. / 1059, 2021.

---

Improving Multimodal Learning Balance and Sufficiency through Data Remixing
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1./6600, 2014.
Wang, W. Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22695-12705, 2020.
Kontras, K., Chatzichristos, C., Blaschko, M., and De Vos, M. Improving multimodal learning with multi-loss gradi- ent modulation. arXiv preprint arXiv:2405.0 07930, 2024.
Wei, Y. and Hu, D. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. In Inter- national Conference on Machine Learning, pp. 52559- 52572. PMLR, 2024.
Li, H., Li, X., Hu, P., Lei. Y., Li, C., and Zhou, Y. Boosting multi-modal model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVF Interna- fional Conference on Computer Vision, pp. 22214-22224, 2023.
Wei, Y., Feng, R., Wang. Z., and Hu, D. Enhancing multi- modal cooperation via sample-level modality valuation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 17338-27344) 2024.
Lin, B., Lin, Z., Guo, Y., Zhang, Y., Zou, J., and Fan, S. Variational probabilistic fusion network for rgb-t semantic segmentation. arXiv preprint MxXiv:3307.08536 2023.
Wei, Y., Li, S., Feng, R., and Hu, D. Diagnosing and re- learning for balanced multimodal learning. In European Conference on Computer Vision, pp. 71-86. Springer, 2025.
McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
2018.
Wu, N., Jastrzehski, S., Cho, K., and Geras, K. Charac- terizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Con- ference on Machine Learning, pp. 24043-24055. PMLR, 2022.
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng. A. Y., et al. Multimodal deep learning. In ICML, volume 11, pp. 689-696, 2011.
Owens, A. and Efros, A. A. Audio-visual scene analy- sis with supervvid multisensory features. In Pro- ceedings of the European conference on computer vision (ECCV), pp. 631 231-648,220 2018.
Xu, R., Feng, R., Zhang, S.-X., and Hu, D. Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning. In ICASSP 2023-2023 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Peng, X., Wei, Y., Deng. A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8238-8247, 2022.
Yang. L., Wu, Z., Hong. J., and Long, J. Mcl: A contrastive learning method for multimodal data fusion in violence detection. IEEE Signal Processing Letters, 30:408-411 412, 2022.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general con- ditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Zhang, Q., Wei, Y., Han, Z., Fu, H.,, Peng, X., Deng, C., Hu, Q., Xu, C., Wen, J., Hu, D., et al. Multimodal fusion on low-quality data: A comprehensive survey, arXiv preprint wrXi:22404.18947 2024a.
Sun, T., Qian, Z., Li, P., and Zhu, Q. Graph interactive network with adaptive gradient for modti-modal rumor detection. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, pp. 316-324, 2023.
Zhang, X., Yoon, J., Bansal, M., and Yao, H. Multimodal representation learning by alternating unimodal adapta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27456- 27466, 2024b.
Sun, Y., Mai, S., and Hu, H. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 28: 1850-1611 2021.
Zhou, Y., Liang, X., Zheng, S., Xuan, H., and Kumada, T. Adaptive mask co-optimization for modal dependence in multimodal learning. In ICASSP 2023-2023 IEEE In- ternational Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Vielzeuf, V., Lechervy, A., Pateux. S., and Jurie, F. Cen- trainet: a multilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018
Zhu, Y., Wu, Y., Sebe, N., and Yan, Y. Vision+ x: A survey on multimodal learning in the light of data. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2024.
10


Reviews:
[{"review_id": "z13V4BgUNo", "reviewer": "Reviewer_hxRi", "summary": "The paper tackles the problem of multimodal learning and specifically how to make all the modalities to contribute equally to the training objectives. The authors suggest multiple steps to alleviate the issue, including decoupling\nmultimodal data and filtering hard samples for each modality to mitigate modality imbalance; and then batch-level reassembling to align the gradient directions and avoid cross-modal interference. The authors demonstrate the effectiveness of their method on CREMAD and Kinetic-Sound dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1:** The method and experiments are limited to certain modalities.\n\n**Response:** \n\n+ **Our method is not restricted to specific modalities**. In our theoretical analysis, we make no prior assumptions about modality properties, ensuring its general applicability. Meanwhile, the key steps of Remix —decoupling and reassembling—are **modality-agnostic**. The process only considers the **accuracy relationship between modality pairs** at sample-level without imposing any constraints on the modality type.\n\n+ **Our method is not limited by the number of modalities.** As the number of modalities increases, our method remains applicable by simply **retaining the modality with the lowest KL divergence during the decoupling process**. The selection  mechanism remains valid and ensures that our method remains effective in more complex scenarios involving **three or more modalities**.\n\n+ To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on the **UCF101** with two modalities(**optical flow, vision**)  and the **CMU-MOSEI** with three modalities(**text, vision, audio**). As shown in the table, Remix consistently improves performance, further validating its wide applicability. \n\n  |           | Baseline | GBlend | OGM   | PMR   | Reample | MLA   | Remix     |\n  | --------- | -------- | ------ | ----- | ----- | ------- | ----- | --------- |\n  | UCF101    | 80.78    | 82.82  | 82.55 | 81.87 | 84.09   | 83.03 | **84.59** |\n  | CMU-MOSEI | 83.32    | 84.45  | 85.03 | 84.13 | 84.50   | 82.84 | **85.89** |\n\n$\\quad$\n\n**Weakness2:** Effectiveness on Text modality and MLLMs.\n\n**Response:** \n\n+ Multimodal Large Language Models (**MLLMs**) differ significantly from our task in terms of architecture and **underlying principles**. MLLMs typically use the **language modality as the foundation**, mapping other modalities onto it. This differs from the modality imbalance issue we address in **co-decision tasks**. That's also the main difference between our method and ImageBind.\n+ Additionally, we follow the existing model structures in the **Balanced Multimodal Learning (BML)** domain to ensure a fair comparison and demonstrate the effectiveness of our proposed method.\n+ Additionally, we have demonstrated the effectiveness of the text modality within our method. In table above, we have supplement our results with the **CMU-MOSEI** dataset, which includes **text, video, and audio modalities**. In the **baseline model**, the accuracy of the **text modality** was **79.96%**, and after applying Remix, it improved to **81.29%**. This result indicates that **Remix is also beneficial for the language modality**, further validating its effectiveness across different modality types.\n\n$\\quad$\n\n**Weakness3:** There are no theoretical claims.\n\n**Response:**\n\n+ In this paper, we introduce a novel phenomenon termed **modality clash**, which defines the **bidirectional interference** between modalities in multimodal learning (as illustrated in Figure 3 in main text, we quantify the optimization direction deviation of strong modalities). This perspective differs from the traditional **modality imbalance** problem, which primarily emphasizes the **one-way suppression** of weaker modalities by stronger ones. \n+ Accordingly, our **Remix method** is designed to directly address this issue. To mitigate **modality clash**, we need to **control the composition of each training batch**, which is achieved through the **assembling** step. To **select appropriate samples** and enable *batch control*, we first perform **decoupling** of multimodal inputs based on **sample-level evaluation**.", "claims_and_evidence": "The authors claim to solve the problems of modality laziness and modality clash when jointly training multimodal models. They provide an experimental evidence comparing to other methods  as well as an Ablation study. My major concern is that the authors do not demonstrate the effectiveness of the for their technique for MLLM training, where adding modalities to language without degrading language performance is hard.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense, but limited to applications of audio and vision modalities only.", "theoretical_claims": "There are no theoretical claims, the method is several heuristic /greedy  steps: First  the complete dataset and multimodal inputs are used for warm-up training to ensure the model has the basic representational capability.  Then, the model is optimized through alternating steps where the multimodal inputs are decoupled  based on the KL divergence of unimodal prediction probabilities and reassemble the data at the batch level according to the remaining modality. Afterwards,  specific training for each modality is performed using the reassembled dataset", "experimental_designs_or_analyses": "Yes, the experimental design an analyses make sense, however is limited to certain modalities", "relation_to_broader_scientific_literature": "The authors provide a comprehensive overview of the related work.", "essential_references_not_discussed": "The related work does not discuss the ImageBind paper, published by Meta AI, which introduced a model that learns a joint embedding space across six modalities – images, text, audio, depth, thermal, and IMU data, enabling cross-modal retrieval and other applications.", "other_strengths_and_weaknesses": "In overall the paper is well written and easy to follow.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TOsA5xNYZG", "reviewer": "Reviewer_y3hZ", "summary": "This paper mainly proposes to combat with the issues of modality laziness and modality clash. Both issues happen when multimodal models prioritize learning from the strong modality and the batch gradient can be interfered across modalities. The authors propose the Data Remixing method to solve such problems. Specifically, a sample-level decoupling of multimodal data is utilized to emphasize the learning of weak modal samples, and a batch-level reassembling of unimodal data is proposed to ensure each batch only contains data from single modal. The authors conduct experiments on CREAMD and Kinetic-Sounds datasets to demonstrate the effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1:** Modality Clash and Modality Imbalance\n\n**Response:** \n+  In summary, **Modality Imbalance** is **unidirectional**, while **Modality Clash** is **bidirectional**. **Modality Imbalance** refers to a scenario where a strong modality dictates the learning process, preventing **other modalities** from being sufficiently trained. **Modality Clash** describes interference between modalities. *Even if modality balance is achieved*, differences between modalities may still lead to insufficient learning across **all modalities**.\n+  The accuarcy of unimodal capabilities presented in the following table demonstrate that **all modalities benefit from our approach**, further validating its effectiveness.\n\n|Dataset|Baseline|Remix|Dataset|Baseline|Remix|\n|-|-|-|-|-|-|\n| CREMAD(A+V) | 64.52|**72.72**|KS(A+V)|50.23|**55.63**|\n| CREMAD(V)|41.67|**53.63**|KS(V)|29.03| **42.68** |\n| CREMAD(A)|53.76|**54.57**|KS(A)|40.32| **44.06** |\n\n$\\quad$\n\n**Weakness2:** The Mutual Information in Multimodal Tasks.\n\n**Response:** We understand the reviewer's concerns regarding the potential decrease in MI due to alternating unimodal training. This issue is carefully considered in our approach. \n+ First and foremost, it is important to clarify that our task is **multimodal co-decision**, which involves **integrating multiple modalities to make a final decision**—abstractly represented as **A + B → C**. The goal is to perform **cross-modal feature selection and integration** based on the **learning outcomes of individual modalities**. This differs from tasks such as **retrieval**, where decision-making relies on **capturing shared information across modalities**—abstractly represented as **A → B**. These tasks require **highlighting consistency information** through similarity measurements, which **inherently mitigates modality imbalance** issues present in co-decision tasks.\n+ Following the reasons, existing co-decision network architectures are inherently **weakly interactive**. For example, they often rely on direct *concatenation* of unimodal representations or *gated fusion* at the decision level. As a result, **our data decomposition does not significantly impact interaction**, since these models already operate with minimal cross-modal dependency.  By conducting experiments measuring the MI between modality-specific features before and after applying our method, we prove our perspective.\n\n||Baseline|Remix|\n|-|-|-|\n|CREMAD-MI|0.078| 0.077|\n|KineticSound-MI|0.016|0.013|\n\n+ Therefore, in co-decision tasks, the **performance bottleneck** lies in **modality imbalance** and **modality clash**, which hinder effective unimodal learning and limit cross-modal synergy. Our work is specifically designed to address this issue by ensuring that each modality is adequately learned before integration, ultimately improving overall model performance.\n+ Additionally, we provide extra data to further support our argument. We construct a **confusion matrix** analyzing the relationship between **model decision correctness** and the **presence of correctly predicted modalities**. The results can be found at: https://anonymous.4open.science/r/ICM-Rebuttal-17C8/Fig1.png. By analyzing it, we find that in co-decision tasks, it is common for a model to have at least one modality predict correctly while the final decision is incorrect. However, after applying the Remix method, this type of misjudgment is **significantly reduced**. This suggests that **Remix can guides samples to the appropriate modality space, and improves decision-making accuracy**.\n\n$\\quad$\n\n**Weakness3:** The method and experiments are limited.\n\n**Response:** \n+ **Our method is not restricted to specific modalities**. In our theoretical analysis, we make no prior assumptions about modality properties, ensuring its general applicability. Meanwhile, the key steps of Remix —decoupling and reassembling—are **modality-agnostic**. The process only considers the **accuracy relationship between modality pairs** at sample-level without imposing any constraints on the modality type.\n+ **Our method is not limited by the number of modalities.** As the number of modalities increases, our method remains applicable by simply **retaining the modality with the lowest KL divergence during the decoupling process**. The selection  mechanism remains valid and ensures that our method remains effective in more complex scenarios involving **three or more modalities**.\n+ To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on the **UCF101** with two modalities(**optical flow, vision**)  and the **CMU-MOSEI** with three modalities(**text, vision, audio**). As shown in the table, Remix consistently improves performance, further validating its wide applicability. \n\n  || Baseline |GBlend |OGM| PMR| Reample|MLA|Remix|\n  |-|-|-|-|-|-|-|-|\n  |UCF101|80.78|82.82 | 82.55 | 81.87 | 84.09| 83.03 | **84.59** |\n  |CMU-MOSEI|83.32| 84.45| 85.03 | 84.13 | 84.50| 82.84 | **85.89** |", "claims_and_evidence": "The modality laziness that proposed to be solved in this paper is a well-known issue of multimodal imbalance that is worthy to be researched. In contrast, the proposed modality clash is somehow less-known, which seems like a consequence or a phnomenon caused by modality imbalance, instead of another issue as the authors claim. Thus the novelty of this paper can be over-claimed, since the main contribution of multimodal decoupling is merely a sample-wise fine-grained of previous methods, and the reassembling part is problematic from my opinion, which will be explained in the following Methods part.", "methods_and_evaluation_criteria": "My major concern is about the proposed method. While I am convinced that the proposed Data Remixing can solve the problem of modality laziness, such strategy is considered to violate the principles of multimodal learning. I am mostly ok with the decoupling method, as it explicitly discriminate samples of weak modality. However, the selected data is then reassembled into unimodal-form batches, where the multimodal model independently learns from a single modality in each batch with input of other modalities masked with zeros. If there is no misunderstanding (supported by Fig. 1(c), \"0 masked audio\" in Fig. 2(c) and pseudo algorithm 1), the mutual information across-modality is never learned throughout the procedure of Data Remixing. Thus such method is doubt to be a simple ensemble of unimodal models, where modality laziness of course doesn't exist. While the authors only provide experiments on two audio-visual datasets with classification tasks, such tasks can be accomplished with less dependency on cross-modal knowledge. I wonder if the author can provide further experiments on more complicated understanding tasks like cross-modal retrieval or reasoning tasks, on multimodal scenarios with more mutual information, like image-text modalities, or the authors can provide more explanation to how Data Remixing learns such mutual information.", "theoretical_claims": "The authors provide theoretical analysis on how Data Remixing solve the problem of modality clash in lines 220-250, which is considered trivial. Such details are recommended to be placed in Appendix.", "experimental_designs_or_analyses": "The authors only conduct experiments two small-scale on dual modality datasets under classification tasks. More experiments on   more popular datasets like Conceptual Captions and YFCC datasets in image-text scenarios, with diverse down-stream tasks including retrieval, reasoning, VQA, grounding tasks should be considered.", "relation_to_broader_scientific_literature": "The considered issue of modality laziness is important, however novelty of the method is considered limited.", "essential_references_not_discussed": "No extra references recommened.", "other_strengths_and_weaknesses": "The legend and footnotes of Fig. 1 and Fig. 2 are too small that can hardly be read. The writing of method part is also recommended to be improved.", "other_comments_or_suggestions": "Please refer to previous parts.", "questions_for_authors": "Please refer to previous parts. My major concern includes the violation of the method to multimodal learning purpose that neglect the learning of mutual information, together with the limited novelty and insufficient experiments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NgMbZT2gHD", "reviewer": "Reviewer_JKcr", "summary": "The authors address the problems of modality laziness and cross-modal clash in multimodal joint learning at the same time. They propose a method by remixing the original multi-modal input pairs, which involves decoupling multimodal data into unimodal subsets and selecting difficult samples for each modality to avoid modality dominance, and then reassembling them at the batch level to enable gradient alignment and avoid cross-modal interference. The experiments on public datasets show large improvements over existing methods, the generality on various fusion models and their compatibility with other methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1:** The applicability of the Remix method on three or even more modalities.\n\n**Response:** \n\n+ **Our method is not limited by the number of modalities.** As the number of modalities increases, our method remains applicable by simply **retaining the modality with the lowest KL divergence during the decoupling process**. The selection  mechanism remains valid and ensures that our method remains effective in more complex scenarios involving **three or more modalities**.\n+ To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on **CMU-MOSEI** with three modalities(**text, vision, audio**). As shown in the table, Remix consistently improves performance, further validating its wide applicability. \n\n|           | Baseline | GBlend | OGM   | PMR   | Reample | MLA   | Remix     |\n| --------- | -------- | ------ | ----- | ----- | ------- | ----- | --------- |\n| CMU-MOSEI | 83.32    | 84.45  | 85.03 | 84.13 | 84.50   | 82.84 | **85.89** |\n\n$\\quad$\n\n**Weakness2:** Analysis of modality imbalance in fusion-based decision-making.\n\n**Response:** \n\n+ **We conducted a further analysis of feature fusion during model decision-making.** We observe that modality imbalance is not limited to **unimodal encoders** but also manifests significantly at the **feature fusion layer**. Taking the concatenation method as an example, both the audio and video modality outputs are 512-dimensional, resulting in a 1024-dimensional fused feature. We compute the weighting distribution of each modality, and the results are presented in the table below.\n\n|              |        | Baseline |            |        | Remix  |            |\n| ------------ | ------ | -------- | ---------- | ------ | ------ | ---------- |\n|              | Audio  | Video    | Ratio(A/V) | Audio  | Video  | Ratio(A/V) |\n| CREMAD       | 0.0258 | 0.0179   | 1.441      | 0.0215 | 0.0199 | 1.080      |\n| KineticSound | 0.0314 | 0.0219   | 1.433      | 0.0287 | 0.0245 | 1.171      |\n\n+ Our findings indicate that the **Remix method not only promotes modality balance at the unimodal encoder level but also enhances balance at the fusion layer**. \n\n+ For more complex models(MMTM or CentalNet), the improvements brought by Remix are relatively smaller. We attribute this to the fact that early-stage interactions may be somewhat suppressed due to our decoupling process. However, **this trade-off is offset by the enhanced unimodal capabilities**, which ultimately contribute to overall performance improvements.\n\n $\\quad$\n\n**Weakness3:** Other metrics to evaluate the performance of the modalities to determine the decoupling results?\n\n**Response:** \n\n+ We are inspired by **the measurement of uncertainty** in Active Learning when choosing KL divergence as a sample-level evaluation method and a criterion for selecting modality-specific training samples. \n+ We have provided training results using other feasible metrics for comparison with results presented in the following table and summary their weaknesses below.\n  + **Entropy:** In classification tasks, Entropy and KL divergence are mathematically equivalent.\n  + **Loss:** Loss tends to overly bias the data toward the weak modality, causing preventing the strong modality from effective training.\n  + **Shapley Value:** Shapley values sometimes fail to distinguish between modalities, as noted in *Footnote 1* of the main text.\n\n|             | Baseline | Loss  | Shapley | KL Divergence |\n| ----------- | -------- | ----- | ------- | ------------- |\n| CREMAD(A+V) | 64.52    | 69.89 | 68.28   | **72.72**     |\n| KS(A+V)     | 50.23    | 54.78 | 53.93   | **55.63**     |", "claims_and_evidence": "Yes, the claims are well supported by extensive experiments and analyses.", "methods_and_evaluation_criteria": "Yes, the proposed method is theoretically feasible and effective.", "theoretical_claims": "I have verified the correctness of the theoretical claims, and there are no issues.", "experimental_designs_or_analyses": "I reviewed the experimental designs of the paper. The experiments  and analyses are sufficient to support the claims.", "relation_to_broader_scientific_literature": "The motivation and ideas differ from others:\n+ Previous methods only focused on modality laziness, while this paper explicitly proposes that the impact of modality imbalance is bidirectional: weak modalities are suppressed by strong modalities, and at the same time, the two modalities interfere with each other's optimization directions.\n+ Previous methods did not address the issue of modality conflict in joint learning from the deeper perspective of batch-level gradient inconsistency. To the best of my knowledge, this is the first work to address modality imbalance at the batch level.", "essential_references_not_discussed": "No essential references are left undiscussed.", "other_strengths_and_weaknesses": "## Strengths\n\n+ Novelty: the data remixing method is novel, especially the idea of batch-level data reassembly to prevent cross-modal gradient interference.\n+ Practicality: the method does not require dataset expansion.\n+ Large improvement and generalizability: experiments on different multimodal fusion methods and architectures are conducted, and the improvement is large and consistent, verifying its efficacy and generalizability.\n+ Compatibility: it can be integrated with other existing methods such as MLA and Resample.\n+ The paper is well organized.\n\n## Weaknesses\n\n+ For the issue of multimodal fusion balance, although most existing works focus primarily on two modalities, I still hope to see the performance of this approach on three or even more modalities. Of course, this would bring new challenges to the design of the resembly strategy. However, it could further demonstrate the generality of the proposed method. Adding this part in the future will further enhance the contribution of the proposed method. \n+ In section 4.4, the athours reports the consistent improvement on different fusion architectures. However, I notice that the improvement varies and it is smaller on more complex fusion architectures. I speculate that the reason for this phenomenon lies in the fact that these more complex fusion models inherently include some adaptive adjustment strategies for feature selection. For example, MMTM recalibrates feature channels from different streams, and CentralNet simultaneously considers the contributions of individual modality features and fused features in decision-making. The authors should include a deeper analysis of this variation in improvement magnitude in the paper. This would better reveal the fundamental causes of modality imbalance and provide a basis for understanding the effectiveness of various strategies based on their contributions.", "other_comments_or_suggestions": "It is recommended to add more descriptions in Fig. 1 and Fig. 2 to make them easier to follow. For example, in Fig. 1, the motivation of the middle column that masking the strong modality can be added. In Fig. 2(c), the batch-level reassembly should be clearly presented.", "questions_for_authors": "+This method includes two steps: data decoupling and resembling. So my question is whether it can be used for online training when the data is dynamically changing.\n+Apart from KL divergence, has the author considered other metrics to evaluate the performance of the modalities to determine the decoupling results? For example, would entropy be more robust?\n+How necessary is the warm-up phase, and can we reduce its duration or directly use pre-trained unimodal encoders instead to further improve the efficiency of multimodal training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WRWqIrglns", "reviewer": "Reviewer_Zifd", "summary": "This paper introduces a method called Data Remixing to alleviate modality laziness and modality clash, which guarantees both sufficiency and multimodal balance. The authors demonstrate that batch-level gradient direction conflicts lead to modality imbalance. Firstly, the authors divide the samples into K subsets based on which modality the model learns worst, which is indicated by the KL divergence between unimodal prediction logits and uniform distribution. Secondly, the authors use these subsets to reassemble batch data and update the model. The ablation experiments show the effectiveness of the method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness1:** There are no concrete measurements to demonstrate the efficiency of Remix.\n\n**Response:** \n+ Our method focuses on variations at the data level and we have reported **size of training set** in Table 2 of the main text to prove the efficiency.\n+ We measure the **training time** of 4 methods under the same conditions as shown below. We observe that sample-level evaluations tend to increase the training time, whereas **Remix does not expand the training set, making it more efficient**. \n\n||Baseline|Remix|Resample|MLA|\n|-|-|-|-|-|\n|CREMAD(sec)|**1536**|2357|4525|6128|\n|KS(sec)|**3849**|4946|10362|12868|\n\n$\\quad$\n\n**Weakness2:** Lack of justification for the effectiveness of the KL metric and evaluation of unimodal performance.\n\n**Response:** \n+ We are inspired by **the measurement of uncertainty** in Active Learning when choosing KL divergence as a sample-level metric and select specific training samples. \n+ We have provided training results using other feasible metrics for comparison with results presented in the following table and summary their weaknesses below.\n  + **Entropy**: Mathematically equivalent with KL.\n  + **Loss:** Loss tends to overly bias the data toward the weak modality, preventing the strong modality from effectively training.\n  + **Shapley:** Shapley values sometimes fail to distinguish between modalities, as noted in *Footnote 1* of the main text.\n\n||Baseline|Loss|Shapley|KL|\n|-|-|-|-|-|\n| CREMAD(A+V)| 64.52| 69.89|68.28|**72.72**|\n| CREMAD(V)| 41.67| **54.17** |53.49 |53.63|\n| CREMAD(A)| 53.76| 52.69|53.90|**54.57**|\n| KS(A+V)| 50.23| 54.78|53.93|**55.63** |\n| KS(V)| 29.03|**44.80** |42.41|42.68|\n| KS(A)| 40.32|40.21|40.90|**44.06**|\n\n+ Another important point is that by observing the unimodal accuracy, we can find that Remix alleviates the insufficient modality learning, where the strong modalities are also improved.\n\n $\\quad$\n\n**Weakness3:** The performance in other modality combinations and multimodal scenarios.\n\n**Response:** \n+ **Our method is not restricted to specific modalities**. In our theoretical analysis, we make no prior assumptions about modality properties, ensuring its general applicability. Meanwhile, the key steps of Remix —decoupling and reassembling—are **modality-agnostic**. The process only considers the **accuracy relationship between modality pairs** at sample-level without imposing any constraints on the modality type.\n+ **Our method is not limited by the number of modalities.** As the number of modalities increases, our method remains applicable by simply **retaining the modality with the lowest KL divergence during the decoupling process**. The selection mechanism remains valid in more complex scenarios involving **three or more modalities**.\n+ To further demonstrate the broad effectiveness of Remix, we conduct additional experiments on the **UCF101** with two modalities(**optical flow, vision**)  and the **CMU-MOSEI** with three modalities(**text, vision, audio**). As shown in the table, Remix consistently improves performance, further validating its wide applicability. \n\n|| Baseline |GBlend |OGM| PMR| Reample|MLA|Remix|\n|-|-|-|-|-|-|-|-|\n|UCF101|80.78|82.82|82.55|81.87|84.09|83.03|**84.59**|\n|CMU-MOSEI|83.32|84.45|85.03|84.13|84.50|82.84|**85.89**|\n\n$\\quad$\n\n**Weakness4:** About Dropout and Head in ablation study.\n\n**Response:** \n+ **Dropout and Head are strategies to obtain unimodal outputs**. **Dropout** refers to masking other modalities and using *the output of the multimodal classification head* as the unimodal output. **Head** involves adding an *independent classification head* to each encoder and using its output as the unimodal output.\n+ In Remix method, we select the **Head** approach for more accurate unimodal results. To synchronously update the parameters, we incorporate a unimodal loss into the loss function, which has been proved to improve multimodal capabilities. This ablation study aims to demonstrate that the improvements brought by Remix do not stem from the introduction of unimodal loss.\n\n$\\quad$\n\n**Weakness5:** How to ensure that using KL divergence doesn't cause the model to learn from excessively noisy signals?\n\n**Response:** \n+ We also encountered similar issues in our experiments. Therefore, we have acknowledged this potential limitation in the summary of main text. Our current approach to address this challenge involves **setting additional thresholds**. \n+ In UCF101, we observe that the KL divergence for optical flow are consistently smaller than those for visual. We adopted two strategies:\n  + **Scaling-Based Adjustment:** We introduce a hyperparameter $\\beta$ to scale the KL divergence of the optical flow and compare $KL_{\\text{video}}$ with $\\beta \\times KL_{\\text{flow}}$, achieving a performance of **84.59%**. \n  + **Minimum KL Threshold:** We set a lower bound $\\alpha$ on KL divergence to ensure the model does not overly focus on samples with minimal information content, resulting in a performance of **84.01%**.", "claims_and_evidence": "The paper presents a framework for addressing modality laziness and clash through batch-level optimization, supported by empirical evidence. While the method shows promise in audio-visual tasks, its effectiveness remains untested in broader scenarios. This raises doubts about whether the approach truly generalizes beyond the tested modalities.\n\nAdditionally, while the paper states there is \"no additional computational overhead during inference,\" this assertion would be strengthened by including specific measurements. A comparative analysis of inference time and memory usage between the proposed method and established baselines would provide valuable insights into the practical scalability of the approach in real-world applications.", "methods_and_evaluation_criteria": "The method provides a framework from a data perspective to alleviate modality laziness. The evaluation criteria is intuitive and sound, but additional evidence would strengthen the demonstration of its effectiveness. Specifically, the paper would benefit from: (1) comprehensive comparisons between unimodal performance baselines and the corresponding unimodal branches in the proposed method, (2) ablation studies testing alternative metrics to KL divergence for modality evaluation, and (3) quantitative measurements of computational efficiency during both training and inference phases. These additional empirical validations would more conclusively establish the method's advantages over existing approaches.", "theoretical_claims": "The theoretical proofs are correct but its contribution is not particularly significant.", "experimental_designs_or_analyses": "The experimental designs and analyses in this paper are reasonable,  but insufficient. Firstly, the paper lacks the comparisons between different methods and architectures on unimodal performance. For example, the paper should demonstrate unimodal baselines (audio-only, visual-only) on CREMAD and compare their accuracy with the corresponding unimodal branches in Data Remixing and other imbalanced multimodal learning methods. By presenting this, the paper can validate the improvement of multimodal learning imbalance further. Secondly, the paper lacks the experiments to validate the effectiveness of KL divergence-based evaluation method. It could conduct ablation studies comparing KL divergence with other alternative metrics for the proposed method. Thirdly, the paper argues that low-KL samples are “insufficiently trained”, but this is an interpretation without proof. It should visualize feature distributions (e.g. t-SNE) of low-KL samples before and after remixing, or ablate the remixing step for low-KL samples and observe if their accuracy drops significantly. Finally, the paper claims its training efficiency, but it lacks quantitative evidence like training time or FLOPs during training to support this claim.", "relation_to_broader_scientific_literature": "This paper makes an important contribution to the field of imbalanced multimodal learning by addressing the fundamental challenges of modality laziness and modality clash. The authors' analysis of batch-level optimization conflicts and their KL-based method for evaluating modality-specific learning provides valuable insights for the community. This work builds upon previous approaches but takes a more data perspective. However, while the proposed Data Remixing method shows promising results, the paper would be strengthened by more thorough experimental comparisons with state-of-the-art imbalanced multimodal learning techniques such as gradient modulation methods, prototype learning, and knowledge distillation approaches.", "essential_references_not_discussed": "The paper's citations appear comprehensive, covering major works in the field.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel sample-level evaluation method for assessing unimodal learning capacity, which is computationally efficient and readily extensible to multiple modalities.\n2. The proposed framework demonstrates strong flexibility as it operates independently of model architecture, allowing it to be combined with various existing multimodal learning methods.\n3. The work provides an insightful analysis of modality clash at the batch-level optimization stage, offering an intuitive explanation that advances our theoretical understanding of multimodal learning challenges.\n\nWeaknesses:\n1.  Despite claims of efficiency, the paper lacks concrete measurements quantifying this overhead in terms of training time, memory usage, or computational complexity compared to baseline methods.\n2. The effectiveness of using KL divergence as the sample-level evaluation metric remains insufficiently justified. The paper lacks comparative analysis with alternative metrics and doesn't provide empirical evidence showing why this particular approach is optimal for identifying modality-specific training needs.\n3. The experimental validation is limited to audio-visual tasks with relatively simple fusion methods. Additional experiments on more diverse modality combinations and complex multimodal scenarios would better demonstrate the method's generalizability.", "other_comments_or_suggestions": "1. Some expressions in the paper lack clear references. For example, the Resample method in the experiment section is not properly cited, and the headers for dropout and head in the ablation study are confusing.\n2. The rationale for selecting KL divergence as the evaluation metric deserves more thorough explanation. The paper should explicitly discuss why this metric is suitable for assessing modality learning capacity.", "questions_for_authors": "1. How does the KL divergence distribution change during the training process? \n2. Given that different samples inherently contain varying degrees of modality information, how do you ensure that using KL divergence as your discrimination metric doesn't cause the model to learn from excessively noisy signals?\n3. What is the specific computational overhead during model training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the common multimodal learning problems of "modality laziness" (over-reliance on one data type) and "modality clash" (conflicting learning signals). The authors propose a "Data Remixing" method, which involves decoupling data into single modalities, focusing on harder-to-learn samples for each, and then reassembling these into specialized batches. Experiments on the CREMAD and Kinetics-Sounds datasets demonstrate that this technique improves model performance by ensuring more balanced contributions from all modalities and mitigating interference during training.

The reviewers in general are positive about the submission and agreed the experiments demonstrated the effectiveness of the proposed approach. There was still concern on the design principle in the reassembling step of the approach, with which model never explicitly learns the mutual information or the relationship between the different data types. The authors are encouraged to add more clarification and experiments on this in the next version.
Decision: Accept (poster)