M3CoL: Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification
Authors: Raja Kumar
Raghav Singhal
Pranamya Prashant Kulkarni
Deval Mehta
Kshitij Sharad Jadhav
Venue: Accepted by TMLR ()

Abstract: Deep multimodal learning has shown remarkable success by leveraging contrastive learning to capture explicit one-to-one relations across modalities. However, real-world data often exhibits shared relations beyond simple pairwise associations. We propose M3CoL, a Multimodal Mixup Contrastive Learning approach to capture nuanced shared relations inherent in multimodal data. Our key contribution is a Mixup-based contrastive loss that learns robust representations by aligning mixed samples from one modality with their corresponding samples from other modalities thereby capturing shared relations between them. For multimodal classification tasks, we introduce a framework that integrates a fusion module with unimodal prediction modules for auxiliary supervision during training, complemented by our proposed Mixup-based contrastive loss. Through extensive experiments on diverse datasets (N24News, ROSMAP, BRCA, and Food-101), we demonstrate that M3CoL effectively captures shared multimodal relations and generalizes across domains. It outperforms state-of-the-art methods on N24News, ROSMAP, and BRCA, while achieving comparable performance on Food-101. Our work highlights the significance of learning shared relations for robust multimodal learning, opening up promising avenues for future research.

Published in Transactions on Machine Learning Research (05/2025)
M3CoL: Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification
Raja Kumart1, Raghav Singhal Pranamya Kulkarni', Deval Mehta?, and Kshitij Jadhay'
Indian Institute of Technology Bombay, Mumbai, India 2AIM for Health Lab, Department of Data Science & AI, Monash University, Australia
Reviewed OpenReview:
Abstract
Deep multimodal learning has shown remarkable success by leveraging contrastive learning to capture explicit one-to-one relations across modalities. However, real-world data often exhibits shared relations beyond simple pairwise associations. We propose M3CoL, a Multimodal Mixup Contrastive Learning approach capture nuancod shored relations inhorent in multimodal data. Our key contribution Mixup-based contrastive loss that learns robust representations by aligning mixed samples from one modality with their corresponding samples from other modalities thereby capturing shared relations between them. For multimodal classification tasks, we introduce a framework that integrates fusion module with unimodal prediction modules for auxiliary supervision during training, complemented by our proposed Mixup-based contrastive loss. Through extensive experiments on diverse datasets (N24News, ROSMAP, BRCA, and Food-101), we demonstrate that M3CoL effectively captures shared multimodal relations and generalizes across domains. Tt outperforms state-of-the-art methods on N24News, ROSMAP, and BRCA, while achieving comparable performance Food-101. Our work highlights the siguificance of learning shared relations for robust multimodal learning, opening up promising avenues for future research.
1 Introduction
The way we perceive the world shaped by various modalities, such as language, vision, audio, and more. In the era of abundant and accessible multimodal data, it increasingly crucial to equip artificial intelligence with multimodal capabilitics (Baltrušaitis et al., 2018). At the heart of advancements multimodal learning is contrastive learning, which maximizes similarity for positive pairs and minimizes it for negative pairs, making it practical for multimodal representation learning. CLIP (Radford et al, 2021) is prominent example that employs contrastive lcarning to understand the direct link between paired modalities and seamlessly maps images and text into a shared space for cross-modal understanding, which can be later utilized for tasks such as retrieval and classification. However, traditional contrastive learning methods often overlook shared relationships between samples across different modalities, which can result in the learning of representations that are not fully optimized for capturing the underlying connections between diverse data modalities. These methods primarily focus on distinguishing between positive and negative pairs of samples, typically treating each instance as independent entity. They tend to disregard the rich, shared relational information that could exist between samples within and across modalities. This limited focus can prevent the model from leveraging valuable contextual information, such as semantic similarities or complementary patterns, which can enhance robust representation learning. Consequently, this can lead to suboptimal performance in downstream tasks that require optimized shared representations, such as image-text alignment, cross-modal retrieval, or multimodal fusion tasks.
Equal Contributions. Author ordering determined by coin flip over Google Meet.
Our code available https:// wtwpah...co//aagaaaiiiaaaaaa

---

Published in Transactions on Machine Learning Research (05/2025)
M
Mi
?
M
M
M
Spaghetti with tomato basil, parmesan
+Sim
Spaghetti tornaoo sauce, basil, and parmesan
cheese, tomato sauce, basil, and toppings
2
1
1
M
M (2)
Pizza with cheese, tomato sauce, basil, assorted tappings
+Sim
M (1) LJ
M (2) L.
Sim
+Sim
+Sim
+Sim
M (1)
M
? Mi
M (2)
M (2)
Spaghetti with tomato sauce, basil, and
cheese, tomato sauce, basil, and assorted toppings
cheese, toraao sauce, basil, and assorted toppings
-Sim
Figure 1: Comparison of traditional contrastive and our proposed M3Co loss. M." and M denote representations of the i-th sample from modalities 1 and 2, respectively. Traditional contrastive loss (left panel) aligns corresponding sample representations across modalities. M3Co (right panel) mixes the i-th and j-th samples from modality 1 and enforces the representations of this mixture to align with the representations of the corresponding i-th and j-th samples from modality 2, and vice versa. For the text modality, we mix the text embeddings, while we mix the raw inputs for other modalities. Similarity (Sim) represents type of alignment enforced between the embeddings for all modalities.
While traditional contrastive learning methods treat paired modalities as positive samples and non- corresponding ones as negative, they often overlook shared relations between different samples. As shown in the left panel Figure 1 (Left panel), classical contrastive learning approach assumes perfect one-to-one relations between modalities, which is rare in real-world data. For example, shared elements in images text relate even across separate samples, illustrated by the elements like "tomato sauce" and "basil" in Figure 1. Our approach, illustrated in the right panel of Figure 1, goes beyond simple pairwisc alignment by capturing shared relationships across mixed samples. By creating newer data points through convex combinations of data points our method more effectively models complex shared relationships, such as imperfect bijections (Liang et al., 2022a), enhancing multimodal classification performance.
Our approach builds upon the success of data augmentation techniques such as Mixup (Zhang et al., 2017) and their variants (Yun al., 2019; Cubuk et al., 2019; Hendrycks al., 2019), which have proven beneficial for enhancing learned feature spaces, improving both robustness and performance. Mixup trains models synthetic data created through convex combinations of two datapoint-labe pairs (Chapelle et al., 2000). These techniques are particularly valuable in low sample settings, as they help prevent overfitting and the learning of ineffective shortcuts (Chen et al., 2020; Robinson et al., 2021), common in contrastive learning. Building on the success of recent Mixup strategics (Shen et al., 2022; Thulasidasan et al., 2019; Verma et al., 2019) and MixCo (Kim et al., 2020), we introduce M3Co, a novel approach that significantly adapts and enhances contrastive Icarning principles to complex multimodal settings. M3Co modifies the CLIP loss effectively handle multimodal scenarios, addressing the problem of instance discrinination, where models overly focus on distinguishing individual instances instead of capturing relationships between modalities. By leveraging convex combinations of data for contrastive learning, M3Co eliminates instance discrimination and enhances robust representation learning by capturing shared relations. These combinations serve as structured noise and treated as positive pairs with their corresponding samples from other modalities. Our experimental results demonstrate enhanced ability to capture shared relations enabling improvements in performance and genoralization across range of multimodal classification tasks.
Our key contributions summarized follows:
2

---

Published in Transactions on Machine Learning Research (05/2025)
We propose M3Co, multimodal contrastive loss (Eq. 8) that utilizes mixed samples to effectively capture shared relationships across different modalities. By going beyond traditional pairwise aligmment methods, M3Co makes representations more consistent with the complex, intertwined relationships usually observed in real-world data.
We introduce a multimodal learning framework (Figure 2) consisting of unimodal prediction modules, a fusion module, and a novel Mixup-based contrastive loss. Our proposed method modality-agnostic, allowing for flexible application across various types of data, and continuously updates the representations necessary for accurale and consistent predictions.
We demonstrate the effectiveness of our methodology by evaluating it four public multimodal classification benchmark datasets from different domains: two image-text datasets, N24News and Food-101, and two medical datascts, ROSMAP and BRCA (Table 1, 2, 3). Our approach outperforms baseline models, especially on smaller datasets.
2 Methodology
Pipeline Overview: Figure depicts our framework, which compriscs of three components: unimodal prediction modules, fusion module, and a Mixup-based contrastive loss. We oblain latent representations (using learnable modality specific encoders f()) and f(2) of individual modalities and fuse them (denoted by concatenation symbol '+") to generate joint multimodal representation, which is optimized using a supervised objective (through classifier 3). The unimodal prediction modules provide additional supervision during training (via classifier and 2). These strategies enable deeper integration of modalities and allow the models to compensate for the weaknesses of one modality with the strengths of another. The Mixup- based contrastive loss (denoted by Lusco) continuously updates the representations by capturing shared relations inherent in the multimodal data. This comprehensive approach enhances the understanding of multimodal data, improving accuracy and model robustness.
Multimodal Mixup Contrastive Learning: Given a batch of N multimodal samples, let x(" and xx() denote the i-th samples for the first and second modalities, respectively. The modality encoders, f()) and (2)
f(2), generate the corresponding embeddings p; (1) and p.
P = (x(),, p = (x²)
(1)
We generate a mixture, x of the samples x and xf" by taking their convex combination. Similarly, we generate a mixture, (2) X using the convex combination of the samples xx") and x(") (Eq. 2). In the case of text modality, instead of directly mixing the raw inputs, we mix the text embeddings (Guo et al., 2019). The mixing indices j, k are drawn arbitrarily, without replacement, from N1, for both the modalities. We mix both the modalities using a factor À ~ Beta(a, a). Based on the findings of (Zhang et al., 2017), which demonstrated enhanced performance for a values between 0.1 and 0.4, we chose a 0.15 after experimenting with several values in this range. The mixtures are fed through the respective encoders to obtain the embeddings: (()) and P (Eq. 3).
A x (1) + (1 - Ax) -xxx
x X, x (2) + (1 X1) x(()
(2)
P (1) f (x()),
p!? (() (x
(3)
The unidirectional contrastive loss (Sohn, 2016; Chen et al., 2020; Oord el al., 2018; Wu et al., 2018; Zhang et al., 2022) over 222) is conventionally defined as:
exp (1) (p, .p. .p..9// /T)
N log
Laim-cony (2)
1
(4)
N
Σ exp (p!") p/,?//1
N
1=1
1=1
3

---

Published in Transactions on Machine Learning Research (05/2025)
(1)
(1)
Classifier-1 y3 (1)
x1"
Pi
x (1)
Encoder f(1)
Pij
(1)
1
(1)
(1)
Pj
L33C
Classifier-3
y final
(2)
(2)
Pk
x(2
Encoder f(2)
D
ik
(2)
(2)
Classifier-2 ..... Di (2)
Pi
Learnable
Concatenation
Training only
Training Inference
(1) x and modality (x (2)) along with their respective mixed data x(1) and x are fed into encoders (() and f()) to generate embeddings. Unimodal embeddings p.") and p. are processed through classifier 1 and 2 to produce predictions g( and y:22) for training supervision only. The unimodal embeddings (()) and p are concatenated and processed through classifier to yield ýfnal, utilized during training and inference. Additionally, unimodal embeddings p. Pj (1) p. Pk and mixed embeddings pi.j and P.k are utilized by our contrastive loss Lm3Co for shared alignment.
Figure 2: Archilecture of our proposed M3CoL model. Samples from modality 1 (x("),
(2)
where indicates dot product and is temperature hyperparameter. While this formulation is needed for computing similarity among aligned samples from different modalities, our loss handles both aligned and non-aligned samples, as this enables to learn better representation space. To achieve this, we define the unidirectional multimodal contrastive loss between p," (1) and (2) over (2) as:
exp
P
(1)
(2)
/TT
Csim (p,") (p(); log
(5)
N Σ
exp
p pp .p,./// /T)
==1
where (!) and (?)) are 2 normalized, is temperature hyperparameter, and m is sample index in [] Although the unidirectional multimodal contrastive loss (Eq. 5) can learn indirect relations, it is insufficient for learning shared semi-positive relations between modalities. Therefore, we introduce a Mixup-based contrastive loss to capture those relations that promotes generalized lcarning, as this process is more maancdd than simply discriminating positives from negatives. Now, we make our loss bidirectional to encourage improved alignment the shared representation space and efficient use of training data (Radford al., 2021; Oord al., 2018; Sohn, 2016). We define this bidirectional Mixup contrastive loss M3Co for each modality (Eq. 6, 7) and the total M3Co loss (Eq. 8) as:
N
(1) M3Co
A Ceim Ca p())
(.) + A²) Lp (P))) ()));
A Cuim (1))) + (1 - Aa) Coim (p, (2)
(6)
N
1=1
4

---

Published in Transactions on Machine Learning Research (05/2025)
N
(
A Sim (p P () + -1--0 (1-x,) falm (D (2)
i, (1) ki)
M3Co
N
i=1
A4 Caim (pi (2)
:)) (1 ())
X.) Ca (2) .p())
(7)
N
1=
c11.2) MBCO
2 (C + MMCC)
(8)
where (1)), ((1) ( and p(2) are L normalized. Note that the parts of the loss functions Eq. (6, 7) inside curly parenthesis make them bidirectional. tixuppbbsed methods enhance generalization by capturing clean patterns in the carly training stages but eventually overfit to noise if continued for larger number of epochs (Liu et al., 2023; Yu et al., 2021; Golatkar et al., 2019). To address this, implement schedule that transitions from the Mixup-based M3Co loss to non-Mixup multimodal contrastive loss. We design this transition so that the non-Mixup loss retains the ability to learn shared or indirect relationships between modalities. By using bidirectional SoftClip-based loss (Gao et al., 2024; Sohn, 2016; Chen et al., 2020), we relax the rigid one-to-one correspondence, allowing the model to capture many-to- to-many relations (Gao et al., 2024; 2022). The bidiroctional MultiSoftClip loss for each modality (Eq. 9, 10) and its combination (Eq. 11) is:
N
exp
P (1)
PP: ////T)
C MultiSClip
(Csim(() (p.,", ,p(1:1) ()+++]]])) + faim (p,") pp),)()))))
(9)
N
(1) (1) /T)
1=1
exp
Pi
Pi
(2) (2)
/T)
N
exp
P;
Pi
C (2)
Caim (p, (2(): + faim (p, :p();
MultiSClip
N
(10)
exp P Pt /T)
(2)
(2)
MultiSClip
(1.2)
1 (1) MulriCUlpp + 2 Mutussun)
(11)
where (1) and (22) are 2 normalized. The M3Co and MultiSClip losses for M modalities
M ΣD
M
LM300
M3Co
(12)
1=1
7>
M
M
MuuldsCUip
ΣΣ
((J) MultiSClip
(13)
Unimodal Predictions and Fusion: The encoders produce latent representations for each of the M modalities, serving inputs to individual classifiers that generate modality-specifie predictions (mm) These representations are used for modality-specific supervision only during training. The unimodal prediction task involves minimizing the cross-entropy loss LCE between these predictions and the corresponding ground truth labels (y), for each modality. The unimodal cross-entropy loss is:
LCE-UMI Σ
LcEly, (1m))
(14)
m=1
We merge the unimodal latent representations by concatenating them and pass the combined representation to the output classifier. These predictions serve as the final oulputs y used during inference. The multimodal prediction process aims to minimize the cross-entropy loss between &y and the corresponding labels. The multimodal cross-cntropy loss is:
LCE-Multi LcB(y.81)
(15)
5

---

Published in Transactions on Machine Learning Research (05/2025)
Combined Learning Objective: Our overall loss objective utilizes schedule to combine M3Co and MultiSClip loss functions weighted by a hyperparamater 3, along with the unimodal and multimodal cross-entropy losses. We use M3Co for the first one-third (Liu et al., 2023) part of training, and then transition to MultiSClip as over-training with a Mixuppbased loss can potentially harm gencralization. The end-to-end loss is defined as:
TTtal B LMsCo| | MultiSClip + LCE-Uni + CCE-Mulu
(16)
3 Experiments
Datasets. We evaluate our approach on four diverse publicly available multimodal classification datasets: N24News (Wang et al., 2022), Food-101 (Wang et al., 2015), ROSMAP (Wang et al., 2021a), and BRCA (Wang et al., 2021a). N24News and Food-101 are both bimodal image-text classification datasets. Food-101 is a food classification dataset, where each sample is linked with a recipe description gathered from web pages and an associated image. N24News a news classification dataset consisting of four text types (Abstract, Caption, Heading, and Body) along with the corresponding images. Following other works (Zou et al., 2023), we use the first three text types for our experiments. ROSMAP and BRCA are publicly available multimodal medical datasets, each containing three modalities: DNA methylation, miRNA expression, and mRNA expression. ROSMAP is an Alzcihmer's diagnosis datasct, while BRCA is used for breast invasive carcinoma PAM50 sublype classification. Appendix A.3 provides information about the train-val-test splits.
Evaluation Metrics. The evaluation metric used for N24News and Food- 101 is classification accuracy (ACC). For BRCA, we report accuracy (ACC), macro-averagod F1 score (MF1), and weighted F1 score (WF1). For ROSMAP, we use accuracy (ACC), area under the ROC curve (AUC), and F1 score (F1) as the evaluation metrics.
Implementation Details. We use ViT (pre-trained on the ImageNet-21k dataset) (Dosovitskiy et al., 2020) as the image encoder for N24News and Food-101. For N24News, the lext encoder is a pret.rinee BERT/ RRBERRo (Devlin et al., 2018; Zhuang et al., 2021), while we use a pretrained BERT as the text encoder for Food-101. The classifiers for the above two datasets are three layer MLPs with ReLU activations. For ROSMAP and BRCA, which are small datasets, we use two layer MLPs as feature encoders for each modality, and two layer MLPs with ReLU activations as classifiers. The hyperparameter settings and all other details are given in Appendix A.1.
Hyperparameters Tuning, To determine the optimal hyperparameters, we conducted experiments by varying a (the Mixup coefficient, where À is sampled from a Beta distribution, Beta(a, at), 3 (the contrastive loss weighting factor), and the M3Co Ratio (the point at which the loss transitions from M3Co to MultiSClip) on the ROSMAP dalasel. As inferred from Tables 8, 9, and 10 in Appendix A.2, the optimal values were found to be a- = 0.15, B 0.1, and M3Co Ratio 0.33, aligning with prior findings (Liu al., 2023; Zhang et al., 2017) on Mixup augmentation in unimodal settings. Deviations from these values resulted in performance degradation.
Baselines. We compare our method with various multimodal classification approaches (Van De Wiel et al., 2016; Wang et al., 2021a; Han et al., 2020; Abavisani et al., 2020; Han et al., 2022; Zou et al., 2023; Liang et al., 2022b; Kiela et al., 2019; 2018; Wang et al., 2022; Vielzeuf et al., 2018; Arevalo et al., 2017; Li et al., 2019; Huang et al., 2020; Kim et al., 2021; Narayana et al., 2019; Liu et al., 2021; Hong et al., 2020; Huang et al., 2021; Singh et al., 2019; Wang et al., 2024). Some methods (Kiela et al., 2019; Vielzeuf et al., 2018; Arevalo et al., 2017) focus on integrating global features from individual modality-ppecific backbones to enhance classification. Others (Li et al., 2019; Kim et al., 2021; Huang et al., 2020; Narayana et al., 2019) use sophisticated pre-trained architectures finc-tuned for specific tasks. UniS-MMC (Zou et al., 2023), the previous stale-of-the-art on Food-101 and N24News, uses contrastive learning to align features across modalities with supervision from unimodal predictions. Similarly, Dynamics (Han et al., 2022), the previous state-of-the-an on ROSMAP and BRCA, applics a dynamic multimodal classification stratogy. On Food-101 and N24News, we compare against baseline unimodal networks (ViT and BERT /RoBERTa) and our UniConcat bascline, where pre-trained image and text encoders are fine-tuned independently, and the unimodal representations
6

---

Published in Transactions on Machine Learning Research (05/2025)
are simply concatenated for classification. These typical baselines used in multimodal classification tasks.
Detailed baseline descriptions are discussed in Appendix A.8.
4 Results
4.1
Comparison with Baselines
The results are reported as the average and standard deviation over three runs on Food-101/N24News and five on ROSMAP/ /BRCAA The best score is highlighted in bold, while the second-best score is underlined. The classification accuracy on N24News and Food-101 are displayed in Table and respectively, In the result tables, ALI denotes alignment (indicating if the method employs contrastive component), while AGG specifies whether aggregation is early (combining unimodal feature) or late fusion (combining unimodal decisions).
The experimental results from Table 1, 2, 3, reveal the following findings: (i) M3CoL consistently outperforms all SOTA methods across all text sources on N24News when using the same encoders, beats SOTA on all evaluation metrics ROSMAP and BRCA, and also achieves competitive results Food-101; (ii) contrastive-based methods with any form of alignment demonstrate superior performance compared to other multimodal methods; (iii) our proposed M3Col. method, which employs contrastive-based approach with shared alignment, improves over the traditional contrastive santrattvve-baed models and the latest SOTA multimodal methods; (iv) our method augments training samples by creating convex mixtures of raw data points, leading to better and more stable performance through improved contrastive represenlations, as supported by i-mix (Lee et al., 2020). The marginal improvement in N24News (49K samples) and the slight underperformance in Food-101 (60K samples) suggest that the benefits of M3CoL are more pronounced in low-sample regimes, where data augmentation and contrastive learning play critical role. This aligns with the nature of the datasets, as smaller datasets often benefit more from structured noise and shared relation learning. We visualize the unimodal and combined representation distribution of our proposed method using UMAP plots in Figure in Appendix A.6.
Fusion
Backbone
ACC
Method
AGG
ALI
Image
Text
Headline
Caption
Abstract
Image-only
VIT
54.1
(no tert source used)
Text-only
BERT
72.1
72.7
78.3
UniConcat
Early
VIT
BERT
78.6
76.8
80.8
UnIS-MMC
Early
VIT
BERT
80.3
77.5
83.2
M3CoL (Ours)
Early
VIT
BERT
80.840.05
78.0=0.03
83.8+0.06
Text-only
RoBERTa
71.8
72.9
79.7
UniConcat
Early
X
VIT
RoBERTa
78.9
77.9
83.5
N24News
Early
VIT
RoBERTa
79.41
77.45
83.33
UnIS-MMC
Early
VIT
RoBERTa
80.3
78.1
84.2
M3CoL (Ours)
Early
VIT
RoBERTa
80.9
79.2
84.7 10.03
Table 1: Accuracy (ACC) on N24News on three text sources. AGG denotes carly modality fusion, ALI indicates presence presence/absence /absence of aligument. Our method consistently outperforms SOTA across all text sources and backbone combinations. Baseline details are provided in Appendix A.8.
4.2 Analysis of Our Method
Effect of Vanilla Mixup. Mixup involves two main components: the random convex combination of raw inputs and the corresponding convex combination of onc-hot label encodings. To assess the performance of our M3CoL method in comparison to this Mixup strategy, we conduct experiments on Food-101 and N24News (text source: abstract). We remove the contrastive loss from framework (Eq. 16) while keeping
7

---

Published in Transactions on Machine Learning Research (05/2025)
Fusion
ROSMAP
BRCA
Method
AGG ALI
ACC
F1 1
AUC T
ACC
WF1 1
MF1
GRidge
Early
X
76.0
76.9
84.1
74.5
72.6
65.6
BPLSDA
Early
74.2
75.5
83.0
64.2
53.4
36.9
BSPLSDA
Early
75.3
76.4
83.8
63.9
52.2
35.1
MOGONET
Late
X
81.5
82.1
87.4
82.9
82.5
77.4
TMC
Late
X
82.5
82.3
88.5
84.2
84.4
80.6
CF
Early
X
78.4
78.8
88.0
81.5
81.5
77.1
GMU
Early
X
77.6
78.4
86.9
80.0
79.8
74.6
MOSEGCN
Early
83.0
82.7
83.2
86.7
86.8
81.1
DYNAMICS
Early
85.7
86.3
91.1
87.7
88.0
84.5
M3CoL (Ours)
Early 88.7 88.7
88.5
92.640.59
88.440.57
88.
89.0
86,
86.2+0.54
Table 2:
Comparison of Accuracy (ACC), Area Under the Curve (AUC), F1 score (F1) on ROSMAP, and Accuracy (ACC), Weighted F1 score (WF1), and Micro F1 score (MF1) on BRCA datasets. AGG denotes early/ late modality fusion, ALI indicates presence/absee absence of aligument. Our method significantly outperforms SOTA across all metries. Baseline details are provided in Appendix A.8.
Fusion
Backbone
Method
ACC 1
AGG
ALI
Image
Text
Image-only
ViT
73.1
Text-only
BERT
86.8
UniConcat
Early
x
VIT
BERT
93.7
MCCE
Early
DenseNet
BERT
91.3
CentralNet
Early
LeNet5
LeNet5
91.5
GMU
Early
RNN
VGG
90.6
ELS-MMC
Early
ResNct-152
BOW features
90.8
MMBT
Early
ResNet-152
BERT
91.7
HUSE
Early
Graph-RISE
BERT
92.3
VisualBERT
x
FasterkCNN+BEER
BERT
92.3
PixcIBERT
Early
ResNet
BERT
92.6
ViLT
Early
ViT
BERT
92.9
CMA-CLIP
Early
ViT
BERT
93.1
ME
Early
Net
BERT
94.7
UMC
Early
ViT
BERT
94.7
M3CoL (Ours)
Early
ViT
BERT
94.3+0.01
Table 3: Accuracy (ACC) comparison on Food-101. AGG denotes carly /late modality fusion, ALI indicates presence/absence of aligument. Baseline details are provided in Appendix A.8.
the rest of the modules unchanged. Table 4 shows that the Mixup technique underperforms relative to our proposed M3CoL approach (Testing accuracy curves shown in Figure 6a). The observed accuracy gap can be attributed to excessive noise introduced by label mixing, and the lack of contrastive approach with alignment component. This indicates that the vanilla Mixup strategy introduces additional noise which impairs the model's ability to learn effective representations, while our M3CoL framework benefits from the structured contrastive approach.
Effect of Loss & Unimodality Supervision. To assess the necessity of each component the framework, investigate several design choices: (i) the framework's performance without the supervision of unimodal modules during training, and (ii) the performance differences between using only MultiSClip and only M3Co

---

Published in Transactions on Machine Learning Research (05/2025)
ACC
Method
ROSMAP
BRCA
Food-101
N24News
Mixup
84.13
84.52
93.14 02
81.57
M3CoL (No Unimodal Supervision)
85.14
86.93
94.12 94.12 2o.02
84.261..11
M3CoL (only MultiSClip)
86.84
87.38
94.23
84.06+u.18
M3CoL (only M3Co)
87.42
87.74
94.24
44.57
M3CoL (0.33 M3Co + 0.67 MultiSClip)
88.67 20
88.38
94.27 to
84.724o.0
Table Accuracy (ACC) on ROSMAP, BRCA, N24News, and Food-101 datasets under different settings of our method. For N24News, source: abstract and encoder: RoBERTa.
loss during end-to-end training. The M3CoL (No Unimodal Supervision) result indicates that excluding the unimodal prediction module results in decline in performance as shown in Table 4 and Figure 6a, highlighting its importance as it allows the model to compensate for the weaknesses of one modality with the strengths of another. Additionally, the M3Co loss (only M3Co) outperforms the MultiSClip loss (only MultiSClip) by learning more robust representations through Mixup-based techniques, which prevent trivial discrimination of positive pairs. Furthermore, using an individual contrastive alignment approach (only M3Co) throughout the entire training process without transitioning to the MultiSClip loss results in suboptimal outcomes. This can be attributed to the risk of over-training with Mixup-basod loss, which may negatively impact generalization. This demonstrates the necessity of the transition of the contrastive loss during training (0.33 M3Co 0.67 MultiSClip). Figure 6b displays the accuracy plots on the N24News dataset, for these losses. In addition to the ACC scores presented in Table 4, we also report the performance of other metries, where available, on the ROSMAP and BRCA datasets under varions settings of our method, as shown in Table 5.
ROSMAP
BRCA
Method
F1 t
AUC
WF1 t
MF1 1
Mixup
84.45.to.4
88.73+t.5a
84.66
82.88
M3CoL (No Unimodal Supervision)
86.27 86.37
90.20
86.92
85.08
M3CoL (only MultiSClip)
86.76 76
90. 90.75+0.40
87.41
85.48+t..0
M3CoL (only M3Co)
87.54 to.54
91.41 to.sa
88.06
85.82
M3CoL (0.33 M3Co + 0.67 MultiSClip)
92.62 88.51+0.91
89.02
86.20
Table 5: Comparison of F1 score (F1), Area Under the Curve (AUC) on ROSMAP, and Weighted F1 score (WF1), Micro Fl score (MF1) on BRCA, under different settings of our method.
Visualization of Attention Heatmaps. The attention heatmaps generated using the embeddings from our trained M3CoL model in Figure 3 and 4 highlight image regions most relevant to the input word. We generate text embeddings for class label words and corresponding image patch embeddings, computing attention scores as their dot product. This visualization aids understanding the model's focus, decision-making process, and association between class labels and specific image regions. Importantly, also indicates the correctness of the learned multimodal representations, revealing the model's ability to learn shared relations amongst. different modalities, and ground visual concepts to semantically meaningful regions.
Testing on Random Data and Single-Corrupt Modalities. To showcase the benefits of our framework over traditional contrastive methods, we evaluate the impact of incorporating Mixup-based contrastive loss (M3Co) during training, highlighting its improvements over standard approaches. It is well-established that deep networks tend to exhibit overconfidence, particularly when making predictions on random or adversarial inputs (Hendrycks & Gimpel, 2016). Previous research has demonstrated that Mixup can mitigate this issue, and goal is to validate its effectiveness in this context (Thulasidasan et al., 2019). We evaluate
9

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Image (b) Ice cream
(c) Cream
(d) Ice
(e) Image
(1) Falafel
(g) Salad
(h) Rice
Figure 3: Text-guided visual grounding with varying input prompts. (a, e) Original images. (b-d) Attention heatmaps for "ice cream" class. (f-h) Heatmaps for "falafel" class. Ice cream example: (b) "Ice cream" Concentrated focus on ice cream, (c) "Cream": Maintained but diffused focus, (d) "Ice": Dispersed attention. Falafel example: (f) "Falafel"; Localized focus on falafel, (g) "Salad"; Attention shift salad component, (h)
"Rice": Minimal attention (absent in image). Warmer colors indicate higher attention scores.
(a) Risotto
(b) Mixup
(c) No Unim
(d) MultiSClip
(e) M3Co
(f) M3CoL
Figure 4: Text-guided visual grounding with ablated model variations. (a) Original image. (b-f) Attention heatmaps generated using text embedding (class name: "Risotto") and patch embeddings for different variations of the model. Our proposed M3CoL model (f) demonstrates superior attention localization compared to ablated versions (b-e), corroborating the quantitative results presented in Table 4. Warmer colors indicate higher attention scores. (Here, No Unim: No Unimodal Supervision)
the confidence scores produced using M3CoL (0.33 M3Co + 0.67 MultiSClip) loss in comparison to only MultiSClip loss when predicting on random noise images and text encoder outputs. Our results show that the model trained with M3CoL exhibits lower confidence its predictions when both modalities are replaced with random inputs. This demonstrates that incorporating M3CoL enhances the reliability of predictions, especially in the presence of corrupted random inpuls.
To evaluate the robustness of our approach, we conduct experiments where one input modality corrupted with random noise. Table 6 comparcs the performance of M3CoL (0.33 M3Co + 0.67 MultiSClip) against only MuluiSClip under these conditions. Our M3CoL met demonstrates superior robustness to modality corruption, consistently outperforming MultiSClip. For image corruption, we substituted the original images
MICs MurtSshhppp
SortCilo
0.0
0.2
0.4
0.6
0.8
1.0
Confidence Figure 5: N24News Confidence scores when tested on random inputs.
with random noise sampled from a Gaussian distribution, parameterized to match the mean and variance of the training set. Similarly, for text corruption, we replaced the original text embeddings with random outputs from the text encoder, again following Gaussian distribution with statistics matching the training
10

---

Published in Transactions on Machine Learning Research (05/2025)
Modality Corrupted
Method
None
Image
Text
MutliSClip
84.06
76.06
46.91
M3CoL
84.72
77.24
47.94
Table 6: N24News- Accuracy when tested on data with modality corruption.
data. For both the above experiments, we use the N24News dataset, with the abstract as the text source and a RoBERTa-basee based text encoder.
Error Analysis. To evaluate the efficacy of our multimodal approach in integrating and leveraging image and text features, we performed a comprehensive error analysis, comparing it with image-only (ViT) and text-only (RoBERTa) models using the N24News dataset (see Table 12 in Appendix A.5). The analysis reveals that our method excels when both modalities are correctly classified (42.71:0.03 correct-to-incoree ratio). This shows that our model can learn valuable insights from the fusion of image and text features, which may not be discovered when processing them separately. In cases where only one modality is correctly classified, our model effectively leverages the accurate modality ratio. As shown in Table 12, our method excels not only when both modalities are correctly classified but also demonstrates strong performance even when both are misclassified, indicating effective feature fusion. In addition, it successfully leverages information when only one modality is classified correctly. This analysis demonstrates our method's robustness and highlights its superiority over unimodal approaches.
correct- to incorrect
5 Related Work
Contrastive Learning. Contrastive learning has driven significant progress in unimodal and multimodal representation learning by distinguishing between similar (positive) and dissimilar (negative) pairs. In multimodal contexts, cross-modal contrastive techniques align representations from different modalities (Radford et al., 2021; Jia et al., 2021; Kamath et al., 2021), with approaches like CrossCL. (Zolfaghari et al., 2021) and GMC (Poklukar et al., 2022) focusing on global and modality-specifie representations. Contrastive learning approaches for paired image- text data, such as CLIP (Radford et al., 2021), ALIGN (Jia al., 2021), and BASIC (Pham et al., 2023), have demonstrated remarkable success across diverse vision-language tasks. Subscquent works have aimed to enhance the efficacy and data efficiency of CLIP training, incorporating self-supervised techniques (SLIP (Mu el al., 2022), DeCLIP (Li et al., 2021)) and fine-grained alignment. (FILIP (Li et al., 2023)). The CLIP framework relies on data augmentations to prevent overfitting and the lcarning of ineffective shortcuts (Chen et al., 2020; Robinson et al., 2021), a common practice in contrastive learning.
Unimodal and Multimodal Data Augmentation. Data augmentation has been integral to the success of deep learning, especially for small training sets. In computer vision, techniques have evolved from basic transformations to advanced methods like Cutout (DeVries & Taylor, 2017), Mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), and automated approaches (Cubuk et al., 2019; 2020). NL. augmentation includes paraphrasing, token replacement (Zhang et al., 2015; Jiao et al., 2019), and noise injection (Yan et al., 2019). Multimodal data augmentation, primarily focused on vision-text tasks, has seen limited exploration, with approaches including back-translation for visual question answering (Tang et al., 2020), text generation from images (Wang et al., 2021b), and external knowledge qucrying for cross-modal retricval (Gur et al., 2021). MixGen (Hao et al., 2023) generates new image-text pairs through image interpolation and text concatenation. In contrast, our proposed angmentation technique focusing the early training phase is fully automatic, applicable to arbitrary modalities, and designed to leverage inherent shared relations in multimodal data.
Relation to Mixup. Mixup (Zhang et al., 2017), a pivotal regularization strategy, enhances model robustness and generalization by generating samples through convex combinations of existing data points. Originally introduced for computer vision, has been adapted to NLP by applying the technique to text emboddings (Guo et al., 2019). Our proposed augmentation differs from Mixup in several key aspects: it is designed for multimodal data, takes inputs from different modalities, and does not rely one-hot label encodings. By
11

---

Published in Transactions on Machine Learning Research (05/2025)
extending the Mixup paradigm to complex, multimodal scenarios and focusing on the early training phase, our method broadens its applicability while leveraging inherent shared relations in multimodal data.
Relation to Previous Work. Our method builds upon and extends several key ideas from recent advancements in multimodal learning. M3CoL shares the use of contrastive learning for multimodal aligument but goes beyond by introducing Mixup-based augmentation to capture shared relations across modalities, which UnIS-MMC (Zou et al., 2023) does not address. While Dynamics (Han et al., 2020) focuses on dynamic fusion for trustworthy classification, M3CoL emphasizes learning robust representations through Mixup-based contrastive loss, enhancing generalization and interpretability. SoftCLIP (Gao et al., 2024) introduces softer cross-modal alignment, whereas M3CoL utilizes Mixup to create structured noise and align mixed samples, capturing nuanced shared relations. MixGen (Hao et al., 2023) explores multimodal data augmentation through image interpolation and text concatenation. M3CoL differs by focusing on Mixup-based contrastive learning to align mixed samples, which is more targeted at capturing shared relations rather than generating new data points. M3CoL advances multimodal learning by addressing the limitations of traditional contrastive methods and introducing a novel approach to capluring shared relations.
6 Discussion and Limitations
Discussion and Conclusions. We propose M3Co, novel contrastive baated alignment method that captures shared relations beyond explicit pairwise associations by aligning mixed samples from one modality with corresponding samples from others. The M3Co loss, combined with an architecture leveraging unimodal and fusion modules, enables continuous updating of representations necessary for accurate predictions and deeper integration of modalities. This method generalizes across diverse domains, including image-text high- dimensional multi-omies, and data with more than two modalities. Experiments on four public multimodal classification datasets demonstrate the effectiveness of our approach in learning robust representations that. surpass traditional multimodal alignment techniques.
The computation cost in loss calculation in M3CoL scales as O(2M²N3), which is only factor of 2 larger than conventional contrastive learning (3) This is because, in a batch of size N, each sample is mixed pairwise with one other sample (e.g., image 1 with image N, image 2 with image N 1, etc.), and the resulting mixture creates two positive relations with corresponding samples from other modalities. While the computational complexity scales quadratically with both the number of modalities (M²) and the batch size (N²), the overall overhead is not significantly higher than conventional contrastive approaches. We did not observe significant differences in training times compared to conventional contrastive learning. Additionally, the Mixup-based augmentation is applied only during the first one-third of training, not during evaluation, which helps manage computational overhead.
Analysis of Learned Representations. Aligning representations across modalitics presents significant challenges due to the complex, often non-bijective relationships in real-world multimodal data. These relationships can involve many-to-many mappings or even lack clear associations, as exemplified by linguistic ambiguities and synonymy in vision-language tasks (Liang et al., 2022a). Our approach incorporates Mixup- based contrastive learning, introducing controlled noise that mirrors the inherent variability in multimodal data, thus enhancing robustness and generalizability. The visualization of attention heatmaps (Figure 3 and Figure 4) shows that M3CoL learns semantically meaningful multimodal representations by grounding visual concepts to relevant image regions. When tested with corrupted or random inputs, M3CoL demonstrates lower confidence scores (Table 6), reducing overconfidence and enhancing reliability.
Limitations and Future Work. M3CoL demonstrates promising results, yet faces optimization challenges due to the inherent limitations of multimodal frameworks, particularly extended training times on large-scale datasets like Food-101 and involves multiple hyperparameters, making it more complex to identify the optimal configuration. The method's modality-agnostic nature and effective use of mixup augmentation suggest its polential adaptability to various multinodal tasks, especially where data augnentation and learning real-world inter-modal relationships are crucial. Future work should focus on investigating domain adaptation strategies, validating M3CoL's 's utility on downstream tasks such as visual question answering and information retrieval, and enhancing interpretability through explainable AI techniques. These advancements, coupled with comprehensive hyperparameter tuning, will likely broaden M3CoL's impact in multinodal research.
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Mahdi Abavisani, Liwei Wu, Shengli Hu, Joel Tetreault, and Alejandro Jaimes. Multimodal categorization of crisis events in social media. In Proceedings of the IEEE/CV /CF Conference on Computer Vision and Pottern Recognition, pp. 14679-14689) 2020.
John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. arxiv preprinl Xivi 702.0022000 01992, 2017.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Phillppee Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, (1(2):433-445) 2018.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in neural information processing systems, 13, 2000.
Ting Chen, Simon Korublith, Mohammad Norouzi, and GeofTrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVE /CV conference compuler vision and pottern recognition workshops, pp. 702-703, 2020.
Jacob Devlin, Ming-We Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl arXiv:1810.04580 2018.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1 100xv170..... 04550, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint ar arxiv 11929, 2020.
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chumhua Shen. Pyramidelip: Hicrarchical feature alignment for vision-language model pretraining. Advances in neural information processing systems, 35:35959 35970, 2022.
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softelip: Softer cross-modal alignment makes clip stronger. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 1860-1868, 2024.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatio. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. Advances in Newral Information Processing Systems, 32, 2019.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classification: An empirical study. arXiv preprint arXi::1905.088) 2019.
Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, and Austin Reiter. Cross-modal retrieval augmentation for multi-modal classification. arXiv preprint wxXiv2214..01100 2021.
Zongbo Han, Changqing Zhang, Huazhu Fu, and Jocy Tianyi Zhou Trustod multi-view classification. In International Conference on Learning Representations, 2020.
Zongbo Han, Fan Yang. Junzhou Huang, Changqing Zhang, and Jianhua Yao. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classifiation. In Proceedings of the IEEE/CVF conference on compuler vision and paltern recognition, pp. 20707, 2022.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 379 399, 2023.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXie preprint arXiv:1110002133 2016.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. orXiv preprint arXiv:191.002881, 2019.
Danfeng Hong, Lianru Gao, Naoto Yokoya, Jing Yao, Jocelyn Chanussot, Qian Du, and Bing Zhang, More diverse means better: Multimodal deep learning meets remote-sensing imagery classification. IEEE Transactions on Geoscience and Remole Sensing, (9(5):4340-43355 2020.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Newral Information Processing Systems, 34:10944-10956, 2021.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.008/ 2020.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint (tti::999.1035.44 2019.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Aidetr-modulated detection for end-to-end molti- understanding. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pp. 1780-1790, 2021.
Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. Efficient large-scale multi-modal classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXie: 190.00000002000 2019.
Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual
representation. arXiv preprint ar wxXii:20100063 2020.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languuge transformer without convolution or
region supervision. In International conference on machine learning, pp. 5583 5594. PMLR, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.660 6980, 2014.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostin strategy for contrastive reprosentation learning. arXiv preprint arXi::2010.08891 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint Xrx::11908. 03557,, 2019.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive Ainuugg-imaae pre-training paradigm. arXiv preprinl xrXiv:2110.05:3 052000 2021.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23390-234 2023.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprinl arXi::2209.08 03430, 2022a.
Tao Liang, Guosheng Lin, Mingyang Wan, Tianrui Li, Guojun Ma, and Fengmao Lv. Expanding large pre-trained unimodal models with multimodal information injection for image-text multimodal classification. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognilion, pp. 15492- 15501, 2022b.
Huidong Liu, Shaoyuan Xu, Jinmiao Fu, Yang Liu, Ning Xie, Chien-Chih Wang, Bryan Wang, and Yi Sun. Cma-clip: Crossimodality attention clip for image-lext classification. Xiv preprinl ar urXiv:2112.032 02.6262, 2021.
Zixuan Liu, Ziqiao Wang. Hongyu Guo, and Yongyi Mao. Over-training with mixup may hurt generalization. arXiv preprint arXit:2303.01433 0177,, 2023.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-inage pre-training. In European conference on computer vision, pp- 529-544 Springer, 2022.
Pradyumna Narayana, Aniket Podnekar, Abishek Krishnamoorthy, Kazoo Sone, and Sugato Basu. Huse: Hierarchical universal semantic embeddings. arXiv preprint arXiv:1911.05978 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint ar Xiv: 03748,748 2018.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer lcarning. Neurocompuling, 555:1 16658, 2023.
Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S Melo, Ana Paiva, and Danica Kragic. Geometric multimodal contrastive representation learning. In International Conference on Machine Learning, pp. 17782 17800. PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shorteut solutions? Advances in neural information processing systems, 34:4974-4986, 2021.
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing. Un-mix: Rethinking image mixtures for unsupervised visual representation learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 2216 (224,, 2022.
Amrit Singh, Casey P Shannon, Benoit Gautier, Florian Rohart, Michaël Vacher, Scott J Tebbutt, and Kim-Anh Le Cao. Diablo: an integrative approach for identifying key molccular drivers from multi-omics assays. Bioinfoormalies,, 35(17):3055 3062, 2019.
Kihyuk Sohn. Improved deep metric learning with multi-class n- n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Ruixue Tang, Chao Ma, Wei Emma Zhang, Qi Wu, and Xiaokang Yang. Semantic equivalent adversarial data augmentation for visual question answering. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pp. 437-453. Springer, 2020.
Gemini Team, Rohan Anil, Scbastian Borgeaud, Yonghui Wu, Cean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. Xixiv preprint or Xiv:2331 11805, 2023.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Sunil Thulasidasan, Gopinath Chenmupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Aduances in neural information processing systems, 32, 2019.
Mark A Van De Wiel, Tonje G Lien, Wina Verlaat, Wessel van Wieringen, and Saskia M Wilting. Better prediction by use of co-data: adaptive group-regularized ridge regression. Statistics in medicine, 35(3): 368 381, 2016.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Toannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International conference on machine learning, pp. 1438-6447 PMLR, 2019.
Valentin Vielzeuf, Alexis Lochervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: a multilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Jiahui Wang, Nanqing Liao, Xiaofei Du, Qingfeng Chen, and Bizhong Wei. A semi-supervised approach for the integration of multi-omics data based on transformer multi-head self-attention mechanism and graph convolutional networks. BMC genomics, 25(1):86, 2024.
Tongxin Wang, Wei Shao, Zhi Huang, Haixu Tang, Jie Zhang, Zhengming Ding, and Kun Huang. Mogonet integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification. Nature commumications,, 12(1):3445, 2021a.
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1-6. IEFE, 2015.
Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. N24news: A new dataset for multimodal news classification. In Proceedings of the Language Resources and Evaluation Conference, pp. 6768 6775, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthollee org/2022. 1rec-1.729.
Zixu Wang, Yishu Miao, and Lucia Specia. Cross-modee generative augmentation for visual question answering. arXiv preprint arxv:22105.0 04055.0770. 2021b.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 3733 3042, 2018.
Ge Yan, Yu Li, Shu Zhang, and Zhenyu Chen. Data augmentation for deep learning of judgment documents. In Intelligence Science and Big Data Engineering. Big Data and Machine Learning: 9th International Conference, IScIDE 2019, Nanjing, China, Oclober 17 20, 2019, Proceedings, Part II 9, pp. 232 242. Springer, 2019.
Hao Yu, Huanyu Wang, and Jianxin Wu. Mixup without hesitation. In Image and Graphics: 11th International Conference, ICIG 2021, Haikou, China, August 6-8, 2021, Proceedings, Part II 11, pp. 143-154. Springer, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the FRFE/CVE international conference computer vision, pp. 6023 6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz, mixup: Beyond empirical risk minimization. arXiv preprint 1eXiv::710.0944 09412, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character- convolutional networks for text classification. A Aduances in neural information processing systems, 28, 2015.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healtheare Conference, pp. 2 25. PMLR, 2022.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized bert pre-training approach with post-training. In Proceedings of the 20th chinese national conference computational linguistics, pp- 1218 1227, 2021.
Mohammadreza Zolfaghari, Yi Zhu, Peter Gchler, and Thomas Brox. Crossclr: Cross-modal contrastive learning for multi-modal video representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1450-1459, 2021.
Heqing Zou, Meng Shen, Chen Chen, Yuchen Hu, Deepu Rajan, and Eng Siong Chng. Unis-mme: Multimodal classification unimodality-supevvit multimodal contrastive learning. arXiv preprint ar orXiv22305.00 09299, 2023.
17

---

Published in Transactions on Machine Learning Research (05/2025)
A Appendix
A.1
Experimental Details
The models were trained on either an NVIDIA RTX A6000 or an NVIDIA A100-SXM4-80GB GPU. The results are reported as the average and standard deviation over three runs on Food-101 and N24News, and five runs ROSMAP and BRCA. We hyperparameters. The temperature parameter for the M3Co and MultiSClip losses is set to 0.1. The corresponding loss coefficient, / is 0.1 to keep the loss value in the same range as the other losses. We use the Adam optimizer (Kingma & Ba, 2014) for all datascts. For Food-101 and N24News, the lcarning rate scheduler is ReduceL.ROnPlatess
grid search on the validation to search for optimal
with validation accuracy as the monitored metric, Ir factor of 0.2, and Ir patience of 2. For ROSMAP and BRCA, we use the StepLR scheduler with a step size 250. For Food- 101 and N24News, the maximum token length of the text input for the BERT RoBERTa encoders 512. Other hyperparameter details provided in Table 7.
Hyperparameter
N24News
Food-101
ROSMAP
BRCA
Embedding dimension
768
768
1000
768
Classifier dimension
256
256
1000
768
Learning rate
10-1
10-1
5-10-3
10-3
Weight decay
10
10-4
10-3
10-3
Batch size
32
32
Batch gradient
128
128
Dropout (classifier)
0
0
0.5
0.5
Epochs
50
50
500
500
Table 7: Experimental hyperparameter values for our proposed model across all the four datasets.
A.2 Hyperparameters Selection
We illustrate how performance varies with different hyperparameters and describe our selection process through Tables 8, 9, and 10.
ACC 1
F1 1
AUC 1
0.05
84.91
84.11
91.34
0.15
87.74
87.62
92.98
0.4
85.85
85.18
91.55
0.8
83.96
84.11
91.52
Table 8: Performance comparison for different a values. a is used to calculate the Mixup coefficient as Beta(a,
ACC f
FI f
AUC 1
0.05
84.91
85.72
91.77
0.1
87.74
87.62
92.98
0.5
85.85
86.49
92.55
85.85
85.72
92.41
Table Performance comparison for different values. the contrasttve loss scaling factor.
18

---

Published in Transactions on Machine Learning Research (05/2025)
M3Co Ratio
ACC f
F1 f
AUC f
85.85
85.98
91.66
0.2
85.85
85.71
92.41
0.33
87.74
87.62
92.98
0.5
86.79
86.79
92.58
0.8
86.79
86.27
92.58
86.79
86.27
92.58
Table 10: Performance comparison for different M3Co Ratio values, which docides when switch from the M3Co loss to the MultiSClip loss.
A.3 Dataset Information and Splits
The datasets used in our experiments can be downloaded from the following sources: Food-101 from sttps://visith issrr upmc.fr, N24News from httpp:://gthub woom/xxWagg/MOCC//
http://ggttaa...omiiiiiitee com/billyywaa117/224100. and BRCA
and ROSMAP from https://gitte
To ensure a fair comparison with previous works, we adopt the default split method detailed in Table 11. As the Food-101 dataset does not include a validation set, we partition 5,000 samples from the training set to create one, which conistent with other basclines.
Dataset
Modalities
Modality Types
Train
Validation
Test
Classes
Food-101
Image, lext Image, text
60101
5000
21695
101
N24News
2
48988
6123
6124
24
ROSMAP
mRNA, miRNA, DNA
245
106
2
BRCA
3
mRNA, miRNA, DNA
612
263
5
Table 11: Statistics for the four datasets: Food-100 101, N24News, ROSMAP, and BRCA. Note: miRNA stands for microRNA, and mRNA stands for messenger RNA.
A.4 Ablation Studies on the N24News Dataset
The accuracy plots for the N24News dataset (text source: abstract, text encoder: RoBERTa) are used to compare our method and its variants. Our proposed M3CoL approach outperforms the Mixup technique, as shown in Figure 6a. Ablating the unimodal supervision in M3CoT, leads a performance decline, indicating the importance of the unimodal prediction module, as shown in Figure 6a. Furthermore, the M3Co loss achieves better results than the MultiSClip loss. Training solely with either the M3Co loss or the MultiSClip loss alignment approach yields suboptimal performance when compared to their strategic combination, shown in Figure 6b. The quantitative results are given in Table 4.
A.5 Error Analysis Table
We provide an in-depth error analysis in Table 12 the N24News dataset, discussed in Section 4.2.
A.6
UMAP Plots
We generate UMAP plots on embeddings derived from the N24News and Food-101 datasels to visualize the clustering performance our M3CoL model. For each dataset, we randomly select 10 classes and generate the corresponding embeddings from the image encoder, text encoder, and their concatenated multimodal representatio, using our trained M3CoL model. Figure shows that the image embeddings depict less distinct clusters, indicating less effective inter-cluster separation compared to the text encoder embeddings. The
19

---

Published in Transactions on Machine Learning Research (05/2025)
Test Accuracy N24News
Test Accuracy N24News
84
Accuracy (%)
Accuracy (%)
84.0
83.5
81
Mixup
83.0
Only MultiSClip
Unimodal
Only
0.33 M3Co 0.67 MultiSClip
0.33 M3Co 0.67 MultiSCllp
80
82.5
5000
15000
25000
35000
45000
5000
15000
25000
35000
45000
Step
Step
(a) Comparison of M3CoL and its variants using Mixup and No Unimodal Supervision.
(b) Comparison M3CoL and its variants using only M3Co and only MultiSoft Clip loss.
Figure 6: Test accuracy plots showing comparison of M3Col and its variants on the N24News dataset (text source: abstract, text encoder: RoBERTa).
Unimodal Prediction
Multimodal Prediction %
Text
Image
Correct
Incorrect
True
True
42.71
0.03
True
False
27.77
1.29
False
True
8.11
3.25
False
False
2.31
14.53
Table 12: Error analysis N24News with text encoder RoBERTa and text source 'headline'. True and False denote the correctness of the unimodal predictions. Multimodal Prediction % shows the resulting test set. ratio of the final predictions.
concatenated embeddings, however, result in the best-defined clusters, suggesting that the final multimodal representations preserve and potentially enhance dlasnniisig features. These observations align with our quantitative results presented Table and 3.
A.7
Additional Visualization Attention Heatmaps
Following Section 4.2, generate heatmaps using class embeddings and patch embeddings for more examples in Food-101. These are depicted in Figure 8.
A.8 Baseline Details
The baselines used our comparsions are described in details as follows:
GRidge (Van De Wiel et al., 2016) dynamically incorporates multimodal data to adjust regularization penalties, improving predictive accuracy in genomic classification scenarios.
BPLSDA (Block partial least. squares discriminant analysis) (Singh et al., 2019) analyzes multimodal data in latent space and BSPLSDA (Block sparse partial least squares discriminant analysis) (Singh et al., 2019) adds sparsity constraints to BPLSDA to extract relevant features.
MOGONET (Wang al., 2021a) integrates GCNs with View Correlation Discovery Network (VCDN) to process multi-omics data. The initial predictions from each omics-specific GCN are
20

---

Published in Transactions on Machine Learning Research (05/2025)
N24NeWs
N24News
N24News
20
UNeP direroioon
URKe dimerssonn
UMMP dimersioon
10
-20
-15
-10
10
15
20
25
-10
10
-20
-10
20
30
UMAP dimension
(a) Image Embeddings
(b) Text Embeddings
(c) Concatenated Embeddings
Food-101
Food-101
Food-101
30
20
UNAP dimersion
UNAP dimersionz
UNAP Pimmerooo
10
10
20
-10
-10
-20
-20
-10
20
30
-20
-20
-10
20
dimension
(d) Image Embeddings
Text Embeddings
(f) Concatenated Embeddings
Figure 7: UMAP plots of embeddings from the N24News (source: abstract and encoder: RoBERTa) and Food-101 datasets. We generate UMAP plots for the representations generated by the image encoder, text encoder, and their concatenated multimodal representations, using our trained M3CoL model. Concatenated embeddings exhibit superior clustering, while text embeddings outperform image embeddings. Consistent patterns across datasets demonstrate M3CoL's offectiveness in fusing multimodal information and enhancing semantic representations.
consolidated using the VCDN, which identifics and leverages cross-omics label correlations to improve prediction accuracy.
CF (Concatenation of Final Multimodal Representations) (Hong et al., 2020; Huang et al., 2021) creates representations by combining late stage representations multiple modalities.
TMC (Han et al., 2020) enhances decision-making by dynamically integrating multiple views based on confidence levels, for robust and reliable fusion.
MCCE (Abavisani et al., 2020) uses DenseNet and BERT for feature extraction then applying stochastic transitions between multi-modal embeddings during training to enhance generalization and to handle sparse data effectively.
Dynamics (Han et al., 2022) presents an approach for trustworthy multi-modal classification, specifically designed for high-stakes environments like medical diagnosis. The model dynamically assesses both feature-level and modalityy-evel informativeness, using a sparse gating mechanism filter and integrate the most relevant features and modalities per sample.
UniS-MMC (Zou et al., 2023) uses a contrastive learning approach that relies on making unimodal predictions, evaluating the agreement discrepancy between these predictions and the ground truth, and using this insight to align feature vectors across various modalities through contrastive loss.
ME (Liang et al., 2022b) leverages cross-modal information by transforming features between modalities. It achieves this by integrating Multimodal Information Injection Plug-in (MI2P) with pre-trained models, enabling them to process image-text, pairs without structural modifications.
21

---

Published in Transactions on Machine Learning Research (05/2025)
(a) Risotto
(b) Mac and cheese
èc)
Beignet
(d) Cannoli
(e) French toast
(f) Ice cream
(g) Tuna tartare
(h) Falafel
Figure 8: Attention heatmaps demonstrating text-guided visual grounding for samples in the Food-101
dataset. Warmer colors indicate higher attention scores.
MMBT (Kiela et al., 2019) leverages the strengths of pre-trained text and image encoders, effectively combining them within a BERT-like framework by mapping image embeddings into the textual token space.
ELS-MMC (Kiela al., 2018) investigates various multimodal fusion techniques for integrating discrete (text) and continuous (visual) modalities to enhance classification tasks in resource-efficient manner.
N24News (Wang et al., 2022) presents a novel dataset from The New York Times with text and image data across 24 catogorics. It utilizes a multitask multimodal strategy, employing ViT for image processing and RoBERTa for text analysis, with features concatenated for final classification.
CentralNet (Vielzeuf al., 2018) uses scparate convolutional networks for each modality, linked via a central network that generates a unified feature representation and also applics multi-task learning to refine and regulate these features.
22

---

Published in Transactions on Machine Learning Research (05/2025)
GMU (Arevalo et al., 2017) employs multiplicative gates that dynamically adjust the influence of each modality its activation thereby deriving a sophisticated intermediate representation ation tailored for specific applications.
VisualBERT (Li et al., 2019) employs a series of transformer layers to align textual elements and corresponding image regions through self-attention mechanisms.
PixelBert (Huang et al., 2020) directly aligns image pixels with textual descriptions using a deep multi-modal transformer and establishes a direct semantic connection at the pixel and text level.
ViLT (Kim et al., 2021) implements a BERT-like transformer model that processes visual data in a convolution-Iree manner, similar to textual data, thereby simplifying inpul feature extraction and reducing computational demands.
HUSE (Narayana et al., 2019) constructs a shared latent space that aligns image and text embeddings based on their semantic similarity, enhancing cross-modal representation.
CMA-CLIP (Liu et al., 2021) enhances CLIP (Radford et al., 2021) by integrating two cross cross-modee modality attention mochanisms: sequence-wer wisc and modality-wise attention. These attention modules refine the relationships between image patches and text lokens, allowing the model to focus on relevant modalities for specific tasks.
MOSEGCN (Wang et al., 2024) utilizes transformer multi-head self-attention and Similarity Network Fusion (SNF) to learn correlations within and among different omics. This information is then fed into a self-ensembling Graph Convolutional Network (SEGCN) for semi-supervised training and testing.
A.9 Use of Generative Al Models
In this work, we use the following generative AT model:
Gemini 1.0 Pro (Team et al., 2023) to generate food item images and captions as displayed in Figure
1. which serve as sample representations from the Food-101 dataset.
23


Reviews:
[{"review_id": "oECC0MT0TT", "reviewer": "Reviewer_KokD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper lists a large number of experiments, on different modalities and domains, including comparison with baselines, ablation studies and qualitative experiments. \n\n- The motivations are clearly articulated: modeling intra-modal relationships and addressing the bijectivity assumption paves the way for capturing more general structural relations—an area that, to the best of my knowledge, remains underexplored in multimodal contrastive learning.\n\n- The paper is well written and mostly easy to follow.\n\n- The approach reaches good performance on multimodal benchmarks, obtaining SOTA in a subset of the cases.\n\n###  Weaknesses\n\n- The major weakness of the paper stays in the lack of a good methodological contextualization with previous work. It seem to me that the method  is in combining previous cited  work such as SoftClip ( Gao et al., 2024) to relax the bijective contrastive learning assumption, previous multimodal works using mixup (e.g. [1])   and  methods that leverage single modality classification task to improve multimodal learning ((Zou et al., 2023), (Han et al 2022)).  Differences with respect to the these work should be discussed more in depth: is the proposed approach based on combining the previous method?  If there are fundamental differences then they should be highlighted. \n\n- The motivation behind the proposed loss could be strengthen by adding results: in particular: \n    - A discussion or experimental analysis exploring whether, in some domains or tasks, only a subset of the loss terms (e.g., using a cross-entropy term instead of a contrastive term) might suffice would further strengthen the motivation behind the proposed loss. For example, Table 1 shows that the improvement over UniS-MMC is only marginal on News24 and slightly underperforms on Food 101, while Table 2 demonstrates that M3CoL performs significantly better than DYNAMICS. Does this say something about the nature of the datasets and the quality of labels available? \n   - It would be nice to have some preliminary experimental evidence, or at least some discussion of the features space learned by M3CoL as the authors anticipated in the conclusive section. Some examples of interesting questions in this direction are: (i) Does the feature space adapts better/worse to novel tasks w.r.t. to  (ii) Can it be showed that some tasks that leverage the intramodal similarity structure features learned from M3CoL perform better? \n\n\n- In limitations extended training times on large-scale datasets are mentioned, but no data is reported around training times.\n\n\n\n_[1] Hao, Xiaoshuai, et al. \"Mixgen: A new multi-modal data augmentation.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2023._\n\n\n\n**minor:**\n \nI spotted some typos: \n- page 5 \"parantheses\" -> \"parenthesis\"\n- section 6 \"Conlusions\" -> \"Conclusions\"\n\n**formatting:** something looks to be off with the *Visualization of Attention Heatmaps* paragraph at page 9, probably due to negative space formatting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel loss for training multimodal models, starting from the observations that often there does not exist a bijective correspondence between samples of the two modalities (which for examples for images and text does not hold), while typical contrastive learning approaches assume so.\n In order to deal with the many-to-many correspondence problem they propose to combine existent approach to (i) relax the objectivity assumptions with an objective by leveraging intra modal similarities (ii) use classification heads on top of each modality (and their concatenation) (iii) Apply Mixup regularization on intra-modal samples, with regularization applied only during the early training stages for optimal performance \n\nThe method is validates on 4 different benchmarks for multimodal learning, comprising the image and text modality, small scale and large scale setting and medical datasets.  Ablation studies are performed to assess to contribution of each term of the loss", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I kindly ask to address the weaknesses listed in the weaknesses section, in particular: \n\n- As mentioned, the paper would need a better methodological contextualitazion for the proposed method  with respect to previous work  (Zou et al., 2023, Han et al 202, Gao et al., 2024, [1]).\n\n- Better justification ( possibly experimental or via discussion) of the benefits proposed method, rather than downstream performance on benchmarks. For more details, see the weaknesses section.\n\n\n\n_[1] Hao, Xiaoshuai, et al. \"Mixgen: A new multi-modal data augmentation.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2023._\n\n- I have some questions about the method. Some of the answers could be added to the paper to improve clarity:\n     - What does the multimodal label in Cross entropy corresponds to in practice? While is easy to think about labels for each modality, is not entirely clear when this labels should be available for multiple modalities. If this is the case then, how the problem differs from  a supervised classification problem  from multiple modalities? What I the impact of the contrastive term when the labels from single modalities and shared space are representative? \n     -  Does the regularization parameter $\\beta$ weighting the loss term need to change across different dataset? Is there a reason behind not weighting the cross entropy terms? Some discussion and insights on this would help simplify the use of M3CoL for end users.\n     - In the case multimodal labels are available and classes are perfectly separable, is it correct to say that the contrastive term of the loss is not needed?  (related to W2)\n    - In the case modalities are structured differently (i.e. classes are almost perfect separable in one modality but not in the other, as it seems the case for example from the UMAP plots on News24 in Figure  7 (a) and (b) ). Does the contrastive term account for this?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x9qmcHBITz", "reviewer": "Reviewer_gosT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- **Clarity** -- Excellent:\n\n  - The paper is clear and easy to follow.\n\n- **Novelty** -- Good:\n\n  - The approach is novel it combines mixup with multimodal and contrastive loss in a very smart way.\n\n- **Experiments**-- Excellent:\n\n  - The paper did a great job with the experimental section; they compared it with over 10 baselines on four datasets.\n\n  - The ablations justify the need for different losses introduced by M3CoL.\n\n  - The paper compared the robustness of M3CoL under conditions where one modality is random.\n\n## Weakness\n\n- **Method**\n\n  - The problem with M3CoL is that it has too many losses and many hyperparameters. First, mixup $\\lambda$, then the M3Co loss and the MutliSClip, and then 3 classifier losses, all of which require hyperparameters. Plus, when to switch between M3Co and MutliSClip. Reading the paper, it seems like they know the right value for all of this, which is kinda surprising, especially this ratio 0.33 M3Co + 0.67 MultiSClip. In practice, when there are many hyperparameters, it is usually very difficult to find the right ones, and as the number of losses and switches increases, complexity increases. So, training M3CoL is actually pretty difficult. Given that the performance is on par with others, it might be hard to justify why someone would use M3CoL over simpler methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces M3Col, which is a multimodal modal contrastive framework.\n\n-----------------------\n### Method\nM3Col consists of the following:\n-  First there are two encoders, one for each modality the input to the encoders are embeddings from a pretrained model.\n- The encoders take either embedding of the input itself, and the output of the encoder, in this case, is the input to three different classifier heads that try to predict a downstream task.\n     - Each of the first two classifier heads takes the input from a single modality equations 1 and  14.\n     - The third classifier head gets the concatenation of both modalities equation 15.\n- Each encoder also takes the weighted combinations of different samples given by equations 2 and 3; this is the mixup part.\n    - For the mixed inputs, the output of the encoders bidirectional Mixup contrastive loss M3Co for each modality given in equations 6,7 and 8.\n  - To reduce overfitting that might happen due to a mixup they implement a schedule that transitions from the mixup loss to a non-mixup multimodal contrastive loss given in equations 9,10 and 11.\n - Overall loss is given by equation 16.\n\n-----------------------\n### Experiments\n\nThe paper tested their approach on 4 different datasets two large ones N24News and Food-101 where they used pretrained models to get embeddings and two smaller ones  ROSMAP and BRCA where they trained their own encoders. They compared with over 10 baselines.\n\n#### Results in comparison to baselines:\n-  For larger datasets, in N24News  M3Col slightly outperform the baselines, while for Food-101 M3Col perform slightly worse. Overall, I would say their performance is on par with UniS-MMC\n\n- On smaller datasets, M3Col outperforms other baselines.\n#### Ablations:\nThe paper tested the effect of each loss they added to get the final M3Col on ROSMAP and BRCA datasets. They showed that, in fact, each loss contributes to the performance of the model in a positive way. They also validated through saliency maps that M3Col seems to focus on the correct object in comparison to other variations. \n\n#### Calibration and Robustness:\n- The paper compared M3CoL with multi-modal contrastive loss MutliSClip when predicting on random noise images and text encoder outputs. They showed that the model's confidence for M3CoL is lower in its predictions when both modalities are replaced\nwith random inputs in comparison to MutliSClip.\n- The paper looked into the effect of corrupting one modality; they found that M3CoL is more robust to corrupt modality in comparison to MutliSClip.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please give more details on how the hyperparameters are selected.\n- Please add the computational cost of the training  M3CoL in comparison to other models.\n- Please provide additional results showing the drop in performance when the optimal hyperparameters are not given especially for the switch between M3Co and MutliSClip.\n\n-Typos: \"Conlusions\" is misspelled on page 11.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ivnYBhWk7I", "reviewer": "Reviewer_Wg4t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The idea of relaxing the iid assumption through data-augmentation is quite interesting.\n2. The method seems to empirically work well, and the qualitative examples are convincing.\n3. Illustrations help a lot to understand the proposed approach.\n4. Experiments are quite extensive and it compares with many different baselines. Qualitative results are quite interesting and I think add value to the manuscript.\n\n**Weakness**\n1.  More details should be provided in the experimental part, to fully understand the results. For example:\n     1. In Table 6, what is the accuracy when none of the modalities are corrupted?\n     2. It would be better to use a histogram for Figure 5 (or two). KDE can be quite misleading at times.\n     3. It is not clear to me whether the results for the baselines were reproduced by the authors, or taken from somewhere else (maybe I missed it). But I would try to give enough details to convince the reader that comparisons are fair, which is not the case for me right now.\n     4. I would try to explain more in detail the Error Analysis and what it means. As of now, it feels that it has been overly summarized.\n     5. I find tables like table 3 too misleading, as they do a poor job at summarizing each method. Trying to obtain any kind of conclusions on \"which elements work better\" is a bad idea, as many nuances of \"seemingly equal models\" are hidden to the reader.\n     6. The use of bold is also misleading to my eye. There is no statistical significance on the difference between the top methods in many cases.\n2. The proposed method is rather complicated as it includes many moving parts, which I am not sure how much tuning they would need in other settings. Moreover, it scales as $O(M^2N^2)$, so it cannot include many modalities or samples. There is no mention as far as I can see on training times.\n3. Claims seem to strong in general. For example, it is said that the method is \"modality-agnostic\", but then it is said that the text modality needs to be treated differently for the mixing.\n4. Writing could be improved when explaining the maths, for example, I would completely drop the notation $\\tilde p_i^{(1)}$ and use $\\tilde p_{i,j}^{(1)}$ everywhere, as it is confusing otherwise.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors introduce a multi-modal architecture to learn the relationships between modalities and learn relationships between samples beyond pairwise-ones. To this end, the authors soften the iid assumption by pairing samples with data-augmented modalities created from their counterpart modalities and randomly picked samples. The contrastive losses are simmetrized for all modalities and samples, combined with unimodal and multimodal classifiers to further guide the training. Empirically, the authors show that their approach is competitive with existing methods, while qualitatively showing to have learnt better relationships across samples, and being more robust to modality noise, than other approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is lacking clarity in the writing as well as on the experimental details, as described above. These problems would need to be addressed in my view. There are other clarity problems and nitpicks I highlight below.\n\nNitpicks and typos:\n- $M_j^{(1)}$ in figure 1 left should be $M_j^{(2)}$. I also do not understand why using Gemini to generate the samples, instead of taking two from the Food101 dataset.\n- The dot product with \"$L^2$-normalized\" vector is the cosine similarity, the authors should say the use that instead. Similarly with all the softmax operations written explicitly everywhere.\n- Indexing in the context can by confusing. I would encourage authors to pick a consistent way of using the $i$, $j$, and $k$ subindices and stick to that during the entire manuscript.", "broader_impact_concerns": "I do not think there are any concern worth mentioning in the broader impact beyond the additional computational resources required by the proposed approach.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received the comments of three reviewers. In the initial review, three reviewers mainly show concerns about lack of a good methodological contextualization, unclear motivation, too many hyperparameters and more training details that should be reported etc. After the rebuttal, the authors successfully persuade the reviewers to change the opinions. For the present, three reviewers all agree with "Acceptance" towards this submission. However, there are some remaining points raised by the reviewers, which AC thinks should be followed by the authors in the further revision.

- Hyperparameter tuning: It is not clear a priori how to select the hyperparameters for the proposed loss, nor how each term contributes to the overall objective depending on the data. While some clarifications were provided during the rebuttal, it remains unclear in which settings M3COL offers a clear advantage over simpler alternatives.

- Analysis of learned representations: A deeper analysis of the representations learned by M3COL—such as performance on downstream tasks or feature interpretability—would help clarify the method's practical utility and better contextualize its benefits.

Please following the remaining suggestion to further improve the submission, then it will reach to the acceptance without concerns from AE.

Best,

AE
Decision: Accept (poster)