M3CoL: Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification

Authors: raja  kumar
raghav  singhal
pranamya  kulkarni
deval  mehta
kshitij  jadhav
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-19 11:55:17

Abstract: Deep multimodal learning has shown remarkable success by leveraging contrastive learning to capture explicit one-to-one relations across modalities. However, real-world data often exhibits shared relations beyond simple pairwise associations. We propose M3CoL, a Multimodal Mixup Contrastive Learning approach to capture nuanced shared relations inherent in multimodal data. Our key contribution is a Mixup-based contrastive loss that learns robust representations by aligning mixed samples from one modality with the corresponding samples from other modalities. For multimodal classification tasks, we introduce a framework that integrates a fusion module with unimodal prediction modules for auxiliary supervision during training, complemented by our proposed Mixup-based contrastive loss. Through extensive experiments on diverse datasets (N24News, ROSMAP, BRCA, and Food-101), we demonstrate that M3CoL effectively captures shared multimodal relations and generalizes across domains. It outperforms state-of-the-art methods on N24News, ROSMAP, and BRCA, while achieving comparable performance on Food-101. Our work highlights the significance of learning shared relations for robust multimodal learning, opening up promising avenues for future research.

Main paper: 
 introduction: 
 In the era of abundant multimodal data, it is crucial to equip artificial intelligence with multimodal capabilities [1] . At the heart of advancements in multimodal learning is contrastive learning, which maximizes similarity for positive pairs and minimizes it for negative pairs, making it practical for multimodal representation learning. CLIP [2] is a prominent example that employs contrastive learning to understand the direct link between paired modalities and seamlessly maps images and text into a shared space for cross-modal understanding. However, traditional contrastive learning methods often overlook shared relationships between samples across different modalities, which can result in the learning of representations that are not fully optimized for capturing the underlying connections between diverse data types. These methods focus on distinguishing between positive and negative pairs of samples, typically treating each instance as an independent entity. They tend to disregard the rich, shared relational information that could exist between samples within and across modalities.While traditional contrastive learning methods treat paired modalities as positive samples and noncorresponding ones as negative, they often overlook shared relations between different samples. As shown in the left panel of Figure 1 (Left panel), classical contrastive learning approach assumes perfect one-to-one relations between modalities, which is rare in real-world data. For example, shared ‡ Equal Contributions. Author ordering determined by coin flip over Google Meet. Our code is available at: https://github.com/RaghavSinghal10/M3CoL. elements in images or text can relate even across separate samples, as illustrated by the elements like "tomato sauce" and "basil" in Figure 1 . Our approach, illustrated in the right panel of Figure 1 , goes beyond simple pairwise alignment by capturing shared relationships across mixed samples. By creating newer data points through convex combinations of data points our method effectively models complex relationships, such as imperfect bijections [3] , enhancing multimodal performance.Our approach builds upon the success of data augmentation techniques such as Mixup [4] and their variants [5] [6] [7] , which have proven beneficial for enhancing learned feature spaces, improving both robustness and performance. Mixup trains models on synthetic data created through convex combinations of two datapoint-label pairs [8] . These techniques are particularly valuable in low sample settings, as they help prevent overfitting and the learning of ineffective shortcuts [9, 10] , common in contrastive learning. Building on the success of recent Mixup strategies [11] [12] [13] and MixCo [14] , we introduce M3Co, a novel approach that adapts and enhances contrastive learning principles to complex multimodal settings. M3Co modifies the CLIP loss to handle multimodal scenarios, addressing the problem of instance discrimination, where models overly focus on distinguishing individual instances instead of capturing relationships between modalities. M3Co eliminates instance discrimination and enhances robust representation learning by capturing shared relations. Our results demonstrate improvements in performance and generalization across a range of multimodal tasks. 
methodology: 
 Pipeline Overview. Figure 2 depicts our framework, which comprises of three components: unimodal prediction modules, a fusion module, and a Mixup-based contrastive loss. We obtain latent representations (using learnable modality specific encoders f (1) and f (2) ) of individual modalities and fuse them (denoted by concatenation symbol '+') to generate a joint multimodal representation, which is optimized using a supervised objective (through classifier 3). The unimodal prediction modules provide additional supervision during training (via classifier 1 and 2). These strategies enable deeper integration of modalities and allow the models to compensate for the weaknesses of one modality with the strengths of another. The Mixup-based contrastive loss (denoted by L M3Co ) updates the representations by capturing shared relations inherent in the multimodal data. i denote the i-th samples for the first and second modalities, respectively. The modality encoders, f (1) and f (2) , generate the corresponding embeddings p (1) i and p(2) i :EQUATIONWe generate a mixture, x(1) i,j , of the samples x(1) i and x(1) j by taking their convex combination.Similarly, we generate a mixture, x(2) i,k , using the convex combination of the samples xi and x(2) k (Eq. 2). For the text modality, instead of directly mixing the raw inputs, we mix the text embeddings [15] . The mixing indices j, k are drawn arbitrarily, without replacement, from [1, N ], for both the modalities. We mix both the modalities using a factor λ ∼ Beta(α, α). Based on the findings of [4] , which demonstrated enhanced performance for α values between 0.1 and 0.4, we chose α = 0.15 after experimenting with several values in this range. The mixtures are fed through the respective encoders to obtain the embeddings: p(1) i,j , and p(2) i,k (Eq. 3).x(1)EQUATIONp( 1)EQUATIONEQUATIONi,k )We generate embeddings for the entire batch p(1) and p( 2) , where the i-th elements, pi and p(2) i , correspond to p(1) i,mi and p(2) i,mi , respectively. The unidirectional contrastive loss [9, [16] [17] [18] [19] over p (2) is conventionally defined as:EQUATIONEQUATIONEQUATIONwhere • indicates dot product and τ is a temperature hyperparameter. While this formulation is suitable for computing similarity among aligned samples from different modalities, our method requires flexibility to handle both aligned and non-aligned samples. To achieve this, we define the unidirectional multimodal contrastive loss between pi and pm over p (2) as:EQUATIONi , p (2) ; m) = -log exp p(1)EQUATIONwhere p (1) and p (2) are L 2 normalized, τ is a temperature hyperparameter, and m is a sample index in [1, N ] . Although the multimodal contrastive loss (Eq. 5) can learn indirect relations, it is insufficient for learning shared semi-positive relations between modalities. Therefore, we introduce a Mixup-based contrastive loss to capture these relations that promotes generalized learning, as this process is more nuanced than simply discriminating positives from negatives. Now, following standard works [2, [16] [17] [18] , we make our loss bidirectional. We define this bidirectional Mixup contrastive loss M3Co for each modality (Eq. 6, 7) and the total M3Co loss as:EQUATIONi,j , p (2) ; i)EQUATIONi,j , p (2) ; j)EQUATION(2)EQUATIONi,k , p (1) ; i)EQUATIONi,k , p (1) ; k)EQUATIONEQUATIONk , p(2) ; i)EQUATIONM3Co , where p (1) , p(1) , p (2) , and p(2) are L 2 normalized. Note that the parts of the loss functions in Eq. (6, 7) inside curly parantheses make them bidirectional. Mixupbased methods enhance generalization by capturing clean patterns in the early training stages but can eventually overfit to noise if continued for too long [20] [21] [22] . To address this, we implement a schedule that transitions from the Mixup-based M3Co loss to a non-Mixup multimodal contrastive loss. We design this transition so that the non-Mixup loss retains the ability to learn shared or indirect relationships between modalities. By using a bidirectional SoftClip-based loss [9, 16, 23] , we relax the rigid one-to-one correspondence, allowing the model to capture many-to-many relations [23, 24] . The bidirectional MultiSoftClip loss for each modality (Eq. 8, 9) and its combination is:EQUATIONi , p (1) ; l) + L sim (p(1)l , p (2) ; i)(8) L (2) MultiSClip = 1 N N i=1 N l=1 exp p (2) i • p (2) l /τ N t=1 exp p (2) i • p (2) t /τ • L sim (p (1) i , p (2) ; l) + L sim (p (2)l , p (1) ; i)EQUATIONMultiSClip , where p (1) and p (2) are L 2 normalized. The M3Co and MultiSClip losses for M modalities is:L M3Co = M i=1 M j>i L (i,j) M3Co , L MultiSClip = M i=1 M j>i L (i,j) MultiSClip (10)Unimodal Predictions and Fusion. The encoders produce latent representations for each of the M modalities, serving as inputs to individual classifiers that generate modality-specific predictions. These representations are used for modality-specific supervision only during training. The unimodal prediction task, L CE-Uni , involves minimizing the cross-entropy loss L CE between these predictions and the corresponding ground truth labels, for each modality. We merge the unimodal latent representations by concatenating them and pass the combined representation to the output classifier. These predictions serve as the final outputs used during inference. The multimodal prediction process, L CE-Multi , minimizes the cross-entropy loss between the predictions and the corresponding labels.Combined Learning Objective. Our overall loss objective utilizes a schedule to combine our M3Co and MultiSClip loss functions weighted by a hyperparameter β, along with the unimodal and multimodal cross-entropy losses. We use M3Co for the first one-third [20] part of training, and then transition to MultiSClip. The end-to-end loss is defined as:EQUATION3 Experiments and ResultsDatasets and Implementation Details. We evaluate on four diverse multimodal classification datasets: N24News [25] , Food-101 [26] , ROSMAP [27] , and BRCA [27] . N24News and Food-101 are image-text classification datasets. ROSMAP and BRCA are medical datasets, each containing three modalities: DNA methylation, miRNA expression, and mRNA expression. We use a ViT [28] as the image encoder for N24News and Food-101. For N24News, the text encoder is a pretrained BERT/RoBERTa [29, 30] , while we use a pretrained BERT as the text encoder for Food-101. The classifiers for the above two datasets are three layer MLPs with ReLU activations. For ROSMAP and BRCA, which are small datasets, we use two layer MLPs as feature encoders for each modality, and two layer MLPs as classifiers. Details and related work are presented in Appendix A.1 and A.5. Discussion and Conlusions. Aligning representations across modalities presents significant challenges due to the complex, often non-bijective relationships in real-world multimodal data [3] . These relationships can involve many-to-many mappings or even lack clear associations, as exemplified by linguistic ambiguities and synonymy in vision-language tasks. We propose M3Co, a novel contrastive-based alignment method that captures shared relations beyond explicit pairwise associations by aligning mixed samples from one modality with corresponding samples from others. Our approach incorporates Mixup-based contrastive learning, introducing controlled noise that mirrors the inherent variability in multimodal data, thus enhancing robustness and generalizability. The M3Co loss, combined with an architecture leveraging unimodal and fusion modules, enables continuous updating of representations necessary for accurate predictions and deeper integration of modalities. This method generalizes across diverse domains, including image-text, high-dimensional multi-omics, and data with more than two modalities. Experiments on four public multimodal classification datasets demonstrate the effectiveness of our approach in learning robust representations that surpass traditional multimodal alignment techniques.M3Co) outperforms the MultiSClip loss (only MultiSClip) by learning more robust representations through Mixup-based techniques, which prevent trivial discrimination of positive pairs. Furthermore, using an individual contrastive alignment approach (only M3Co) throughout the entire training process without transitioning to the MultiSClip loss results in suboptimal outcomes. This can be attributed to the risk of over-training with Mixup-based loss, which may negatively impact generalization. This demonstrates the necessity of the transition of the contrastive loss during training (0. The attention heatmaps generated using the embeddings from our trained M3CoL model in Figure 3 and 4 highlight image regions most relevant to the input word. We generate text embeddings for class label words and corresponding image patch embeddings, computing attention scores as their dot product. This visualization aids in understanding the model's focus, decision-making process, and association between class labels and specific image regions. Importantly, it also indicates the correctness of the learned multimodal representations, revealing the model's ability to ground visual concepts to semantically meaningful regions. Approaches in multimodal learning are broadly categorized into alignment-based methods, which capture modality-invariant characteristics [51, 52] , and aggregation-based techniques that combine features across modalities [53, 54] . The design of multimodal networks is typically informed by the task objective, available data, and computational constraints [55] [56] [57] [58] . Common strategies include inputting all modalities as token embeddings, performing cross-attention between modalities, concatenating representations, and ensemble-based combination of modality-specific predictions [1] .Contrastive Learning. Contrastive learning has driven significant progress in unimodal and multimodal representation learning by distinguishing between similar (positive) and dissimilar (negative) pairs. In multimodal contexts, cross-modal contrastive techniques align representations from different modalities [2, 59, 60] , with approaches like CrossCLR [61] and GMC [62] focusing on global and modality-specific representations. Contrastive learning approaches for paired image-text data, such as CLIP [2] , ALIGN [59] , and BASIC [63] , have demonstrated remarkable success across diverse vision-language tasks. Subsequent works have aimed to enhance the efficacy and data efficiency of CLIP training, incorporating self-supervised techniques (SLIP [64] , DeCLIP [65] ) and fine-grained alignment (FILIP [66] ). The CLIP framework relies on data augmentations to prevent overfitting and the learning of ineffective shortcuts [9, 10] , a common practice in contrastive learning.Unimodal and Multimodal Data Augmentation. Data augmentation has been integral to the success of deep learning, especially for small training sets. In computer vision, techniques have evolved from basic transformations to advanced methods like Cutout [67] , Mixup [4] , CutMix [5] , and automated approaches [6, 68] . NLP augmentation includes paraphrasing, token replacement [69, 70] , and noise injection [71] . Multimodal data augmentation, primarily focused on vision-text tasks, has seen limited exploration, with approaches including back-translation for visual question answering [72] , text generation from images [73] , and external knowledge querying for cross-modal retrieval [74] .MixGen [75] generates new image-text pairs through image interpolation and text concatenation. In contrast, our proposed augmentation technique focusing on the early training phase is fully automatic, applicable to arbitrary modalities, and designed to leverage inherent shared relations in multimodal data.Relation to Mixup. Mixup [4] , a pivotal regularization strategy, enhances model robustness and generalization by generating synthetic samples through convex combinations of existing data points.Originally introduced for computer vision, it has been adapted to NLP by applying the technique to text embeddings [15] . Our proposed augmentation differs from Mixup in several key aspects: it is designed for multi-modal data, takes inputs from different modalities, and does not rely on one-hot label encodings. By extending the Mixup paradigm to complex, multi-modal scenarios and focusing on the early training phase, our method broadens its applicability while leveraging inherent shared relations in multimodal data. 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: M3CoL: Promising Multimodal Mixup Approach with Room for Clarity and Improvement
main_review: main_review: ## Overview:

This paper introduces M3CoL, a Multimodal Mixup Contrastive Learning approach for multimodal classification tasks. The method aims to capture shared relations across different samples in multimodal data by using a Mixup-based contrastive loss. 

## Pros:

1. Clear motivation: The paper provides a compelling rationale for capturing shared relations in multimodal data.
2. Well Detailed Architecture: The authors present an architecture that is easy to understand and well motivated.
3. Comprehensive evaluation: The authors test their approach on diverse datasets across different domains.

## Cons:

1. Complexity: The proposed method appears quite complicated, which may make it challenging to implement or interpret. Specifically, the loss function although well detailed seems very large (O(n^2) terms for the batch size n).
2. Potential overfitting: The method of mixing potentially unrelated elements might lead to overfitting to noise, which they compensate for by a scheduled loss function.
3. Lack of clarity on novelty: The distinction between this work and previous methods (references 11-13 and MixCo) is not clearly articulated. Is it that no previous methods perform multimodal mixing?
4. Improve results presentation: Include a succinct results table in the main paper, possibly moving some detailed equations to the appendix to make room. Maybe move the full multiSClip loss to the appendix? or try to combine eq. 8 and 9 with $\mathcal{L}_\text{M3CO}^{(K)} $ for $ K \in {1,2}$ or something similar.

## Suggestions:

1. Clarify the mixing process: Explain how the method handles cases where mixed elements are completely distinct and how this affects the model's learning.
2. Consider unimodal inference: Explore whether the unimodal classifiers that were used during training can perform inference alone, which could be a valuable feature. It might be interesting to see if the unimodal methods trained through M3CoL perform the same as a regular unimodal mode, giving M3CoL the flexibility to perform with each modality individually as well.

## Typos and Clarifications Needed:

1. Figure 1: Are the +sim's mixed in the same way with the λ's? 
2. Line 72: Define $m_i$, as its meaning is not clear from the context.
title: title: Review
main_review: main_review: ## Summary
The paper introduces a method M3CoL to enhance multimodal learning by capturing shared relations between different modalities using Mixup-based contrastive learning. The M3CoL aligns mixed samples from one modality with corresponding samples from other modalities, thereby capturing nuanced shared relations inherent in multimodal data. The framework integrates a fusion module with unimodal prediction modules for auxiliary supervision during training. Extensive experiments on diverse datasets demonstrate that M3CoL effectively captures shared multimodal relations, outperforming state-of-the-art methods on several benchmarks.

## Strengths
1. The introduction of Mixup-based contrastive loss to capture shared relationships between different modalities that extends beyond traditional one-to-one contrastive learning.
2. The paper provides extensive experimental results on diverse datasets, demonstrating the effectiveness of M3CoL.

## Weaknesses
1. The performance of M3CoL may be sensitive to the choice of hyperparameters, such as the temperature parameter and the Mixup coefficient. It should be discussed.
2. It would be better to discuss the performance of the method on other types of multimodal data, such as  audio-image/video.
3. It would be better to analyze and discuss the limitations of the method and the samples that failed to predict.
title: title: Paper proposes a MixUp augmentation for Multi-model contrastive learning; The proposed method seems connected to introducing label noise in the first stages of the representation learning and leads to downstream performance gains.
main_review: main_review: This paper proposes a Mixup-based augmentation for multimodel contrastive learning which seems quite similar to having label noise. The authors show that the proposed augmentation which is applied toward the start of the training but abandoned later on is beneficial and leads to downstream performance gains on text-image and multimodel medical datasets. 

The contribution is based on heuristics and could be further developed to bring more insights into the inner workings of the proposed method that led to this performance increase (i.e., link to label noise, link to temperature parameter). That being said the topic is aligned with the interests of the audience, the paper is well-written and the idea can be interesting if further developed notably to better understand the key components behind multi-model representation learning. 

The format of this submission should be improved, the 4-page format does not include the experimental results which are delegated to the appendix. Meanwhile, the explanation behind the method which I believe could hold in 2 pages takes up 3 out of the 4 pages of this submission.
title: title: Evaluation of paper "M3CoL: Multimodal classification using shared relationships through multimodal hybrid contrastive learning"
main_review: main_review: The author proposes a novel multimodal classification method called M3CoL, which introduces multimodal mixed contrastive learning to capture shared relationships beyond simple pairwise associations. This article addresses the limitations of traditional contrastive learning methods by focusing on shared multimodal relationships and applies them to classification tasks across different datasets. This method outperforms several state-of-the-art methods on multiple datasets, highlighting its importance.