What to align in multimodal contrastive learning?
Authors: Benoit Dufumier
Javiera Castillo Navarro
Devis Tuia
Jean-Philippe Thiran
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal representation learning
Self-supervised learning
Contrastive learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We present CoMM, a new multimodal representation learning method that learns synergistic, unique and redundant information between modalities.

Abstract: Humans perceive the world through multisensory integration, blending the information of different modalities to adapt their behavior.
Contrastive learning offers an appealing solution for multimodal self-supervised learning. Indeed, by considering each modality as a different view of the same entity, it learns to align features of different modalities in a shared representation space. However, this approach is intrinsically limited as it only learns shared or redundant information between modalities, while multimodal interactions can arise in other ways. In this work, we introduce CoMM, a Contrastive Multimodal learning strategy that enables the communication between modalities in a single multimodal space. Instead of imposing cross- or intra- modality constraints, we propose to align multimodal representations by maximizing the mutual information between augmented versions of these multimodal features. Our theoretical analysis shows that shared, synergistic and unique terms of information naturally emerge from this formulation, allowing us to estimate multimodal interactions beyond redundancy. We test CoMM both in a controlled and in a series of real-world settings: in the former, we demonstrate that CoMM effectively captures redundant, unique and synergistic information between modalities. In the latter, CoMM learns complex multimodal interactions and achieves state-of-the-art results on seven multimodal tasks.

Published as a conference paper at ICLR 2025
WHAT TO ALIGN IN MULTIMODAL CONTRASTIVE LEARNING?
Benoit Dufumier! Javiera Castillo Navarro
Devis Tuia*
Jean-Philippe Thiran
EPFL 2 NNeuroSpin, CEA 3 *CEDRIC, CNAM
Radiology Department, CHUV
ABSTRACT
Humans perceive the world through multisensory integration, blending the infor- mation different modalities to adapt their behavior. Alignment through con- trastive learning offers an appealing solution for multimodal sulf-suue learn- ing. Indeed, by considering each modality as a different view of the same entity, it learns align features of different modalities in shared representation space. However, this approach is intrinsically limited as it only learns shared or redun- dant information between modalities, while multimodal interactions can arise in other ways. In this work, we introduce CoMM, Contrastive Multimodal learn- ing strategy that enables the communication between modalities in a single multi- modal space. Instead of imposing cross- or intra- modality constraints, we propose to align multimodal representations by maximizing the mutual information be- tween augmented versions of these multimodal features. Our theoretical analysis shows that shared, synergistic and unique terms of information naturally emerge from this formulation, allowing to estimate multimodal interactions beyond re- dundancy. We test CoMM both in controlled and in a series of real-world set- tings: in the former, we demonstrate that CoMM effectively captures redundant, unique and synergistic information between modalities. In the latter, we show that CoMM learns complex multimodal interactions and achieves state-of-the-ar results on seven multimodal tasks. Code is available here.
INTRODUCTION
Multisensory multimodal learning (Baltrušaitis et al., 2018) involves extracting and processing information from multiple sources (e-g. text, audio, images, tabular data, etc.). The whole human experience is inherently multimodal: different sensory signals are combined to give us the necessary information to explore our environ- ment. Many of the simplest tasks we tackle in our daily lives are multimodal:the way we perceive the flavor of our food or drinks does not depend solely on our taste, but also on what we see (Morrot al., 2001) or what we hear (Woods al., 201 while we eat. McGurk & MacDonald (1976) have also shown that visual stimuli interact with audio signals to perform human speech recognition.
simultaneously see, hear, smell, taste and feel, and these
Despite the inherent multimodality of sensory systems, machine learning has largely concentrated on single-modality models, with few exceptions in areas like audio-visual speech recognition ( (Yuhas al., 1989; Ngiam et al., 2011), multimedia content retrieval (Atrey et 2010; Snoek & Worring, 2005), and video-based human behavior analysis (Kraaij et al., 2005). Nowadays, with the emer- gence self-supervised strategies and their impressive capacities for learning representations in computer vision (Chen et al., 2020a; He et al., 2020; Caron et al., 2021), NLP (Radford et al., 2018; Devlin et al., 2019) or audio (Oord et al., 2018; Niizumi et al., 2021), the paradigm has shifted to learning general multimodal representations from unlabeled data and then fine-tune the models to specific multimodal tasks. Recent works have shown success at training multimodal represen- tations by using cross-modal contrastive objectives (Radford al., 2021: Jia et al., 2021) align the representations in a shared embedding space. However, this training strategy only works under the multiview redundancy assumption, i.e., considering that all task-relevant information is shared between modalities and redundantly contained in either one of them separately. In particular, for vision-languee language tasks, this can be seen as clever way to perform supervised learning on a visual encoder, which explains their success to transfer to visual classification tasks.
ddeottes equal contribution. Contact information: surnaee)

---

Published as a conference paper at ICLR 2025
CoMM
Modalities
Method
Interactions
Task
X1
X2
Cross
CoMM
X1
U1
texture( (x) dots
x
Uniqueness
X2
Multimodsi pressenntttoo
U2
texturee (g) stripes
x
Cross
shape( Xg. circle
Redundancy
R
X1
shope(X1..Xa) square
sim
((toxture(x)), (g)) C CMM)
X2
Symergo
x
Ze Unimmdda repanssntaaiiess
(1)exture(x1))
a) Multimodal models
b) Multimodal interactions
Figure 1: a) We propose CoMM, a contrastive multimodal approach that allows the interplay of multiple modalities and learns multimodal interactions. Unlike previous multimodal models (Cross) that align cross-modal features, CoMM aligns multimodal features in shared representation space. b) Multimodal interactions are task-dependent, thus a model needs to capture of them to generalize to any multimodal task. CoMM's new paradigm captures multimodal interactions beyond redundancy.
Nonetheless, these solutions are insufficient in many cases, as the interactions between modalities can arise in several ways to perform a specific task (Bertschinger et al., 2014): redundancy (R) arises when the task can be performed using either of the modalities because they contain redundant information; uniqueness (U) appears when only one of the modalities contains all the necessary information complete the task; synergy (S) emerges when both modalities have complementary information, and they are needed simultaneously to fulfill the task. Modeling these interactions to perform multimodal learning is highly challenging as the interplay between Rt, S and U is task- dependent and difficult to measure in complex real-life scenarios. Fig. lb shows simple tasks where the predominant type of interaction can be easily identified. We observe that we need to model specific interactions for the same input modalities to perform a specific task. Therefore, model must capture all these terms learn task-agnostic multimodal representations.
To achieve task-agnostic multimodal learning, we propose Contrastive MultiModal self- supervised pre-training method (CoMM) that enables the communication of modalities in a single multimodal space. Unlike previous contrastive multimodal methods that impose cross-modal con- straints to align unimodal representations, we propose to leverage simple multimodal architecture to fuse multimodal inputs into common representation and then align the multimodal features by maximizing the mutual information between augmented versions of these features (see Fig. la). CoMM enables to model multimodal interactions -including redundancy, uniqueness and synergy- in the context of multimodal representation learning as these terms naturally emerge from our contrastive multimodal formulation.
CoMM's formulation is well-aligned with the global workspace theory (Baars, 1988; Goyal & Ben- gio, 2022) in cognitive neuroscience, which considers the nervous system set of multiple spe- cialized processors working in parallel and claims the existence of a shared representation, which can be modified by any selected processor and whose content broadcast all processors. By analogy, CoMM considers a shared representation space built from parallel streams of modality- specific processors. When task requires knowledge from a given modality or interactions between modalities, only these parts of the representation space should be used.
Based on Partial Information Decomposition (Williams& & Beer, 2010; Bertschinger et al., 2014), we built strong theoretical basis for CoMM to learn multimodal interactions in self-supervised way. Empirically, we show that CoMM effectively captures redundant, unique and synergistic informa- tion between modalities in controlled environment, where the type of interaction is known. Then, in a series of real-world datasets -from different domains (healthcare, robotics, etc.) and including diverse data types (image, text, audio, etc.)-, CoMM achieves state-of tftt--of-the-at results seven mul- timodal tasks with two or three modalities. In cases, CoMM showed to be a versatile framework capable of handling any number of modalities, various data types, and different domains.
2

---

Published as a conference paper at ICLR 2025
2 QUANTIFYING MULTIMODAL INTERACTIONS
Problem setup. Let X1; X2,..., Xn be random variables representing n. different data modalities (e.g. images, text, audio, tabular data, etc.), and a given task Y. Our goal is to learn latent variable = f(X) that is good representation of X = (X, Xn) for Y.
For our theoretical analysis, we set n 2, as multimodal interactions have not been characterized yet by PID for larger n. In practice, CoMM's implementation for n > 2 is straightforward and tested in Section 4.3. All proof' can be found in Appendix G.
For to be good representation of X it should capture the task-relevant information that X contains. Therefore, we need to model the information between the joint variable X and the task Y: I(; = [(X1 Y).
Partial information decomposition (PID) (Williams & Beer, 2010; Bertschinger et al., 2014) states that multivariate mutual information [(X,, X2; Y) is decomposed into three forms of interactions: (i) Uniqueness. This term appears when the task Y can be completed by leveraging only one of the modalities. U, (resp. U2) refers to the case when X1 (resp. X2) contains all task-relevant information.
iii) Redundancy. When X1 and X2 contain the same information about Y. R corresponds to this redundant or shared information.
iiii)
Synergy. Noted by S, this term only emerges when X1 and X2 are simultancously present, because they bring different and complementary task-relevant information.
Thus, the information that (2) has about Y can be written as the contribution of four terms:
T(X1, X2;Y) R + S + U1 + U2
Moreover, Eq. (1) and the application of the chain rule of mutual information yield the following consistency equations between R, S, U1 and U2:
1(X;;Y) = R+ U1
Y(X2; R U2,
1(x R-S
(2)
Existing methods using contrastive objectives to learn multimodal representations (Jia et al., 2021; Radford et al., 2021; Tian et al., 2020a) impose cross-modal constraints by maximizing an estima- tor of [(1;; X2) as an approximation of X2; However, this strategy is limited by the multiview redundancy assumption.
Definition (Multi-view redundancy) 3e > 0 such that / ((;;X1|X2) E and I(Y; X1)
This assumption states that most task-relevant information is shared across modalities and the non- shared information is (at most) a small e. In other words, any of the modalities contains enough information to fulfill the downstream task Y, and they can provide some kind of "supervision" to one another, which explains their success for zero-shot image classification (El Banani al., 2023).
Lemma 1 Under the multiview redundancy assumption, cross-modal contrastive learning methods are limited only learn the redundant information
What happens when other sources of multimodal information intervene? FactorCL (Liang et al., 2023b) is good initial step to integrate uniqueness and redundancy into multimodal contrastive leaming by applying multimodal augmentations. However, it heavily relies on the assumption that optimal multimodal augmentations can be obtained by applying a two-step process based on condi- tional augmentations. We argue that this hypothesis is unrealistic, as the first unique augmentation strongly related to task-relevant information of different modalities. For example, if the text cap- tion is "a yellow flower", then color jittering should not be applied to image depicting a flower. Besides, the factorized formulation of FactorCL impractical as it is prone to cumulative errors. Finally, this method does not consider the synergy between modalities.
In contrast, we propose a model that relies solely in the main hypothesis of contrastive learning, extended to the multimodal case, without relying strong assumptions about multimodal relation- ships nor conditional augmentations.
Assumption (Minimal label-preserving multimodal augmentations) We assume the existence of T*, a set of multimodal augmentations such that for any t. E T and X' = t(X), we have T(X, X') ((X
3

---

Published as a conference paper at ICLR 2025
Even if Assumption might seem strong at first glance, it actually makes sense in the context of multimodal representation learning. Indeed, coming back to the example of the flower image with "a yellow flower" a caption, applying color jittering the image would allow the model to focus on other interactions (uniqueness or synergy) rather than color redundancy or even to refocus on other features (like the flower shape). This is discussed more in-depth in Appendix C.4.
Moreover, our assumption allows for a larger spectrum of augmentations, without being constrained the set T {((X) (X),, (2()))) of transformations that can be decomposed in indepen- dent unimodal augmentations.
3 COMM: CONTRASTIVE MULTIMODAL LEARNING
We aim to learn multimodal representations that are transferable to any multimodal task. Contrastive learning has shown promising results in multimodal learning. However, current approaches fail to capture multimodal interactions other than redundancy, as shown in Section 2.
Our strategy builds upon multiview contrastive learning theory and extends to the multimodal case. It is based on two main components:
(i) A multimodal architecture, with specialized encoders to process any data type, and an attention based fusion module to obtain a final multimodal representation.
(ii) contrastive objective that naturally captures unique, redundant, and synergistic interactions between different data modalities.
3.1 TOWARDS EFFECTIVE MULTIMODAL REPRESENTATIONS
To obtain robust, task task- agnostic and common representations Z that capture uniqueness, redundancy and synergy from different input modalities, we design fo - neural network parameterized by 8- such that Zo fo(X) fa(X, X2).
We define X' t(X) with T stochastic mapping! (multimodal augmentation) of X and Z/ = fx()).
Given data processing inequalities for Markov chains X X Zyp and Zi X Zg, have:
I(Zo; Z%)= S T(X, %4)= < 7(X,
With these inequalities, we can prove the following lemmas:
Lemma By optimizing fe to maximize I 1(oo; Kb). and if we assume an expressive enough network fo. we have at optimum:
I(Zg- Zz. I(X,'')
(4)
Lemma Let So- be optimal, i.e. fo- maximizes I I(Za, Zz). Then, we have the equality
T(Zg-; T(X';) Y). If we consider the special case TT (1:) such that X' 1,(X) = X;
and Z/. = fe- (X.) = Ze for [,, then it follows:
I(Z, 1(X;;)) R+ U
(5)
Lemma implies that optimal representations Z, preserve all the task-relevant information contained in modality i. Interestingly, we do not require Assumption for this equality to hold.
The previous theoretical developments lead the key ingredients for CoMM's contrastive objec- tives to succeed at capturing multimodal interactions (see Section 3.3 for practical implementation):
(i) Following Lemma 2, U1 + U2 + R + S = T(X, Y) can be learned by optimizing the term T(Zo, %%) for T = T since T(X, X') Y) by Assumption 1;
(ii) Thanks to Lemma 3, R + U, for E [1,2} can be directly learned by optimizing the term T(Za, %%) for T {44).
Here, T can be any of mappings, egg. T T
4

---

Published as a conference paper at ICLR 2025
3.2 MULTIMODAL ARCHITECTURE
Our architecture for multimodal rep- resentation learning is presented in Fig. 2. In order to capture multi- modal interactions, the model con- sists of mainly three components: Encoders. Each modality is en- coded independently by one the modality-speciine encoders.
Latont
Commalcattoo
Thorsforme
X
Ensoder
Latent Comverter
Zo = fo(X)
Latent converters. Linear mod- ules that transform features into mppelity sequences of em- beddings. After the latent convert- ers, concatenation operation gath- ers these sequences to be fed into transformer architecture.
Figure 2: CoMM's model architecture. Inputs from dif- ferent modalities X (X1,..., ........)) are first encoded by modality-specifie encoders. Modality-seecific features are processed by latent converters map them into sequences of embeddings which are concatenated and fused by trans- former block. The output is a single multimodal feature Zg.
Transformer block. The goal of this module is to perfom the fusion of the modality-specifii embeddings through the multihead self-attention layers in the transformer block, obtaining the final multimodal embedding Z.
More information about the specific modules used as modality encoders, about the latent converters and the transformer block architecture be found in Appendix B.
3.3 TRAINING
Given a multimodal input X = -X.,.... Xn) and a set of label-preserving multimodal transfor- mations T two augmentations f', t" drawn from T and applied X, obtaining X' ((() and X" ("(X). We also consider projections (with a slight abuse of notation) X; ([MSK), ..... X;, .... [MSK) for i {1,...,, n), where every modality is masked except for the i-th.
These terms are encoded by the network to obtain (n-+2) embeddings, namely: Z',z", and [Z: } (2n + 1) mutual information terms are then optimized through backpropagation: 1(2', 2") to max- imize 1 2²) in Eq. (3) and both 1 1(2,, Z') and I (2,, Z") to better approximate R + U; in Eq. (5) for E n..
We use the InfoNCE (Oord et al., 2018) estimator of mutual information due to its simplicity and strong results in the self-super learning literature:
exp 2, 2(TO))
INCE Z, Z') = E -p 2')) ~p(2))
log
(6)
%/ exp sim 2/EE))
Given this estimator, our final training loss can be written as:
LCMM İNCR(Z', 2")
Σ (NNC((Z,,)) Z') INCE Zi, 2n)) =: C+
Σ
L
*R+S+E2
RR+U,
Fig. 3 illustrates CoMM's training process for the case = 2. The pseudocode for the general case n 2 2 is available in Appendix D. It is worth to notice that the loss terms in Eq. (7) grow linearly with the number of modalities
At inference, no augmentations are applied. The multimodal input (XI,.., "i.....,, Xn) ) is processed through the network to obtain the multimodal feature Zø fX)).. This multimodal representation can then directly transferred to any task either by performing linear probing or by fine-tuning the whole architecture.
4 EXPERIMENTS
We design different sets of experiments to evaluate our model: first, over a controlled environment inspired by the Trifeature dataset (Hermann & Lampinen, 2020), carefully formulate tasks that
5

---

Published as a conference paper at ICLR 2025
Z1
X1
MSK]
Z
X
fo
sim
X"
X = (2)
X2
(MSK'
z"
Ze
Figure 3: COMM training for Two multimodal augmentations are applied to X to obtain X' and X" We also consider the projection operators to get {X,) These n + 2 transformed versions of X are processed by the network fo. trained to maximize the agreement between these + 2 terms using contrastive objectives.
need a specific kind of interaction (uniqueness, redundancy, or synergy) to assess the model's ability to capture them separately. Second, in order to evaluate the capacities of the model to learn multi- modal representations for complex tasks requiring different levels of shared, unique, and synergistic information, we use real-world benchmark datasets with modalities.
Experimental settings. We report mean and standard deviation over 5 runs for all our results (except when using public model weights). All prree-par details can found in Appendix B.
Evaluation. Given pre-trained model f. we perform evaluation through linear probing, i.e., we train linear layer gw (z) = (f) (f fixed) to minimize the classification regression loss (depending on the task). We also report results obtained with fine-tuning, iee after further training in supervised way.
4.1 CONTROLLED EXPERIMENTS ON THE BIMODAL TRIFEATURE DATASET
To evaluate whether CoMM learns redundant, synergistic and unique information for given task, we design synthetic bimodal dataset based on Trifeature (Hermann & Lampinen, 2020). Briefly, we generate Trifeature dataset (as first modality) containing images of one of ten shapes, for of textures and one of ten colors (1 000 combinations in total). We augment each image three times using rotations and translations. Then, we pair each image with a second one (as second modality) from the same Trifeature dataset, allowing us to con- trol the shared, unique and synergistic attributes between the two modalities. As Hermann & Lampinen (2020), we use AlexNet (Krizhevsky et al., 2012) modality-specifit encoder for both modalities all experiments with an embedding dimension 512.
DREE 167 95M
PectarCL
Croes
27
Cimere Silt
OoMM tours
80
Redundancy
Uniquesess
Synergy
Figure 4: Linear probing accuracy of redun- dancy (shape), uniqueness (texture) and synergy (color and texture) on bimodal Trifeature dataset. CoMM is the only model capturing all three task- related interactions between modalities.
Experiment Modeling shared and unique interactions. We choose the shape as shared fea- ture and texture as unique feature for each modality by selecting only paired images with the same shape. Training and test set follow the same distribution, with 100000 and 4 096 images, respectively. We measure the linear probing accuracy of shape (respectively texture) to test whether redundancy (resp. uniqueness) has been captured in the latent representation of the model (chance level=10%).
Experiment Modeling synergy. In the previous experiment, texture and color features are in- dependent between the two modalities. To introduce synergy between these features, we bias the training set by defining a mapping M between the ten textures and ten colors (e.g. stripes-red, dots=green, etc.). Then, seleet randomly 10000 pairs of images that respect this mapping, thus artificially introducing a strong correlation between texture from the first modality and color from the second modality in the training set. The test set is left unchanged from previous Exper-

---

Published as a conference paper at ICLR 2025
Regression
Classification
Model
V&T EE
MIMIC 1
MOSI
UR-FUNNY
MUSTARD ? Average"
Cross! (Radford et ul., 2021)
33.09=3.67
66.7
17 851.8
50.1
53.5g29
54.52
Cross+Self (Yuan et al., 2021)
g.50
65 65.4920.
49 19.0g1.1
59.920.9
53.9g40
57.07
FactorCL (Liang et al., 2023b)
10 10.8220.56
67.3
51.2+16
60.5.0.8
25.80
58.7
CoMM (ours)
4.55
66.420.32
67.5
63. g.0.c
63.9ga.01
65.22
SupCon! (Khosla 2020)
67 67.4
47.2a1.2
50.1
52.7=22
54,35
FactorCL-SUP' (Liung et ul., 2023b)
1.72
76.8 76.8200
69 69.1....
63.5
69.9
69.82
CoMM (fine-tuned)
1.34
68.18 18..1
74.98.0.43
65.96 26.44
70.42.0.16
69.88
Table I: Linear evaluation top-1 accuracy (in %) for classification tasks and MSE (x 100"4 for regression task (V&TEnd Effector) on MultiBench after 100 epochs. 'Results obtained from (Liang al., 2023b). Average is taken over classification results only. Rows in color are supervised.
iment and the task Y is to detect whether a given pair of images respects the mapping M, i.e. Y =
E M) (chance level=50%).
In Fig. 4, we show that cross-modality constraints with InfoNCE (Radford et al., 2021) ("Cross" model) allow to perfectly capture redundant information but completely remove unique and syn- ergistic information,
predicted in Lemma 1. Self-super constraints ("Cross+Self" (Yuan et al., 2021)) capture accurately unique information but fail at preserving syn- ergy, FactorCL, the method most closely related to our work, also performs poorly on synergy. CoMM is the only model learning all three interactions related to the task.
each encoder
4.2 EXPERIMENTS WITH 2 MODALITIES ON WORLD DATASETS
MultiBench. Following (Liang et al., 2023b), we use subset of real-world multimodal datasets from MultiBench (Liang et al., 2021), with different degrees of shared and unique task-relevant in- formation, including: Vision & Touch (Lee al., 2020) a robotics dataset that includes images, force, and proprioception data for end-effector position prediction (regression) and contact prediction (bi- nary classification), MIMIC (Johnson et al., 2016), dataset for mortality and disease prediction from medical records, including tabular data and medical time series from ICU; MOSI (Zadeh et al., 2016). a dataset for sentiment analysis from videos (vision, audio, and language): UR- FUNNY (Hasan et al., 2019), humor detection from videos (vision, audio and language); and MUs- TARD (Castro et al., 2019), a dataset for sarcasm detection from TV shows (vision, audio, and language). For fair comparisons, as previous works, using pre-extracted text, video and audio features for training (Liang et al., 2021; 2023b). More details about these datasets and the data pre-processing can be found in Appendix E. We train CoMM on the same data modalities as FactorCL and use the same backbone networks.
experiments using the same data pre-processing steps
We consider two differrnnt experimental settings. In the self-supervised setting, we perform pre- training and evaluate with linear probing. We consider FactorCL. "Cross" and "Cross+self" methods for comparison. the fine-tuning setting, CoMM is fully fine-tuned in a supervised way after pre- training. We compare it against SupCon and FactorCL-Sup as supervised methods.
Results for these experiments are in Table In the self-supervised experiments with linear probing evaluation, CoMM surpasses FactorCL (second best) by large margins (16.3%, 2.6% and 8.1% of top-1 accuracy on MOSI, UR-FUNNY and MUsTARD, respectively) on three out of four classifi- cation datasets. On MIMIC, margins are considerably narrower, with FactorCI. performing slightly better, and CoMM showing comparable results with "Cross" and "Cross+Sell". In the regression task of VT, CoMM is considerably better than competitors (3 x 10 10-4 lower MSE than second best). In the fine-tuning scenario, we observe the same pattern, with CoMM outperforming competitors on four datasets and FactorCL taking the lead on MIMIC
These experiments not only show CoMM's efficiency at learning multimodal representations, but also exhibit CoMM's versatility to process different data domains (time series, audio, images, text, etc.) and to adapt to diverse backbones.
7

---

Published as a conference paper at ICLR 2025
Multimodal IMDb (MM-IMDb) (Arevalo et al., 2017) is a dataset for movie genre prediction. We consider two modalities: images and text (the movie poster and its plot's description, respectively). Since each movie can
Model
Modulities weighted-ff
macro-fl
SimCLR! (Chen et al., 20200)
40.35 0.23
27. 27.9910.33
51.5
40.8
CLIP (Radfond 2021)
51.0
43.0
V+L
58.9
50.9
BLIP-2 (Laveall 2023)
V+I.
57.4
49.9
SLIP (Mu et al. 2022)
V+L
56.54_0.19
47.3D
be classified into one more genres, it is a multi-label classification task,
CLIP (Radfood 2021)
V+L
54.49_0.19
44.94
CoMM (our, CLIP backbone)
V+L
61.48-0.18
54. 54.63+0.22
with 23 categories. MM-IMDb pro- vides suitable example of life-like multimodal task as the genre predic- tion cannot be performed accurately from the movie poster or the movie plot alone, while results significantly improve by considering both (Arevalo et al., 2017), suggesting that syn- ergistic interactions are needed to fulfill this task. We compare CoMM's performance on MM-IMDb against important baselines under two different settings. First, the self-supervised setting, we consider CLIP (Radford et al., 2021). representing "Cross" methods, SLIP (Mu et al., 2022) representing "ells methods, SimCLR (Chen al., 2020a) for unimodal self-supervised methods, BLIP-2 (Li et al., 2023) as a recent powerful vision and language model, as baselines. All models were trained on unlabeled MM-IMDb. For CLIP, we also report results with the publicly released weights without further training. CoMM is initialized with pre-trained weights from CLIP and BLIP-2. Second, in the fine-tuning setting, we compare CoMM fine-tuned with state-of-the-an fully supervised baselines. We also include results for LLaVA-NeXT (Li et al., 2024) representing new vision-and- and-language generative models, however these scores are not fully comparable to ours, since the model might have seen the IMDb database during training. See Appendix B for implementation details.
CoMM (ous, BLIP 2 backbonel
V+L
65.01
58. 44..044
MFAS (PicezRRia et 2019)
V+L
62.50
55.6
ReFNet (Sunkornn et al, 2022)
V+I.
56.7
CoMM (oum, IP baothome)
V+L
64.90-0.21
58.97
CoMM tours, IILIP22 backbone)
V+L
67. 27.39
62. 200.2
VAN 2024)
V+L
64.28
56.51
Table 2: I Linear evaluation F1-score (weighted and macro) (in %) on MM-IMDb after 70 epochs. f indicates further training on unlabeled data. means supervised fine-tuning. Rows in color are supervised.
Table 2 shows that CoMM outperforms all models in both settings. In the self-supervised setting, CoMM has a margin of 7.5% and 5.8% of macro and weighted Fl-scores, respectively, with the second-best method. It interesting to observe that CLIP performs better with the publicly released weights (probably because of the large-scale and diversity of the data it was trained on), than with further training on MM-IMDb. This result suggests that the genre movie prediction does not benefit from learning redundant information. Including uniqueness from the image modality allows for some improvement (SLIP). In the fine-tuning setting, CoMM fine-tuned outperforms existing fully supervised baselines, even if MFAS has been designed to search for the best fusion strategy, and vision-language generative models (LLAVA-NeXT).
4.3 EXPERIMENTS WITH 3 MODALITIES ON REAL-WORLD DATASETS
We test CoMM's abilities to learn multimodal interactions beyond 2 modalities. We perform experiments on two large datasets including tri- modal data from MultiBench: Vision & Touch (contact prediction task) and UR-FUNNY.
Model
#Mod.
V&T CP UR-FUNNY
Cross
2
84.4
50.1
Cross+Self
2
86.8
59,9
CnMM (ours)
2
8
63.1
CMC (Tian et al., 2020a)
3
94,1
59.2
In Table 3. we compare CoMM trained on the three modalities in a self-supervised way against CMC (Tian et al., 2020a). We also com- pare with bi-modal models: CoMM, Cross and Celss+Self trained on image and propriocep- tion data for Vision& Touch and image and text data for UR-FUNNY.
CoMM (ours)
3
94.2
64.6
Table 3: Linear evaluation top-1 accuracy (%) on Shiion&Tochh and UR-FUNNY.
First, we observe a consistent improvement (+6.1% on V&T, +1.5% on UR-FUNNY) when adding third modality with CoMM (compared to only using two), which demonstrates its versatility. Second, we improve the state-of-the -ftte-att (+0.1% and +5.4% on V&T and UR-FUNNY, respectively).
art for SSL methods on datasets with more than two modalities
8

---

Published as a conference paper at ICLR 2025
Redundancy
Uniqueness
Synergy
Average
Accuracy (%)
80-
60
Σ 4
40-
Coan
2000
7990 12200 19393
5i0e 7900 1993 12300
2500
1933 Iteration
13500
2900 7993 19393 33900
Iteration
Figure 5: Linear probing accuracy of redundancy R. uniqueness U = == - U4 and synergy S on bimodal Trifeature when optimizing each term separately LCOMM- Minimizing L, allows to learn U, and R. approximating [(x;; Y) for i n). Optimizing L = -- i('',, Z") learn R. Uj and S. CoMM quickly captures all information.
allows to slowly
5 ABLATION STUDIES
We test three main components of our framework (the loss, fusion module and augmentation strat- egy) against important control baselines on bimodal Trifeature dataset (see Section 4.1).
Loss function. First, we check our claim that optimizing both L and Σ; C. -as in Eq. (7)- required to accurately capture uniqueness, synergy, and redundancy. In Fig. 5, we show that minimizing Σx=1 Li improves redundancy and uniqueness, as guaranteed by our Lemma 3, but fails for synergy. Conversely, minimizing C allows one to learn all information terms but very slowly. In particular, for synergy, argue this is because the model has to learn modality-specifie features first (phase 1) before learning their interactions (phase 2). The former is learned through T(4, 7' T(Z,, Z") while the latter is captured with T(Z', Z"). Hence, LCMMM speeds up phase I and can learn synergy more efficiently in phase 2.
Fusion module. We compare our attention-based latent fusion module with shallow linear fusion. We project modality-specific representations to the common latent space using only linear layers and remove latent con- verters. Table 4 shows that synergy not captured with linear fusion. This is expected as the XOR gate (typical example of synergistic interactions) cannot be approximated by linear function. Additionally, uniqueness accuracy is also degraded compared to CoMM (-6%), suggesting that only modeling linear interactions limits the model's representation power.
Fasion
Redundoncy
Uniqueness
Synergy
Average
Concat Linear
99.71 99.71+0.06
81. 41.49.2.85
50.0=a.0
77.07
CoMM
99.92
87 83 21.55
71.87
86.83
Table 4: Linear fusion module versus attention- based fu- sion with latent converters (CoMM). Non-linearitic are re- quired to learn synergistic interactions between modalities.
Data augmentation. Finally, we show in Table 5 that strong data augmentation is crucial for learning multi-modal inter- actions, in line with the liter- ature on unimodal contrastive methods. Contrary Fac- torCL (Liang et al., 2023b), ap- plying strong augmentation on both modalities is beneficial for
Augmentarions
R
U2
Average
Modality
Modality 2
[AlI)
98.72
79. 29
46.44_a74
50.0g00
Ø
(AlI)
99 99.58
53 33.8918..
26.04
50.0100
72.37
(crop) (All)
88.79 88.79
25 10
84.00_14
50.0g00
02.11
(Al) (crop)
83
23.22
228
21.74_116
50.0100
61.36
CoMM
99 92 Loos 84335 22.77 91.18 10.97
71.87
86 66.83
Table 5: Effect of data augmentation linear probing accuracy (%) of multimodal interactions. All refers to SimCLR augmenta- tions (Chen et al., 2020a). CoMM uses All for both modalities.
CoMM and we do not require task-dependent augmentations, highlighting the versatility of our framework.
RELATED WORK
Multimodal learning refers to methods that connect and integrate information from multiple sources of data (Baltrušaitis et al., 2018; Akkus et al., 2023). Early works focused training

---

Published as a conference paper at ICLR 2025
separate encoders and studied different fusion mechanisms to blend the information from different inputs (Zeng et al., 2007; Pérez-Rúa et al., 2019). With the development of transformers and ViTs, the focus has shifted towards training a unique architecture and designing specific tasks and loss functions to integrate multimodal interactions (Xu et al., 2023; Lu et al., 2019; Sun et al., 2019; Chen al., 2020b; Lu al., 2020). Today multimodal learning includes different research lines, from generative multimodal learning (Suzuki & Matsuo, 2022; Ramesh et al., 2021; Saharia et al., 2022; Wang et al., 2022b; Alayrac et al., 2022) to multimodal representation learning (Zong al., 2024; Tian et al., 2020a). CoMM belongs to the latter category.
Self-supervised multimodal representation learning. Self-supervised learning aims to learn general representations through supervisory signals from the data itself (known as the pretext task) (Balestriero et al., 2023). In the multimodal context (Zong et al., 2024), self-supervised meth- ods can be grouped according to their pretext task. Clustering-basee 2020; Asano et al., 2020), where cluster assignments are used as pseudo-labels to train the model and different modalities can be used supervisory signals to each other; masking modeling meth- ods (Mizrahi et al., 2023; Bachmann et al., 2022; Lu et al., 2023) that reconstruct pieces of informa- tion that have been masked from input data. In the multimodal case, masked modelling is used in a cross-modal way, by predicting missing information conditioned in other modalities. Contrastive methods (Radford et al., 2021; Jia et al., 2021) in the multimodal context have been mostly used on matched data from different modalities to obtain aligned-yet distinct-represenattios In this work, instead of representing each modality separately, we take different perspective for contrastive methods by learning a single multimodal representation of the inputs.
methods (Alwassel et al.,
Multimodal contrastive learning. Current methods in contrastive learning are inspired by the idea of multiview learning (Li et al., 2018) and have shown remarkable results in representation learning in unimodal tasks (Chen et al., 2020a; He al., 2020; Tian et al., 2020a; Oord al., 2018). The natural extension of these approaches to multimodal settings is to optimize a mooss-modal contrastive loss (Radford et al., 2021; Alayrac et al., 2020; Jia et al., 2021). Other works have gone even further by introducing cross- and intra-modal contrastive objectives (Mu et al., 2022; Jain et al., 2021; Mi et al., 2024), or by adding other intra-modality regularization terms (Wang et al., 2022a; Kim et al., 2022). However, these approaches are designed to learn redundant information, neglecting the contributions of uniqueness or synergy. Recently, FactorCL (Liang et al., 2023b) has proposed solution to model shared and unique task-relevant information explicitly. Yet, the method relies heavily on assumptions that are hard to meet in practice; it proposes a factorized approximation of multimodal interactions that is prone cumulative errors and does not model synergistic information. Alternatively, we propose CoMM, a contrastive multimodal approach that leverages multimodal augmentations with modular architecture optimized through information theory-grounde losses to capture unique, redundant and synergistic interactions.
7 CONCLUSIONS
Multisensory integration is at the core of human perception, allowing us to build coherent represen- tations of our environment. In this paper, we introduce CoMM, a contrastive multimodal method that enables the integration of multiple modalities in single multimodal representation space. Un- like existing multimodal contrastive models, CoMM designed to learn multimodal interactions beyond redundancy, through Partial Information Decomposition theory. Our controlled experiments on the bimodal Trifeature dataset demonstrate that CoMM successfully learns redundant, unique and synergistic information. In real-life multimodal datasets from Multibench with two and three input modalities, CoMM outperforms cxisting methods by large margins in almost every case, showing the efficiency and versatility of CoMM to handle data across diverse domains (robotics, healthcare, affective computing, multimedia) and data structures (time series, image, text, audio, tabular).
This work opens large avenues for future research multimodal representation learning, in partic- ular for crafting label-preserving multimodal augmentations not limited to unimodal augmentations. Limitations and perspectives for future research are further discussed in Appendix A. Overall, the simplicity and versatility of CoMM's design make it good candidate to learn deep representations of several modalities across domains. offers promises in better solving real-world problems rang- ing from neuroscience (Preti& & Van De Ville, 2019) and medical imaging (Bochm et al., 2022) to remote sensing (Gómez- Chova et al., 2015).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Jonathan Sauder, Valentin Gabeff and Valérie Zermatten for providing helpful feedback on earlier versions of this work. JCN acknowledges the support from EPFL Science Seed Fund. BD acknowledges the support from the PHRT project number 643.
ETHICS STATEMENT
We acknowledge that there exist potential privacy risks when using human behavior medical data. However, we minimize these potential risks by using publicly available datasets that have been carefully collected, respecting participants' consent, de-identifying medical data and anonymizing video data. this work, we have followed best practices maintaining the privacy and safety of these datasets.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work. we have used publicly available datasets from Multi- Bench (Liang et al., 2021) (MIMIC, MOSI, UR-FUNNY, MUsTARD and Vision&Touch); MM-
IMDb (Arevalo et al., 2017), and the synthetic Trifeatures dataset (Hermann & Lampinen, 2020). Details about these datasets and pre-processing steps can be found in Appendix E. Appendix B thoroughly describes neural network architectures, data augmentation strategies, and experimental settings used in all our experiments. We include in Appendix D pseudo-code for implementing CoMM's training, and we released our code in this Github repository. We also provide details on processing times and model complexity compared to other multimodal models in Appendix F. Finally, proofs for our theoretical developments can be found in Appendix G.
REFERENCES
Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen lauch-Walsed Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schneider, et al. Multimodal 2023.
deep learning. arXiv preprint arXiv:2301.0488
Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised mul- timodal versatile networks. In Advances Neural Information Processing Systems (NeurIPS),
volume 33, pp. 25-37, 2020.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 23716-23736, 2022.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems (NeurIPS), 33:9758-9770 2020.
John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multi- modal units for information fusion. In International Conference Learning Representations Workshops (ICLR-W), 2017.
Yuki Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self suffvvuperviioon Advances in Neural Information Pro- cessing Systems (NeurIPS), 33:4660-4671, 2020.
Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb Saddik, and Mohan S Kankanhalli. Multi- modal fusion for multimedia analysis: survey. Multimedia systems, 16:345-379 2010.
Bernard Baars. A cognitive theory of consciousness. Cambridge University Press, 1988.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi- task masked autoencoders. In European Conference on Computer Vision (ECCV), pp. 348-367, 2022.
11

---

Published as a conference paper at ICLR 2025
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. cookbook of self- supervised learning, arXiv preprint arXiv:2304.122100 2023.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Phillipe Moreney. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2): 423 443, 2018.
Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 16(4):2161-2183, 2014.
Kevin M Boehm, Emily Aherne, Lora Ellenson, Ines Nikolovski, Mohammed Alghamdi, Ignacio Vázquez-Garehe Dmitriy Zamarin, Kara Long Roche, Ying Liu, Druv Patel, et al. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nature Cancer, 3(6):723-733) 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sareasm detection (an obviously perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
pp.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597-1607, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), pp. 104 120. 2020b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume I (Long and Short Papers), pp. 4171-415 2019.
Pre-training of deep
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Mohamed Banani, Karan Desai, and Justin Johnson. Learning visual representations via tnnguage-guided sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19208-19220, 2023.
Luis Somez-Chova Devis Tuia, Gabriele Moser, and Gustau Camps-Vall Multimodal classifica- tion of remote sensing images: A review and future directions. Proceedings of the IEEE, 103(9): 1560-1584, 2015.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems (NeurIPS), 34: 11932-18943, 2021.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 478(2266):202100668, 2022.
GroupLeens research. MovieLens dataset, 2015. URL wwt.hhtpsse.e httpp::://gooolllss.oo///aattatt moviell eenss......
12

---

Published as a conference paper at ICLR 2025
Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Philippe Morency, et al. UR UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLLP), pp. 2046-2056, 2019.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9729 9738, 2020.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS), 33:9995-10006, 2020.
Pengbo Hu, Xingyu Li, and Yi Zhou. SHAPE: A unified approach to evaluate the contribution and cooperation of individual modalities. Proceedings of the International Joint Conference on Artificial Intelligence (ICCAI), pp. 3064-3070, 2022.
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jía, Yinfei Yang. and Jason Baldridge. MURAL: Multimodal, multitask representations actoss languages. In Find- ings of the Association for computational Linguistics: Empirical Methods Natural Language 2021.
Processing (EMNLP), pp.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), pp. 4904- 4916, 2021.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger Mark. MIMIC-III, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 33: 1811116611877 2020.
Byoungjip Kim, Sungik Choi, Dasol Hwang, Moontae Lee, and Honglak Lee. Transferring pre- trained multimodal representations with cross cross-modal similarity matching. Advances in Neural Information Processing Systems (NeurIPS), 35:30826-30839, 2022.
Wessel Kraaij, Thomas Hain, Mike Lincoln, and Wilfried Post. The AMI meeting corpus. In Proceedings of the International Conference on Methods and Techniques in Behavioral Research, 2005.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. ImageNet classification with deep con- volutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 25: 1097-1105, 2012.
Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):582- 596, 2020.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. LexVA-NeXT: Stronger LLMs supercharge multimodal capabilities the wild, May 2024. URL hetps://llava-veee wighraa-iiiigittt io/blog/
1mm//
13

---

Published as a conference paper at ICLR 2025
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- tinuage-imaage pre-training with frozen image encoders and large language models. In International Conference Machine Learning (ICML), pp. 19730-19742, 2023.
Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation learning. IEEE Transactions Knowledge and Data Engineering,
2018.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. MultiBench: Multiscale benchmarks for multimodal represen- tation learning. In Neural Information Processing Systems (NeurIPS) - Track on Datasets and Benchmarks, volume 1, 2021.
Paul Pu Liang. Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood. Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling feature interactions: An information decomposition framework. Advances in Neural Information Processing Systems (NeurIPS), 36:27351-27735 2023a.
Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis- Doiis-Phllippe Morency, and Ruslan Salakhut- dinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems (NeurIPS), 36:32971-32994 2023b.
Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal learning without labeled multimodal data: Guarantees and applications. In International Conference on Learning Repre- sentations (ICLR), 2024.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolin- guistic representations for vision-and lannuuugeee tasks. Advances in Neural Information Processing systems (NeurIPS), 32:13-23, 2019.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10437-10446, 2020.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-10: A unified model for vision, language, and multi-modal tasks. In International Con- ference Learning Representations (ICLR), 2023.
Harry McGurk and John MacDonald. Hearing lips and seeing voices. Nature, 264(5588),466-74,, 1976.
Li Mi, Chang Xu, Javiera Castillo-Navarro, Syrielle Montariol, Wen Yang, Antoine Bosselut, and Devis Tuia. ConGeo: Robust cross view geo-localization across ground view variations. arXiv preprint arXiv:2403. 13965, 2024.
David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4M: Massively multimodal masked modeling. Advances in Neural Information Processing Systems (NeurIPS), 36:58363-58408. 2023.
Gil Morrot, Frédéric Brochet, and Denis Dubourdieu. The color of odors. Brain and language, 79 (2):309-320, 2001.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. SLIP: Self-supervision meets language-image pre-training. In European Conference Computer Vision (ECCV), pp. 529- 544, 2022.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Mul- timodal deep learning. In International Conference on Machine Learning (ICML), pp. 689-696, 2011.
14

---

Published as a conference paper at ICLR 2025
Daisuke Niizumi, Daiki Takeuchi, Yasumori Ohishi, Noboru Harada, and Kunio Kashino. BYOL for general-purpose audio representation. In International Joins Conference on Neural Networks (IJCNN), pp. 1-8, 2021.
audio: superviuee learning for general purpose
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.033 2018.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby. Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without super- vision. Transactions on Machine Learning Research, 2024.
Juan-Manuel Pérez-Ráa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie. MFAS: Multimodal fusion architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6966-6975, 2019.
Maria Giulia Preti and Dimitri Van De Ville. Decoupling of brain function from structure reveals regional behavioral specialization in humans. Nature Communications, 10(1):4747, 2019.
Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models on large healthcare datasets. Journal of biomedical informatics, 12-134, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- 2018.
standing by generative pre-training. https://opemai.c http://oppenai.co//esseeeet
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning (ICML), pp. 77638763, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot -inatto- generation. In International Conference on Machine Learning (ICML), pp. 8821 1831,, 2021.
Nils Reimers and Iryna Gurevych. Contence-BERT Sentence embeddings using siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992, 2019.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36479-36494 2022.
Sethuraman Sankaran, David Yang, and Ser-Nam Lim. Refining multimodal representations using modality-centrie self-supervised module. http:://ppeeeeiiieeeeemmmmmmmmdd
2022.
Cees GM Snoek and Marcel Worring. Multimodal video indexing: A review of the state-of-the-arr. Multimedia Tools and Applications, 25:5-35, 2005.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A joint model for video and language representation learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pp. 7464-7473, 2019.
Masahiro Suzuki and Yutaka Matsuo. A survey of multimodal deep generative models. Advanced Robotics, 36(5-6):261-778 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference Computer Vision (ECCV), pp. 776-794, 2020a.
15

---

Published as a conference paper at ICLR 2025
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing sys- tems (NeurIPS), 33:6827-68399 2020b.
Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient represen- tation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16041-16050, 2022a.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. Trans- actions on Machine Leurning Research, 2022b.
Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arxiv: 1004.251,, 2010.
Andrew T Woods, E Poliakoff, DM Lloyd, J Kuenzel, R Hodson, H Gonda, Batchelor, Garmt B Dijksterhuis, and A Thomas. Effect of background noise on food perception. Food Quality and Preference, 22(1):42 47, 2011.
Peng Xu, Xiatian Zhu, and David Clifton. Multimodal learning with transformers: A survey.
IEEE Transactions Pattern Analysis and Machine Intelligence, 45(10): 2023.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6995- 7004, 2021.
Ben P Yuhas, Moise H Goldstein, and Terrence J Sejnowski. Integration of acoustic and visual 1989.
speech signals using neural networks. IFFE Communications Magazine,
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment in- tensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6): 82- 82-88,2200 2016.
Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas Huang. A survey of affect recog- nition methods: audio, visual and spontaneous expressions. In Proceedings of the International Conference on Multimodal Interfaces (ICMI), pp. 126-133, 2007.
Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-superviser multimodal learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-20, 2024.
16

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE RESEARCH
CoMM's theoretical analysis for more than two modalities is still unclear since PID theory is currently limited to π. 2 modalities (see Section 2 in the main paper). While uniqueness can be easily defined for any 2 2, the number of interactions between modalities (e.g. re- dundancy and synergy) grows exponentially with n, making the analysis harder. Nonetheless, we should recall that CoMM performs empirically very well even for n > (see Section 4.3), and the number of loss terms increases only linearly with n (see Eq. (7)). Another limitation is the additional computational cost associated with adding modality-speciff encoders. A simple workaround is to use large pretrained (frozen) encoders (e.g. from CLIP (Radford et al., 2021) for vision and text or DINOv2 (Oquab et al., 2024) for vision) and to only tune a lightweight fusion transformer in COMM, allowing much faster training.
CoMM's computational cost for data augmentation is higher than in cross moddaliodsss frame- works (such CLIP). This is because for each data tuple X XX,, Xn). CoMM needs to compute X = t(X), while cross-modality methods feed X directly into the neu- ral network. A possible solution would be the implementation of a momentum encoder and queue method, as in MoCo (He et al., 2020).
Interpretability.. Our experiments the Trifeature dataset (see Section 4.1 in the main paper) show that CoMM can efficiently learn unique, redundant and synergistic information. However, it seems difficult to disentangle the contributions of these three interactions in the representation space. Disentanglement might be one direction for future work. However, other approaches to measure such quantities (given dataset and a task) are emerging (Hu al., 2022: Liang et al., 2023a; 2024). Another interesting approach would be to use modality masking (already implemented and handled by CoMM) to analyze the contribution of each modality individually versus collectively.
We believe the above limitations are directions that were outside the scope of the current manuscript; however, they are exciting avenues for future research in multimodal representation learning.
B IMPLEMENTATION DETAILS
For all experiments with COMM, we use an attention-based fusion module that takes input sequence of embeddings. We use 1-layer Transformer with 8 heads that applies self-attention all inputs and we add [CLS] learnable embedding at the beginning of the sequence. The embedding size depends each dataset, detailed below.
B.1 ENCODER ARCHITECTURES BY DATASET
Trifeature composed of two visual modalities, so we use an AlexNet encoder for both modal- ities with 51 2-d embedding space. For CoMM, we remove the last average pooling layer and we apply a linear patch embedding layer (Dosovitskiy et al., 2021) to the 6 feature maps as latent converter. We then add fixed 2D sine-cosine positional embeddings (Dosovitskiy et al., 2021). For Cross, we use a linear projector for each encoder (512-d output space) and we opti- mize the CLIP loss (Radford et al., 2021). For Cross+eelf, we apply 3-layers MLP projection head to each encoder to optimize the SSL objectives as in SLIP (Mu et al., 2022) (1024-d hid- den layers and 256-d output space) and use the same projectors and CLIP objective for Cross.
MIMIC contains tabular and time-series data, seen as two modalities. We use a 2-layers MLP (10-d hidden dimension, 10-d output) as tabular encoder and a GRU (512-d hidden dimension) as time-series encoder (similarly to FactorCL (Liang et al., 2023b)). For tabular data, we use feature tokenizer (Gorishniy al., 2021) as latent converter with 512-d embedding space and converter for time-series.
MOSI. UR-FUNNY and MUSTARD contain visual and textual modalities extracted from videos. Similarly to FactorCL (Liang et al., 2023b), we use a for each modality with 40-d embedding space. We do not use latent converters in this case.
5-head Transformer with 5 layers
17

---

Published as a conference paper at ICLR 2025
MM-IMDb also contains visual and textual modalities but in their raw format. We ViT- B/32 (Dosovitskiy et al., 2021) image encoder pre-trained with CLIP (Radford et al., 2021) and Sentence- BERT multilinguul text encoder' pre-trained with CLIP and distilled with Sentence- BERT (Reimers & Gurevych, 2019). For CoMM, we consider the token embeddings given by the image and text encoders (frozen) and we do not latent converters. For CLIP (Rad- ford et al., 2021), we fine-tune the pre-trained encoders with their original architecture. For SLIP (Mu et al., 2022), we use the same pre-trained encoders CLIP and we add 3-layers visual projection MLP (4096-d hidden layers and 256-d output space) to compute the SSL objectives.
& Touch has visual, force-torque and robot proprioception modalities available. For the binary contact prediction task, we only use visual and proprioception data for the experiments with two modalities and we encode images with a ResNet18 (He et al., 2016) (512-d output space) and proprioception data with 5-layer MLP (512-d output space), as in the original paper (Lee et al., 2020). In the experiments with modalities, force-torque data are encoded with 5-layer causal convolutions network (512-d output space). For CoMM, we remove the last average pooling layer and apply a patch embedding layer (Dosovitskiy et al., 2021) to the 4 x 4 feature maps as latent converter. We consider the 1D feature vector of proprioception data as a I-length sequence in the fusion module. For the end-effector regression task, we use visual and force-torque modalities. We use ResNet18 as image encoder (128-d output space) and a 5-layer causal convolutions network (128-d output space) as force encoder (Lee et al., 2020). For CoMM, add the same latent converter for images in the previous task and add a feature tokenizer (Gorishniy et al., 2021) for force-torque embeddings.
B.2 DATA AUGMENTATION BY MODALITY
For raw images (in Trifeature, MM-IMDb and Touch), we use the default SimCLR augmen- tations (Chen et al., 2020a), which include RandomResizedCrop. ColorJitter, RandonGrayscale, GaussianBlur and Ramdomfforizomulllip (from the PyTorch library).
For tabular data (in MIMIC and Vision& Vision&Touch) we add a random Gaussian noise to each component (assuming they are all continuous).
For time-series data (either extracted from videos in MOSI, UR-FUNNY, MuSTARD, from health Touch), we apply a random composition of Gaussian noise and random dropping between 0 and 80% of the sequence. We have compared several other strategies for this modality and we present the results in Appendix C.
recordings as in MIMIC or from force-torque readings as in Viscon&Touch,
For raw text (in MM-IMDb), randomly mask 15% of input tokens by using special [MASK] token as in BERT (Devlin et al., 2019).
B.3 LATENT CONVERTERS BY ENCODER ARCHITECTURE
Transformer and GRU: the latent converter is the identity since the Transformer and GRU already output a sequence of embeddings;
CNN: the latent converter is a patch embedding projection module originally defined in ViT (Dosovitskiy et al., 2021) that we apply to the features maps of the CNN; MLP: the latent converter is a feature tokenizer originally defined in (Gorishniy et al., 2021) for tabular data. The feature vector h; is transformed into sequential embeddings by applying feature-wise multiplication with learnable matrix and we add a bias term. For proprioception and force-torque data simply consider them as 1- 1-length sequence in the fusion module.
B.4 EXPERIMENTAL SETTINGS
We use AdamW optimizer (Loshchilov & Hutter, 2019) in all experiments and a learning rate a = 3 10 for Trifeature (with weight decay 10 10-4) 4), and MuSTARD (with weight decay 10 30 and = 10 for MM-IMDb and Vision& Touch (with weight decay 10-2). For MM-IMDb, we also use a cosine scheduler with final value 10-6 and a warmup over 10 epochs. All models were optimized during 100 epochs. The critic in the InfoNCE
10 for MIMIC, MOSI, UR-FUNNY
co/ www.eeceeeeeeeee weweenteeneetranaaeeeeee c/cl www.ccpppii---eee 32 mul 1iliingunll---n lingual-vl
18

---

Published as a conference paper at ICLR 2025
losses in CCMMM Eq. (7) is implemented 3-layer MLP (512-d hidden layers and 256-d output space), similarly to the projection head in SimCLR (Chen et al., 2020a). All experiments ran on a single V100 GPU with 32GB of memory.
the SupCon loss (Khosla et al., 2020) to fine-tune CoMM (with no additional parameters). the case of multi-label classifi- cation with MM-IMDb, we use a linear head on top of CoMM and optimize cross-entropy loss for each label. For the regression task on Vision& Touch, we also use linear head and we optimize the MSE loss. We systematically use early-stopping according the validation accuracy in order to prevent over-fitting on the downstream tasks.
Fine-tuning of CoMM. For all downstream classification tasks,
B.5 EXPERIMENTAL SETTINGS THE BIMODAL TRIFEATURE DATASET
To generate our trifeature dataset, we considered the 000 combinations of the three features ex- isting in the original dataset (see Appendix E) and split them into 800 combinations for training and 200 for evaluation. To have more variety in the training set for training, each combination was generated 3 times (the shape and the texture were randomly rotated), obtaining training split of 2 400 images.
The bimodal Trifeature dataset used in our experiments was built by considering the trifeature dataset twice (as two separate modalities) and building pairs from these two dataset copies. In total, we get 5 760 000 pairs (2 400 x 2 4000 available for training, and 000 x200 200) available for evalua- tion.
To create a controlled environment for evaluation multimodal interaction learning, we needed to carefully design tasks where the dominant interaction clearly defined.
To measure uniqueness Un (resp. U2). given pair of trifeature images, the task to predict the texture of the first (resp. the second) image. The task is then 10-class classification problem and chance level is at 10%.
2. To measure redundancy R, given pair of trifeature images with the same shape (but differ- ent color and texture), the task is to predict the shape of the pair (therefore, the redundant information that can be extracted either from the first or the second image). The task is then a 10-class classification problem and chance level is 10%.
3.
To measure synergy S. the definition of the task was more subtle as it should require infor- mation from both modalities simultancously and should not be possible to perform it from one of the images alone. To achieve this, we defined a mapping M between the ten tex- tures and the ten colors (e.g. stripes=red, dots=green, etc.). Then, given pair of trifeature images, the task to predict whether the pair satisfies the mapping or not. The task is then binary classification problem and chance level is at 50%.
To evaluate these tasks, we built two versions of the bimodal trifeature dataset:
(i) For uniqueness and redundancy, we considered 00000 image pairs (out of the 5 760 000 pairs) for training and 4 096 for testing, that have the same shape (to measure redundancy) and different texture (to measure uniqueness).
(ii) For synergy, we considered 10000 image pairs that respect the mapping M and used the same test set before (4 096 image pairs).
B.6 ADDITIONAL DETAILS FOR LLAVA-NEXT EVALUATION
We evaluate LLaVA-NeXT (Li et al., 2024) on MM-IMDb based on its answer to the following prompt:
"From the following plot: {...} and this poster image, give me all the movie genres belongs among the following list: {..}. Give the answer list." We formulate it as close question with limited number of answers to be closer to the linear probing setting for representation learning models.
19

---

Published as a conference paper at ICLR 2025
UR-FUNNY
MUsTARD
crop
Grup
65
dirop
drop
64.0
13.5
65.3
noise
62.4
633
677
62.7
noise
036
53.0 05.2
04
multi crop
62.5
multi crop
crrop
multi drop
Average
crop
drop
notso
mult crop
multi drop
crop
drop
noise
multi crop multi drrp
crop C3.9
65.0
drop
05.3
65.2
01.1
64.7
04.5
augmentation
2nd augmentation
noise
65.3
64.9
65.1
multi crop
64.7
64.7
64.0
MOSI
MIMIC
multi ddoopdrpp
60.2 64.7
64.9
05.9
66.1
10.6
crop drop
noise
multi crop multi drop
drop
64
07.5
65.0
660
86
drop
66.
66.4
M
66.4
noise-
01.5
07.0
66.7
noise
multi crop
multi crop-
mulli drop-
64.7
multi drop
66.0
crop
drop
noiso
mutti ctop
multi drop
crop
drop
noise
multi crop
multi drop
augmectation
augmentation
Figure 6: Linear evaluation accuracy (%) on four datasets from MultiBench (Liang al., 2021) when applying individual or composing data augmentations on the modalities. Diagonal entries cor- respond to one augmentation applied all modalities while off-diagonal entries are a composition of two augmentations, Average is the average of all four matrices. Results are averaged over 5 inde- pendent runs (with different seeds) in each cell.
C ADDITIONAL EXPERIMENTS
In this section we present additional analysis of CoMM, including: benchmark of several aug- mentation strategies for time-series data, ablation studies on our loss function on a real dataset, and supplementary results on the trifeatures dataset.
C.I DATA AUGMENTATION STRATEGY FOR TIME-SERIES
For time-series data like in MIMIC, MOSI, UR-FUNNY and MuSTARD, there is no consensus with respect to the best data augmentation strategy apply for self-supervised contrastive learning. We benchmark several common augmentations for time-series data including random crop between 8% and 100% of the signal, random drop between 0% and 80% of the signal and adding Gaussian noise. We also designed two new multimodal augmentations: multi-crop (resp. multi-drop) consisting of cropping (resp. dropping) a signal across multiple modalities for time-aligned data (introducing consistency in the preserved multimodal signal).
We tested these augmentations along with their composition four datasets from Multi- Bench (Liang et al., 2021) and we plot the results in Fig. 6. Overall, we find that composing Gaussian noise and random drop results in the best performances across datasets and tasks. This our default augmentations strategy in our main experiments. Our proposed multi-drop and multi- crop augmentations can provide better results in some cases (for MIMIC and MuSTARD), but we select the same default augmentations for consistency across all datasets.
C.2 ABLATION STUDIES ON A REAL DATASET
We performed the loss ablation study on MM-IMDb. Results follow the same tendency as in the Tri- Tri-features dataset (Fig. 5 main paper).
20

---

Published as a conference paper at ICLR 2025
Loss
weighted-fl
macro-fl
Σ, Ci
60.71
53.35
20.00
47.13
LCAMM = C E, C.
61.48 10.18
54.63.0.2
Table 6: Ablation study of loss function contribu- tions on MM-IMDb. LCOMM allows to better capture multimodal interactions than each term separately.
C.3 DESIGN CHOICES TRIFEATURES EXPERIMENTS
Hermann & Lampinen (2020) experimented with Trifeatures using AlexNet and ResNet-50 back- bones. Both architectures showed comparable results. Therefore, we chose to use AlexNet in our ex- periments. For completeness, we ran the same experiments on ResNet-50, which show that CoMM the only model to learn all interactions, regardless of the architecture.
Redundancy
Uniqueness
Synergy
Model
Cross
C+S
CoMM
Cross
C+S
CoMM
Cross
C+S
CoMM
AlexNet
100.0+0.02
99.7
99.9
11.6=0.9
86.9
87.8+16
50.0
50.0
71.942.0 50.0+0.0 75.0
ResNet 100.040.04 99.940.04 99.9
6.5
96.2
96.3
50.0
Table 7: Linear probing accuracy of redundancy (shape), uniqueness (texture) and synergy (color and texture) on bimodal Trifeature dataset, with different backbone encoders. These results are complementary Fig.
C.4 ON THE FEASIBILITY OF ASSUMPTION 1
62
Random Level
80
CiO 0.6 CT0, 0.4
Accuracy (%)
Weighted F1 (%)
60
CO.1 C0.3..0000002222
b0,0s 1]
Co.5-M0.2
70
C 1
Cc.-MM0.2
58
C/O,0.0
Ci0 1]
C0.8-M0.8
60
56
C/0,0.2
C0.9-M0.9
110.55,1
50
54
3.6
3.7
3.8
3.9
4.0
3.8
3.9
4.0
4.1
4.2
INCE X')
INCE (X, (X,X')
(a) Trifeature dataset (synergy)
(b)
dataset
Figure 7: Downstream task performance the cropping augmentation strength decreases (for Trifeatures) or when both cropping and masking augmentation strength for image and text re- spectively decreases. INCE(X,X)) measured using the trained encoder fo in Eq. (6). As INCE(X,XX) increases, the downstream performance first increases and then decreases, revealing "sweet spot". suggests the existence of an optimal augmentation policy that preserves task- relevant information while removing nuisance features from the multimodal input data, giving more credit to our Assumption
Assumption states the existence of an optimal augmentation policy T such that I(X, xX)) = T(X,Y) Y) for any T", given task Y. Hence, the set of optimal multimodal transformations
21

---

Published as a conference paper at ICLR 2025
is task-dependent. However, in practice, we have seen that the augmentations applied in our periments are general enough to obtain good results in a variety of datasets (7 datasets, 10 tasks, diverse data modalities). These augmentations were chosen according to the findings in unimodal self-supervised learning [2, 3], and by a comprehensive ablation study on the best augmentation strategies applicable time-series in Appendix C.I.
Theoretically, the set of multimodal augmentations need to be large enough such that no information about the task Y is lost, but small enough to extract this information only. Tian al. (2020b) have referred to this observation as the InfoMin principle. In Section 5, our data augmentation ablation study shows that if the set of augmentations is not large enough, then synergy cannot be learnt (the performance is always random chance). In order to further explore the feasibility of Assumption and inspired by the strategy developed by Tian et al. (2020b), we evaluate CoMM by progressively increasing the strength of the data augmentations applied. We use random crop as the augmentation to control in the vision domain, mainly for two reasons: first, it is intuitive that by decreasing the level of cropping (keeping less information about the images), we are destroying the task-relevant information; and second, because it has been empirically demonstrated that cropping is a critical transformation in self-supervised learning for vision Chen al. (2020a). For the text domain, use masking as the augmentation to control. More specifically, on Trifeatures we randomly crop the original image in the first modality from 0% up to x% (ar 20% is the strongest augmentation while x = 60% is the lightest); and from x% to 100% (x = 0.05% is the strongest, x = 15% the lightest). For MM-IMDb, we also use random crop of the image modality from x% up 100% and masking of text with decreasing probability x% from 90% (the strongest) to 20% (the lightest).
Our results are shown in Fig. 7. They demonstrate, both in the controlled environment of the bi- modal Trifeatures dataset and in the real-world application of MM-IMDb, that the sweet spot of the InfoMin principle can be reached. By gradually increasing the strength of the applied transforma- tions, we enhance model performance by reducing noisy information, up to an optimal point (the sweet spot) where noise is minimized while task-relevant information is preserved. However, ap- plying overly strong augmentations destroys task-relevant information, leading to a degradation in model performance.
D PSEUDO-CODE
Algorithm presents the pseudo-code for CoMM's training. is written the general case when we have modalities. It is complementary to Fig. 3 (main paper), which depicts the case for = 2. CoMM's official implementation is available in this Github repository.
E DATASETS DETAILS
E. TRIFEATURE
The Trifeature dataset (Hermann Lampinen, 2020) was introduced as a controlled environment to study the properties of vision neural networks and how they learn different features (shape, texture and color). Images contain one of ten shapes (triangle, square, circle, etc.), rendered in one of ten textures (solid, stripes, grid, etc.), in one of ten colors (red, green, blue, etc.). Shapes are rendered within a 128 x 128 square, rotated at an angle drawn between [-45%, 45%] and placed at a random position within a larger image (224 x 224), such that the shape is fully contained in the image. Then, an independently rotated texture and a color are applied.
In our experiments, we consider the bimodal version of this dataset explained in Section 4.1 (main paper).
E.2 MULTIBENCH
All the following datasets pre-processed as described in (Liang et al., 2021).
MIMIC (Medical Information Mart for Intensive Care) (Johnson et al., 2016) is a dataset com- prising de-indentified clinical data related patients admitted critical care units large
22

---

Published as a conference paper at ICLR 2025
Algorithm 1 CoMM training algorithm
Require: Multi-modal dataset {-x,, X2, .... X,i label-preserving transformations T*, set of pro- jection transformations T -(1,,.... n....,,,),, J. batch size N, uni-modal encoders (fi)se(1...)) fusion transformer 9
for sampled mini-batch {xIIRE[U..N) = (x.,,..,, [E do
for k E [1..N] do
draw t', T
x' k, x" t- (xx), (xx:) z'k + g(f((x :);
Su (xx(R))
z²x
g(fi(x"
k (), (x))))
for i E [1.n]] do
x (xx)
z + gxf(()))))
end for end for
for i E [1..n]] do
N
expp sim 22
cxpp sim [ exp 2 22"))
L, 2N Σ
log
Σ log
exp sim 8/ 2/1)
end for
exp sim sim (2
Σ
log
N
exp x,,,'
k=1
LCMM <- C+ Ex=1 Li
update (fi)ie to minimize LCMMM
end for return (fi)
Boston-area hospital between 2001 and 2012. It contains information about 53 423 hospital ad- missions, including 38 597 distinct patients. We use the data as provided by MultiBench (Liang et al., 2021), organized in (Purushotham et al., 2018). There time series modality, composed by set of medical measurements of given patient taken ev- ery hour during 24 hours. Each measurement is a vector of size 12 (i.e., including 12 different measured numerical values). Second, a static modality. including medical information about the patient (age, gender, etc.), represented as a vector of size (tabular data). As in (Liang et al., 2023b), in our experiments we address the binary classification task of predicting whether pa- tient fits any disease in the ICD-9 code group 7 (460-519). ICD-9 (International Statistical Classification of Diseases and Relaied Health Problems) codes are used to classify diseases and a variety symptoms. Almost every health condition can be assigned unique ICD-9 code group, where each group includes set of similar diseases.
data modalities: first,
MOSI (Zadeh et al., 2016) is sentiment analysis dataset obtained from 2 199 YouTube video segments. Each sample consists of a video (visual frames), the corresponding audio and tran- scription (text). The original dataset evaluates sentiment intensities with continuous labels ranging from -3 to 3. We follow previous works (Liang et al., 2023b) and consider the binary version of the labels (positive and negative), and the same data modalities for training: text and videos.
UR-FUNNY (Hasan et al., 2019) is a dataset for humor detection human speech. It was created from 1 866 TED talk videos, obtaining more than 160000 samples (parts of the videos). Each sample in the dataset consists videos (visual frames), audio and their transcrips (text). The task is binary classification (humor or non-humor sequence).
MUSTARD
(Castro et al., 2019) is multimodal video dataset for automated sarcasm detection. It contains videos from popular television shows including Friends, The golden girls, The big bang theory and Sarcasmaholics anonymous. Each sample in the dataset correspond to an utterance composed of a video (visual frames), its corresponding audio and the transcription (text), labeled as sarcastic or non-sarcastic. As previous works (Castro al., 2019; Liang al.,
23

---

Published as a conference paper at ICLR 2025
2023b), use the balanced partition consisting of 690 utterances. In experiments with two modalities, we considered only text and vision.
Vision& Touch (Lee al., 2020) robot manipulation multimodal dataset including visual, force and proprioception data for peg insertion task. The data is collected from 7-DoF, torque-controlled Franka Panda robot, with triangle peg attached to its end-effector, Its goal is to insert the peg into triangle hole situated in box attached to a table in front of the robot. By running random policy (the robot takes random actions) and a heuristic policy (the robot attempts peg insertion), 150 trajectories recorded, each of them consisting of I 1000 timesteps. These trajectories contain RGB images, depth maps, force, end-effector position and velocity. Following MultiBench (Liang al., 2021), we consider two tasks on this dataset: (i) the binary task predicting contact or no contact the next timestep, and (ii) predicting the end-effector position (measured in MSE).
E.3 MM-IMDB
Multimodal IMDb (MM-IMDb) (Arevalo et al., 2017) is a multimodal dataset for movie genre pre- diction. It has been built from the Movielens 20M dataset (GroupLens research, 2015) by filter- ing out movies without poster image. Therefore, MM-IMDb comprises 25 959 movies along with their plot, poster, genres and additional metadata (e.g. year. language, writer, etc.). As in previous works (Arevalo et al., 2017; Liang et al., 2021; Pérez-Ris et al., 2019), we consider posters (image) and plots (text) as input data modalities to perform the multilahel classification genre prediction task (23 categories, each movie can be assigned to several categories). Technically, MM-IMDb is part of the Multibench benchmark (Liang et al., 2021), however, since we used raw data instead of the separate dataset.
proposed pre-processed features, treat
F DETAILS ON PROCESSING TIMES AND MODEL COMPLEXITY
We include Table an analysis of the complexity of CoMM against CLIP and BLIP-2. As we observe, the fusion module in CoMM adds marginal computational cost to existing backbones without compromising speed.
Model
FL.OPs
MACs
#Params
Fwd-latency
CLIP CoMM (w/ CLIP)
251G
126G
222M
488ms
281G
140G
229M
493ms
BLIP-2 CoMM (w/ BLIP-2)
9.22T
4.61T
1.17B
14s
9.48T
4.74T
1.17B
15s
Table 8:
Comparison of model complexity and processing times of different multimodal architectures.
G PROOFS
Proof (Lemma 1) Indeed, combining Assumption 1 (multiview redundancy) and equations for ((X); and I(X2; Y(X,) from Eq. (2) (main paper) we obtain:
0< T(X1; + YIX1) =U + U2 + 2S < 28injo
(8)
Since Eigo is supposed to be small and all the terms U1, U2 S 2 0, their contributions are negligible. Thus, under the mulliview redundancy assumption I I(X1, X2; 2 R.
Proof 2 (Lemma 2) Given data processing inequalities for the Markov chains X + X Z/ and Z/₁ + + Zg. we have:
1(Z6; Z%) S I(X, 26) S X(X,X')
(9)
The equality can be achieved, for example, by selecting fo(.) Id(:), ), the identity function.
24

---

Published as a conference paper at ICLR 2025
Proof 3 (Lemma 3) First, prove that [ Y) I(X,
Indeed, we have:
I(X; +
=1Z6g,;;;
(by lemma in (Wang al., 2022a))
because Zz- fo- (t(X))
(10)
Second, let T {ti} such that X' = tx(X) = X, and Zg = fe- (X₂) = Zi for E {1,2}
(with a slight abuse of notation). Thanks the previous result (in Eq. (10)) and by the consistency equations for T(X;;; Y) in Eq. (2) (main paper), the final result follows:
1(2;Y Y) -(Zo-)
= =I(X':Y)
because of Eq. (10)
= =I(X,Y
= =R+U,
because of consistency equations.
(11)
25


Reviews:
[{"review_id": "cetWeQMFIQ", "reviewer": "Reviewer_J1jX", "summary": "The authors present a multimodal representation learning method that uses multimodal augmentations to learn all four multimodal interaction terms. The proposed optimization terms are designed to optimize for $R + U_i$ and $U_1 + U_2 + R + S$. The authors design a multimodal encoder+fusion architecture that achieve state-of-the-art performance on several MM benchmarks.", "questions": "See Weaknesses for questions about theoretical foundation of this paper.\n\nIn Lemma 2, it is shown that optimum is achieved assuming \"an expressive enough network\". Do we have any idea how big/complex the network needs to be to become an expressive enough network? How does performance scale with your network's size, etc.?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "This paper grounds multimodal representation learning in the theoretical framework of multimodal information theory. Lemma 2 & 3 offer insight into novel methods that can be applied to learn multimodal representations with beyond redundancy.", "weaknesses": "I have strong doubts about the soundness of the theoretical foundation of this paper:\n1. Assumption 1 assumes the existence of a multimodal augmentation $t$ such that $I(X; t(X)) = I(X; Y)$. This means that if $X$ contains more information than the label $Y$, $t(X)$ has to \"reduce\" the information in $X$ down to $Y$. This is different from just \"label-preserving multimodal augmentation\". The experiments section also offers little insight about how exactly this augmentation is carried out. The augmentations mentioned for real-world datasets, including cropping, jitter, etc., looks like they certainly would lead to $I(X; t(X)) \\gg I(X; Y)$. This would be a direct contradiction of Assumption 1.\n2. Moreover, for the pretraining paradigm used on real-world datasets, we don't even know at pretraining time what the task $Y$ is yet. Correct me if I'm wrong--it's not clear to me how it is possible for this assumption to hold.\n3. The real gains from this method seems to be the aggressive mutual information optimization between all pairs of single modalities and all modalities as well as all data augmentations.\n\nIt would be helpful if the authors could provide more rigorous justification for Assumption 1. For example, why does Assumption 1 hold in the real world?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yRRlysILaA", "reviewer": "Reviewer_ZXRV", "summary": "This paper proposed CoMM, a Contrastive Multimodal learning framework, that captures multimodal interactions and aligns multimodal representations by maximizing the mutual information between augmented versions of these multimodal features. The paper shows that, under certain data augmentation, this framework can capture uniqueness and synergy better than existing work. Empirically, it is also shown that such framework facilitates multimodal learning on various multimodal benchmarks.", "questions": "Regard weakness 1, answering the following questions can help differentiate CoMM from existing contrastive methods and highlight the necessity for going beyond modeling redundancy:\n1. How applicable is the Multi-view redundancy assumption in the realistic settings?\n2. What makes it necessary to go beyond learning redundancy under the Multi-view redundancy assumption? Why are existing contrastive modeling approaches that focus on primary redundancy interaction insufficient?\n3. Without the Multi-view redundancy assumption, i.e. for other tasks, do those existing contrastive learning framework still capture primarily redundancy?\n\nRegard weakness 2, why is the Label-preserving multimodal augmentations (Assumption 1) considered a weaker and more realistic assumption? How can the mentioned mutual information quantity be measured in a realistic setting in order to choose the right augmentations?\n\nRegard weakness 3, the reviewer expects to see more direct evidence that shows the superior modeling of synergistic interaction of CoMM compared to the existing strong methods (e.g. FactorCL) on real-world datasets.\n\nRegard weakness 4, answering the following questions can provide additional insights into how the proposed framework better models synergy:\n1. How should a 50% probing accuracy for synergy be interpreted? Is it only based on presence/absence?\n2. What is the intuition for the crop augmentation being the key to learning synergy?\n\nRegard weakness 5, please add more details on synthetic experiment setup as previously suggested.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Novel multimodal interaction learning under a contrastive learning framework**: the proposed CoMM framework is both theoretically and empirically shown to be able to capture uniqueness and (especially) synergy better than prior work.\n2. **Comprehensive evaluation and careful ablation**: the proposed method is evaluated on a wide range of multimodal benchmarks, which involves diverse tasks, all requiring different level modeling of interactions. The paper conducts careful ablation in analyzing the importance of each part of the loss in capturing different interactions, which has provided insights in how those interactions are learnt with the proposed framework.", "weaknesses": "1. **Need for clarification about the Multi-view redundancy assumption**: the paper introduces the Multi-view redundancy assumption (Definition 1) to highlight the insufficiency of several existing works that propose cross-modal contrastive learning. Specifically, the paper shows that under this assumption, which states that \"most task-relevant information is shared across modalities\", cross-modal contrastive learning is primarily learning the redundancy interaction while ignoring the others interactions (uniqueness and synergy). The reviewer has the following questions:\n\n    1. it is unclear how applicable this assumption is in the realistic settings: in fact, as many prior work estimating multimodal interaction in real-world multimodal datasets show, multimodal tasks can vary by a diverse range of combinations of those interactions;\n\n    2. given this assumption, it seems that capturing redundancy would be sufficient already to learn task-relevant information, what makes it necessary to go beyond learning redundancy;\n\n    3. do those existing contrastive learning frameworks still capture primarily redundancy, i.e. does Lemma 1 still hold, without the multi-view redundancy assumption?\n\n2. **Strong assumption of Label-preserving multimodal augmentations (Assumption 1)**: the paper points out that existing methods like FactorCL requires strong and often unrealistic assumption about optimal multimodal augmentations based on conditional augmentations. Instead, the paper relaxes the assumption to label-preserving multimodal augmentations, which is still considered strong, as it is hard to exactly measure the mutual information between the original data and its augmented version $I(X,X')$, as well as between the data and its label $I(X,Y)$ ensuring $I(X,X')=I(X,Y)$ holds in a realistic setting. In fact, in the evaluation, the reviewer does not find any evidence to support that when evaluating on the real-world datasets, the augmentation indeed preserves the label. This not only makes the theory and the experiments detached, but also makes the choice of the \"right\" augmentation unclear, which is essential to learn the correct objective.\n3. **Indirect evidence of CoMM's superior learning of synergy**: the paper highlights the novelty of the proposed framework, CoMM, in better capturing the synergistic interaction than existing methods. However, except Figure 4 (which does not include comparison with the most similar work FactorCL), which is shown on synthetic dataset, the paper does not provide any other direct evidence for showing the claimed superior learning of synergy when evaluating on real-world datasets. For example, a large margin of performance when evaluating on datasets with primary synergistic interaction (and minimal other types of interaction), for example, VQA, will constitute strong evidence to support the claim. Also, it'll be useful to provide estimates of the proportion of different types of interaction within each real-world datasets for evaluation to better understand the advantages of CoMM for modeling different combinations of interactions.\n4. **Mysterious number of 50% accuracy for probing synergy**: except for the probing accuracy of synergy for CoMM being 71.87% (which gets round up to 71.9% in Figure 4), the probing accuracy of synergy for all other methods are all 50%. Does this number represent random chance? In other words, how is the probing accuracy of synergy evaluated? Is it only based on presence/absence? In Table 5, it is shown that without the crop augmentation for either modality 1 or modality 2, the synergy interaction seems not to be learnt. What is the intuition for the crop augmentation being the key to learning synergy? Overall, the reviewer feels an insufficient exploration and understanding of modeling synergy (the highlighted novelty of the paper) in the proposed framework, except for a little bit of discussion in the Ablation section (loss function), which only provides a hypothetical explanation.\n5. **Missing details about the synthetic experiment setups**: the reviewer does not find documentation about how redundancy, uniqueness, and synergy are defined for the Trifeature dataset or how to control the amount of each type of interaction, in the main text or in the appendix. The paper should provide more details about the synthetic experiment setups, since the original Trifeature dataset is not designed for multimodal learning but rather for feature learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3X9PYgs7P", "reviewer": "Reviewer_qsrm", "summary": "This work introduces a novel self-supervised learning objective for multimodal representation learning that captures the redundant, unique and synergetic interactions between different modalities. Their proposed approach (CoMM) is composed of a multimodal architecture, with modality-specific encoders and an attention-based fusion mechanism, that encodes a multimodal representation and a novel contrastive learning objective, grounded on the theory of partial information decomposition. The authors extensively evaluate CoMM against other self-supervised objectives, considering datasets of different complexities and number of modalities. The results highlight that the proposed method: (i) is able to capture the proposed interactions; (ii) outperforms other learning objectives in both self-supervised and fine-tuning settings; (iii) can be employed on scenarios with complex modalities (such as images and text) and with more than 2 modalities. Finally, the authors present an ablation study focused on the loss function, fusion module and data augmentation.", "questions": "- **1**: The authors claim in the introduction that they evaluate CoMM across \"diverse data types (images, text and audio)\". However, from my understanding, the datasets used in the MultiBench benchmark are composed of pre-processed features extracted from the original, high-dimensional modalities. Am I correct? Given the use of pre-processed, low-dimensional, features, instead of raw data, it might be a stretch that claiming that the authors evaluate CoMM on audio data. Furthermore, how do you apply label-preserving multimodal transformations to these feature features?\n- **2**: How to define the set of label-preserving multimodal transformations? Is it task-dependent? How large does this set need to be to work for your framework. Perhaps a section, even if in appendix, discussing these questions would be quite interesting, especially for practitioners.\n- **3**: In Section 4.1., I found it interesting that there is no comparison with FactorCL, since this comparison would emphasize the relevance of the contribution (since, to my understanding, FactorCL cannot capture the synergetic interactions in the dataset). How does FactorCL perform in this controlled experiment?\n- **4**: In Table 1, CoMM outperforms the other approaches across most datasets except for MIMIC. Any reason for this? Is there any special feature of the MIMIC dataset?\n- **5**: In Section 5, Fusion module, did you also compare to a Concat+Non-linear fusion mechanism?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **Originality**:\n  - As highlighted by the authors, the use of self-supervision contrastive objectives to learn multimodal representations is not particularly novel (see, for example [1-4] for examples in fully-training and fine-tuning settings). However, the authors focus on the problem of learning not only redundant interactions between the modalities, but also the unique and synergetic interactions. This work seems to follow in line with previous work (FactorCL [6]), extending it to account for synergetic interactions and relaxing the factorization assumption and the need for the use of optimal multimodal augmentations. The use of the Trifeature dataset to quantitively evaluate the effects of redundant, unique and synergetic interactions is quite ingenious and, to the best of my knowledge, novel.\n\n- **Quality**:\n   - The work presented here is undoubtedly of high quality. The authors present in depth their main contribution, CoMM, and ground it in the theory of partial information decomposition. The theoretical results, even if not particularly extensive, give a strong support to the proposed method. Furthermore, the evaluation shown in this paper is also of high-quality: the authors evaluate their approach in both controlled and more complex environments, against a wide range of baselines, and the results mostly support the claims of the authors (see \"Questions\" for more details and experimental clarifications). \n\n- **Clarity**:\n   - The current version of the work is well-written, with no major typos (that I could detect). The authors employ a clear notation throughout the document. Furthermore, the document is easy to read, also due to the intuitive figures created by the authors (the only exception being Figure 1, where it is not clear what does $\\mathcal{M}$ refers to). I would, however, recommend that the authors place the \"Limitations and Future Work\" section of the Appendix in the main document, as it clarifies the current limitations of the method (which is not something necessarly negative) for the reader.\n\n- **Significance**:\n   - This work proposes a novel multimodal learning objective that is able to capture a wide range of interactions between multimodal data for downstream tasks. Given the relevance of multimodal representation learning for the current field of AI, the work can have significant impact in both researchers and practitioners.\n\n\n**References**: \n\n- [1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.\n- [2] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n- [3] Li, Junnan, et al. \"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.\" International conference on machine learning. PMLR, 2022.\n- [4] Poklukar, Petra, et al. \"Geometric multimodal contrastive representation learning.\" International Conference on Machine Learning. PMLR, 2022.\n- [5] Liang, Paul Pu, et al. \"Factorized contrastive learning: Going beyond multi-view redundancy.\" Advances in Neural Information Processing Systems 36 (2024).", "weaknesses": "My two main concerns with the current version of the work are the following:\n- From an architectural point-of-view, the work presents little novelty: the multimodal fusion mechanism, based on concatenation and a transformer block, is also not particularly novel (see, for example, [1], where the authors discuss several works that employ concatenation for multimodal inputs in transformer architectures). Also, despite introducing some novel ideas, the work is still a continuation of FactorCL [2], where the use of partial information decomposition theory was also explored to derive a suitable learning objective (even if a different one from the one proposed).\n- The majority of evaluations presented by the authors are in controlled datasets (Section 4.1) or simplified environments (Section 4.2 - Multibench, Section 4.3), where the high-dimensional raw data are preprocessed into low-dimensional feature vectors (as described by the authors on Section 4.2). As such, it is not clear how the learned representations by the proposed method would work on more complex, real-world tasks (beyond the Multimodal IMDB dataset), for example on the ones proposed in [3].\n\n\n**References**:\n\n[1] - Xu, Peng, Xiatian Zhu, and David A. Clifton. \"Multimodal learning with transformers: A survey.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 45.10 (2023): 12113-12132.\n\n[2] - Liang, Paul Pu, et al. \"Factorized contrastive learning: Going beyond multi-view redundancy.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] - Ying, Kaining, et al. \"MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI.\" Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MJdmFXX5Eb", "reviewer": "Reviewer_gLzH", "summary": "In this paper, the authors proposed CoMM, a novel multimodal contrastive representation learning method that performs multiple InfoNCE contrastive losses between unimodal representations and augmented multimodal representations in the same space. The authors first theoretically justified their approach through information theory and breaking down multimodal interactions into Redundancy, Uniqueness and Synergy. The authors evaluated their proposed method first on a synthetic dataset to show that CoMM can better capture the interactions, and then evaluated CoMM against other self-supervised representation learning methods on many datasets from MultiBench and MM-IMDB with linear evaluation classification accuracy, and CoMM outperformed all baselines. The authors also conducted ablation studies on the training objective and architecture of CoMM on the synthetic dataset.", "questions": "(Not important, just out of curiosity) Why is MM-IMDB not listed as part of MultiBench? (Since MM-IMDB is included in MultiBench)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The authors proposed a novel method for an important problem (self-supervised multimodal representation learning), CoMM, that outperformed all baselines in almost all tasks. CoMM also only adds a marginal computation overhead compared to baselines.\n\n2. The authors theoretically justified why CoMM works, and confirmed their theory with experiments and analyses on the synthetic dataset.\n\n3. The paper's presentation quality is generally good. No significant grammar errors and the figures/tables are well-made.\n\n4. The authors justified their design choices through ablation studies on both synthetic and real datasets.", "weaknesses": "Minor presentation issue: In section 4.2, the title claims that all tasks in this subsection should be 2-modal, but in lines 356-358 some of the video-affect tasks are 3-modal. It is unclear which 2 modalities out of 3 were used for this experiment, and it can be quite confusing (especially on UR-funny which occurred in both the 2-modal experiments and the 3-modal experiments).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)