What to align in multimodal contrastive learning?

## Abstract

Abstract Humans perceive the world through multisensory integration, blending the information of different modalities to adapt their behavior. Contrastive learning offers an appealing solution for multimodal self-supervised learning. Indeed, by considering each modality as a different view of the same entity, it learns to align features of different modalities in a shared representation space. However, this approach is intrinsically limited as it only learns shared or redundant information between modalities,
while multimodal interactions can arise in other ways. In this work, we introduce CoMM , a Co ntrastive M ulti m odal learning strategy that enables the comm unication between modalities in a single multimodal space. Instead of imposing cross- or intra- modality constraints, we propose to align multimodal representations by maximizing the mutual information between augmented versions of these multimodal features. Our theoretical analysis shows that shared, synergistic and unique terms of information naturally emerge from this formulation, allowing us to estimate multimodal interactions beyond redundancy. We test CoMM both in a controlled and in a series of real-world settings: in the former, we demonstrate that CoMM effectively captures redundant, unique and synergistic information between modalities. In the latter, CoMM learns complex multimodal interactions and achieves state-of-the-art results on the six multimodal benchmarks.

## 1 Introduction

Multisensory or multimodal learning (Baltrušaitis et al., ) involves extracting and processing information from multiple sources (or modalities, e.g. text, audio, images, tabular data, etc.) to perform a task. The whole human experience is inherently multimodal: we simultaneously see, hear, smell, taste and feel, and these different sensory signals are combined to give us the necessary information to explore our environment. Indeed, many of the simplest tasks we tackle in our daily lives are multimodal. For example, the way we perceive the flavor of our food or drinks does not depend solely on our taste, but also on what we see (Morrot et al., ) or even what we hear (Woods et al., ) while we eat. McGurk & MacDonald () have also demonstrated that visual stimuli interact with audio signals to perform human speech recognition.

Despite the inherent multimodality of sensory systems, machine learning has largely concentrated on single-modality models, with few exceptions in areas like audio-visual speech recognition (Yuhas et al., ; Ngiam et al., ), multimedia content retrieval (Atrey et al., ; Snoek & Worring, ), and video-based human behavior analysis (Kraaij et al., ). Nowadays, with the emergence of self-supervised strategies and their impressive capacities for learning representations in computer vision (Chen et al., ; He et al., ; Caron et al., ), NLP (Radford et al., ; Devlin et al., ) or audio (Oord et al., ; Niizumi et al., ), the paradigm has shifted to learning general multimodal representations from unlabeled data and then fine-tune the models to specific multimodal tasks. Recent works have shown success at training multimodal representations by using cross-modal contrastive objectives (Radford et al., ; Jia et al., ) to align the representations in a shared embedding space. However, this training strategy only works under the multiview redundancy assumption, *i.e*., considering that all task-relevant information is shared between modalities and redundantly contained in either one of them separately. In particular, for vision-language tasks, this can be seen as a clever way to perform supervised learning on a visual encoder, which explains their success to transfer to visual classification tasks.

Figure: Figure 1: a) We propose CoMM, a contrastive multimodal approach that allows the interplay of multiple modalities and learns multimodal interactions. Unlike previous multimodal models (Cross) that align cross-modal features, CoMM aligns multimodal features in a shared representation space. b) Multimodal interactions are task-dependent, thus a model needs to capture all of them to generalize to any multimodal task. CoMM’s new paradigm captures multimodal interactions beyond redundancy.
Refer to caption: extracted/5848597/images/Fig1v3.png

Nonetheless, these solutions are insufficient in many cases, as the interactions between modalities can arise in several ways to perform a specific task (Bertschinger et al., ): redundancy ($R$) arises when the task can be performed using either of the modalities because they contain redundant information; uniqueness ($U$) appears when only one of the modalities contains all the necessary information to complete the task; synergy ($S$) emerges when both modalities have complementary information, and they are needed simultaneously to fulfill the task.
Modeling these interactions to perform multimodal learning is highly challenging as the interplay between $R,S$ and $U$ is task-dependent and difficult to measure in complex real-life scenarios. Fig. 1b shows simple tasks where the predominant type of interaction can be easily identified. We observe that we need to model specific interactions for the same input modalities to perform a specific task. Therefore, a model must capture all these terms to learn task-agnostic multimodal representations.

We propose a Contrastive MultiModal self-supervised pre-training method (CoMM) that enables the communication of modalities in a single multimodal space. Unlike previous contrastive multimodal methods that impose cross-modal constraints to align unimodal representations, we propose to leverage a simple multimodal architecture to fuse multimodal inputs into a common representation and then align the multimodal features by maximizing the mutual information between augmented versions of these features (see Fig. 1a). CoMM enables to model multimodal interactions –including redundancy, uniqueness and synergy– in the context of multimodal representation learning for the first time(^1^11to the best of our knowledge)
, as these terms naturally emerge from our contrastive multimodal formulation.

CoMM’s formulation is well-aligned with the global workspace theory (Baars, ; Goyal & Bengio, ) in cognitive neuroscience, which considers the nervous system as a set of multiple specialized processors working in parallel and claims the existence of a shared representation, which can be modified by any selected processor and whose content is broadcast to all processors. By analogy, CoMM considers a shared representation space built from parallel streams of modality-specific processors. When a task requires knowledge from a given modality or interactions between modalities, only these parts of the representation space should be used.

Based on Partial Information Decomposition (Williams & Beer, ; Bertschinger et al., ), we built a strong theoretical basis for CoMM to learn multimodal interactions in a self-supervised way.
Empirically, we show that CoMM effectively captures redundant, unique and synergistic information between modalities in a controlled environment, where the type of interaction is known.
In a series of real-world datasets –from different domains (healthcare, robotics, *etc*.) and including diverse data types (image, text, audio, *etc*.)–, CoMM achieves state-of-the-art results on seven multimodal tasks with two or three modalities. Thus, CoMM shows to be a versatile framework capable of handling any number of modalities, various data types, and different domains.

## 2 Quantifying multimodal interactions

#### Problem setup.

Let $X_{1},X_{2},\ldots,X_{n}$ be random variables representing $n$ different data modalities (e.g. images, text, audio, tabular data, etc.), and a given task $Y$. Our goal is to learn a latent variable $Z=f(X)$ that is a good representation of $X=(X_{1},\ldots,X_{n})$ for $Y$.

For our theoretical analysis, we set $n=2$, as multimodal interactions have not been characterized yet by PID for larger $n$. All proofs can be found in Appendix G.

For $Z$ to be a good representation of $X$ it should capture the task-relevant information that $X$ contains. Therefore, we need to model the information between the joint variable $X$ and the task $Y$: $I(X;Y)=I(X_{1},X_{2};Y)$.

Partial information decomposition (PID) (Williams & Beer, ; Bertschinger et al., ) states that multivariate mutual information $I(X_{1},X_{2};Y)$ is decomposed into three forms of interactions:

- (i)
Uniqueness. This term appears when the task $Y$ can be completed by leveraging only one of the modalities. $U_{1}$ (resp. $U_{2}$) refers to the case when $X_{1}$ (resp. $X_{2}$) contains all task-relevant information.
- (ii)
Redundancy. When $X_{1}$ and $X_{2}$ contain the same information about $Y$. $R$ corresponds to this redundant or shared information.
- (iii)
Synergy. Noted by $S$, this term only emerges when $X_{1}$ and $X_{2}$ are simultaneously present, because they bring different and complementary task-relevant information.

Thus, the information that $(X_{1},X_{2})$ has about $Y$ can be written as the contribution of four terms:

$$ $I(X_{1},X_{2};Y)=R+S+U_{1}+U_{2}$ (1) $$

Moreover, Eq. 1 and the application of the chain rule of mutual information yield the following consistency equations between $R,S,U_{1}$ and $U_{2}$:

$$ $I(X_{1};Y)=R+U_{1},\qquad I(X_{2};Y)=R+U_{2},\qquad I(X_{1};X_{2};Y)=R-S$ (2) $$

Existing methods using contrastive objectives to learn multimodal representations (Jia et al., ; Radford et al., ; Tian et al., ) impose cross-modal constraints by maximizing an estimator of $I(X_{1};X_{2})$ as an approximation of $I(X_{1},X_{2};Y)$.
However, this strategy is limited by the multiview redundancy assumption.

###### Definition 1 ( Multi-view redundancy )

$\exists$ $\varepsilon>0$ such that $I(Y;X_{1}|X_{2})<\varepsilon$ and $I(Y;X_{2}|X_{1})<\varepsilon$.

This assumption states that most task-relevant information is shared across modalities and the non-shared information is (at most) a small $\varepsilon$. In other words, any of the modalities contains enough information to fulfill the downstream task $Y$, and they can provide some kind of “supervision” to one another, which explains their success for zero-shot image classification (El Banani et al., ).

###### Lemma 1

Under the multiview redundancy assumption, cross-modal contrastive learning methods are limited to only learn the redundant information R.

What happens when other sources of multimodal information intervene? FactorCL (Liang et al., ) is a good initial step to integrate uniqueness and redundancy into multimodal contrastive learning by applying multimodal augmentations. However, it heavily relies on the assumption that optimal multimodal augmentations can be obtained by applying a two-step process based on conditional augmentations. We argue that this hypothesis is unrealistic, as the first unique augmentation is strongly related to task-relevant information of different modalities. For example, if the text caption is “a yellow flower”, then color jittering should not be applied to an image depicting a flower. Besides, the factorized formulation of FactorCL is impractical as it is prone to cumulative errors. Finally, this method does not consider the synergy between modalities.
In contrast, we propose a model that relies solely in the main hypothesis of contrastive learning, extended to the multimodal case, without relying on strong assumptions between multimodal relationships nor conditional augmentations.

###### Assumption 1 ( Label-preserving multimodal augmentations )

We assume the existence of $\mathcal{T}^{\star}$, a set of multimodal augmentations such that for any $t\in\mathcal{T}^{\star}$ and $X^{\prime}=t(X)$, we have $I(X,X^{\prime})=I(X,Y)$.

Even if 1 might seem strong at first glance, it actually makes sense in the context of multimodal representation learning. Indeed, coming back to the example of the flower image with “a yellow flower” as a caption, applying color jittering to the image would allow the model to focus on other interactions (uniqueness or synergy) rather than color redundancy or even to refocus on other features (like the flower shape).

Moreover, our assumption allows for a larger spectrum of augmentations, without being constrained to the set $\mathcal{T}^{\star}_{c}=\{t(X)=(t_{1}(X_{1}),t_{2}(X_{2}))\}$ of transformations that can be decomposed in independent unimodal augmentations.

###### Definition 1 ( Multi-view redundancy )

###### Lemma 1

###### Assumption 1 ( Label-preserving multimodal augmentations )

## 3 CoMM: Contrastive Multimodal learning

We aim to learn multimodal representations that are transferable to any multimodal task. Contrastive learning has shown promising results in multimodal learning. However, current approaches fail to capture multimodal interactions other than redundancy, as shown in Section 2.
Our strategy builds upon multiview contrastive learning theory and extends it to the multimodal case. It is based on two main components:

- (i)
A multimodal architecture, with specialized encoders to process any data type, and an attention-based fusion module to obtain a final multimodal representation.
- (ii)
A contrastive objective that naturally captures unique, redundant, and synergistic interactions between different data modalities.

### 3.1 Towards effective multimodal representations

To obtain robust, task-agnostic and common representations $Z$ that capture uniqueness, redundancy and synergy from different input modalities, we design $f_{\theta}$ –a neural network parameterized by $\theta$– such that $Z_{\theta}=f_{\theta}(X)=f_{\theta}(X_{1},X_{2})$.

We define $X^{\prime}=t(X)$ with $t\in\mathcal{T}$ a stochastic mapping(^2^22Here, $\mathcal{T}$ can be any set of mappings, *e.g*., $\mathcal{T}\neq\mathcal{T}^{\star}$.) (multimodal augmentation) of $X$ and $Z_{\theta}^{\prime}=f_{\theta}(X^{\prime})$.

Given data processing inequalities for Markov chains
$X\rightarrow X^{\prime}\rightarrow Z^{\prime}_{\theta}$
and
$Z^{\prime}_{\theta}\rightarrow X\rightarrow Z_{\theta}$
, we have:

$$ $I(Z_{\theta};Z^{\prime}_{\theta})\leq I(X,Z^{\prime}_{\theta})\leq I(X,X^{ \prime})$ (3) $$

With these inequalities, we can prove the following lemmas:

###### Lemma 2

By optimizing $f_{\theta}$ to maximize $I(Z_{\theta};Z^{\prime}_{\theta})$, and if we assume an expressive enough network $f_{\theta}$, we have at optimum:

$$ $I(Z_{\theta^{\star}},Z^{\prime}_{\theta^{\star}})=I(X,X^{\prime})$ (4) $$

###### Lemma 3

Let $f_{\theta^{\star}}$ be optimal, i.e. $f_{\theta^{\star}}$ maximizes $I(Z_{\theta},Z^{\prime}_{\theta})$.
Then, we have the equality $I(Z^{\prime}_{\theta^{\star}};Y)=I(X^{\prime};Y)$. If we consider the special case $\mathcal{T}=\{t_{i}\}$ such that $X^{\prime}=t_{i}(X)=X_{i}$ and $Z^{\prime}_{\theta^{\star}}=f_{\theta^{\star}}(X_{i})=Z_{i}$ for $i\in\{1,2\}$, then it follows:

$$ $I(Z_{i};Y)=I(X_{i};Y)=R+U_{i}$ (5) $$

Lemma 3 implies that optimal representations $Z_{i}$ preserve all the task-relevant information contained in modality ${i}$. Interestingly, we do not require 1 for this equality to hold.

The previous theoretical developments lead us to the key ingredients for CoMM’s contrastive objectives to succeed at capturing multimodal interactions (see Section 3.3 for practical implementation):

- (i)
Following Lemma 2, $U_{1}+U_{2}+R+S=I(X,Y)$ can be learned by optimizing the term $I(Z_{\theta},Z_{\theta}^{\prime})$ for $\mathcal{T}=\mathcal{T}^{\star}$, since $I(X,X^{\prime})=I(X,Y)$ by 1;
- (ii)
Thanks to Lemma 3, $R+U_{i}$ for $i\in\{1,2\}$ can be directly learned by optimizing the term $I(Z_{\theta},Z^{\prime}_{\theta})$ for $\mathcal{T}=\{t_{i}\}$.

###### Lemma 2

###### Lemma 3

### 3.2 Multimodal architecture

Figure: Figure 2: CoMM’s model architecture. Inputs from different modalities $X=(X_{1},...,X_{n})$ are first encoded by modality-specific encoders. Modality-specific features are processed by latent converters to map them into sequences of embeddings which are concatenated and fused by a transformer block. The output is a single multimodal feature $\mathbf{Z_{\theta}}$.
Refer to caption: extracted/5848597/images/model-n.png

Our architecture for multimodal representation learning is presented in Fig. 2. In order to capture multimodal interactions, the model consists of mainly three components:
Encoders. Each modality is encoded independently by one of the $n$ modality-specific encoders.
Latent converters. Linear modules that transform features into modality-specific sequences of embeddings.
After the latent converters, a concatenation operation gathers these sequences to be fed into a transformer architecture.
Transformer block. The goal of this module is to perfom the fusion of the modality-specific embeddings through the multihead self-attention layers in the transformer block, obtaining the final multimodal embedding $Z$.

More information about the specific modules used as modality encoders, about the latent converters and the transformer block architecture can be found in Appendix B.

### 3.3 Training

Given a multimodal input $X=(X_{1},...,X_{n})$ and a set of label-preserving multimodal transformations $\mathcal{T^{\star}}$, two augmentations $t^{\prime},t^{\prime\prime}$ are drawn from $\mathcal{T^{\star}}$ and applied to $X$, obtaining $X^{\prime}=t^{\prime}(X)$ and $X^{\prime\prime}=t^{\prime\prime}(X)$. We also consider projections (with a slight abuse of notation) $X_{i}=(\texttt{[MSK]},...,X_{i},...,\texttt{[MSK]})$ for $i\in\{1,...,n\}$, where every modality is masked except for the $i$-th.

These terms are encoded by the network to obtain $(n+2)$ embeddings, namely: $Z^{\prime},Z^{\prime\prime}$, and $\{Z_{i}\}_{i=1}^{n}$.
$(2n+1)$ mutual information terms are then optimized through backpropagation: $I(Z^{\prime},Z^{\prime\prime})$ to maximize $I(Z,Z^{\prime})$ in Eq. 3 and $I(Z_{i},Z^{\prime})$ and $I(Z_{i},Z^{\prime\prime})$ to better approximate $R+U_{i}$ in Eq. 5 for $i\in\{1,...,n\}$.

We use the InfoNCE (Oord et al., ) estimator of mutual information due to its simplicity and strong results in the self-supervised learning literature:

$$ $\hat{I}_{\text{NCE}}(Z,Z^{\prime})=\mathbb{E}_{\begin{subarray}{c}z,z^{\prime} _{\text{pos}}\sim p(Z,Z^{\prime})\\ z^{\prime}_{\text{neg}}\sim p(Z^{\prime})\end{subarray}}\left[\log\frac{\exp \text{sim}(z,z^{\prime}_{\text{pos}})}{\sum_{z^{\prime}_{\text{neg}}}\exp\text {sim}(z,z^{\prime}_{\text{neg}})}\right]$ (6) $$

Given this estimator, our final training loss can be written as:

$$ $\mathcal{L}_{\text{CoMM}}=-\underbrace{\hat{I}_{\text{NCE}}(Z^{\prime},Z^{ \prime\prime})}_{\approx R+S+\sum_{i=1}^{n}U_{i}}-\sum_{i=1}^{n}\underbrace{ \tfrac{1}{2}\left(\hat{I}_{\text{NCE}}(Z_{i},Z^{\prime})+\hat{I}_{\text{NCE}}( Z_{i},Z^{\prime\prime})\right)}_{\approx R+U_{i}}=:\mathcal{L}+\sum_{i=1}^{n} \mathcal{L}_{i}$ (7) $$

Fig. 3 illustrates CoMM’s training process for the case $n=2$. The pseudocode for the general case $n\geq 2$ is available in Appendix D. It is worth to notice that the loss terms in Eq. 7 grow linearly with the number of modalities $n$.

At inference, no augmentations are applied. The multimodal input $X=(X_{1},...,X_{n})$ is processed through the network to obtain the multimodal feature $Z_{\theta}=f_{\theta}(X)$. This multimodal representation can then be directly transferred to any task either by performing linear probing or by fine-tuning the whole architecture.

Figure: Figure 3: CoMM training for $n=2$. Two multimodal augmentations are applied to $X$ to obtain $X^{\prime}$ and $X^{\prime\prime}$. We also consider the projection operators to get $\{X_{i}\}_{i=1}^{n}$. These $n+2$ transformed versions of $X$ are processed by the network $f_{\theta}$, trained to maximize the agreement between these $n+2$ terms using contrastive objectives.
Refer to caption: extracted/5848597/images/trainingv4.png

## 4 Experiments

We design different sets of experiments to evaluate our model: first, over a controlled environment inspired by the Trifeature dataset (Hermann & Lampinen, ), we carefully formulate tasks that need a specific kind of interaction (uniqueness, redundancy, or synergy) to assess the model’s ability to capture them separately. Second, in order to evaluate the capacities of the model to learn multimodal representations for complex tasks requiring different levels of shared, unique, and synergistic information, we use real-world benchmark datasets with 2 or 3 modalities.

Experimental settings. We report mean and standard deviation over 5 runs for all our results (except when using public model weights). All hyper-parameter details can be found in Appendix B.
Evaluation. Given a pre-trained model $f$, we perform evaluation through linear probing, *i.e*., we train a linear layer $g_{W}(x)=Wf(x)$ ($f$ fixed) to minimize the classification or regression loss (depending on the task); and fine-tuning, further training $f$ in a supervised way.

### 4.1 Controlled experiments on bimodal Trifeature dataset

Figure: Figure 4: Linear probing accuracy of redundancy (shape), uniqueness (texture) and synergy (color and texture) on bimodal Trifeature dataset. CoMM is the only model capturing all three task-related interactions between modalities(^4^44We ran the experiments with the official implementation of FactorCL (Liang et al., ) but performances were close to random chance so we did not integrate them to this analysis.).
Refer to caption: extracted/5848597/images/trifeatures_bar_results.png

Does CoMM learn redundant, synergistic and unique information for a given task? We design a synthetic bimodal dataset based on Trifeature (Hermann & Lampinen, ) to answer. Briefly, we generate a Trifeature dataset (as first modality) containing images of one of ten shapes, for one of ten textures and one of ten colors (1000 combinations in total). We augment each image three times using rotations and translations. Then, we pair each image with a second one (as second modality) from the same Trifeature dataset, allowing us to control the shared, unique and synergistic attributes between the two modalities. As Hermann & Lampinen (), we use AlexNet (Krizhevsky et al., ) as modality-specific encoder for both modalities in all experiments with an embedding dimension $d=512$.
Experiment 1 – Modeling shared and unique interactions. We choose the shape as shared feature and texture as unique feature for each modality by selecting only paired images with the same shape. Training and test set follow the same distribution, with 10000 and 4096 images, respectively.
We measure the linear probing accuracy of shape (respectively texture) to test whether redundancy (resp. uniqueness) has been captured in the latent representation of the model (chance level=10%).
Experiment 2 – Modeling synergy. In the previous experiment, texture and color features are independent between the two modalities. To introduce a synergy between these features, we bias the training set by defining a mapping $\mathcal{M}$ between the ten textures and ten colors (e.g. stripes=red, dots=green, etc.). Then, we select randomly 10000 pairs of images that respect this mapping, thus artificially introducing a strong correlation between texture from the first modality and color from the second modality in the training set. The test set is left unchanged from previous Experiment 1 and the task $Y$ is to detect whether a given pair of images respects the mapping $\mathcal{M}$, i.e. $Y=\mathds{1}(\text{texture}(X_{1}),\text{color}(X_{2})\in\mathcal{M})$ (chance level=50%).

In Fig. 4, we show that cross-modality constraints with InfoNCE (Radford et al., ) (“Cross” model) allow to perfectly capture redundant information but completely remove unique and synergistic information, as predicted in Lemma 1. Self-supervised constraints on each encoder (“Cross+Self” (Yuan et al., )) capture accurately unique information but fail at preserving synergy. CoMM is the only model learning all three interactions related to the task.

### 4.2 Experiments on real-world datasets with 2 modalities

MultiBench.
Following (Liang et al., ), we use a subset of real-world multimodal datasets from MultiBench (Liang et al., ), with different degrees of shared and unique task-relevant information, including: Vision&Touch (Lee et al., ) a robotics dataset that includes images, force, and proprioception data for end-effector position prediction (regression) and contact prediction (binary classification),
MIMIC (Johnson et al., ), a dataset for mortality and disease prediction from medical records, including tabular data and medical time series from ICU;
MOSI (Zadeh et al., ), a dataset for sentiment analysis from videos (vision, audio, and language); UR-FUNNY (Hasan et al., ), humor detection from videos (vision, audio and language); and MUsTARD (Castro et al., ), a dataset for sarcasm detection from TV shows (vision, audio, and language).
For fair comparisons, we run our experiments using the same data pre-processing steps as previous works, using pre-extracted text, video and audio features for training (Liang et al., ; ). More details about these datasets and the data pre-processing can be found in Appendix E. We train CoMM on the same data modalities as FactorCL and use the same backbone networks.

**Table 1: Linear evaluation top-1 accuracy (in %) for classification tasks and MSE ($\times 10^{-4}$) for regression task (V&T End Effector) on MultiBench after 100 epochs. ^†Results obtained from (Liang et al., ). ^∗ Average is taken over classification results only. Rows in color are supervised.**
| Model | Regression | Classification |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| V&T EE $\downarrow$ | MIMIC $\uparrow$ | MOSI $\uparrow$ | UR-FUNNY $\uparrow$ | MUsTARD $\uparrow$ | Average^∗ $\uparrow$ |  |
| Cross^† (Radford et al., ) | $33.09_{\pm 3.67}$ | $66.7_{\pm 0.0}$ | $47.8_{\pm 1.8}$ | $50.1_{\pm 1.9}$ | $53.5_{\pm 2.9}$ | 54.52 |
| Cross+Self^† (Yuan et al., ) | $7.56_{\pm 0.31}$ | $65.49_{\pm 0.0}$ | $49.0_{\pm 1.1}$ | $59.9_{\pm 0.9}$ | $53.9_{\pm 4.0}$ | 57.07 |
| FactorCL^† (Liang et al., ) | $10.82_{\pm 0.56}$ | $\mathbf{67.3}_{\pm 0.0}$ | $51.2_{\pm 1.6}$ | $60.5_{\pm 0.8}$ | $55.80_{\pm 0.9}$ | 58.7 |
| CoMM (ours) | $\mathbf{4.55}_{\pm 0.30}$ | $66.4_{\pm 0.32}$ | $\mathbf{67.5}_{\pm 1.30}$ | $\mathbf{63.1}_{\pm 0.65}$ | $\mathbf{63.9_{\pm 3.01}}$ | 65.22 |
| SupCon^† (Khosla et al., ) | - | $67.4_{\pm 0.0}$ | $47.2_{\pm 1.2}$ | $50.1_{\pm 2.0}$ | $52.7_{\pm 2.2}$ | 54.35 |
| FactorCL-SUP^† (Liang et al., ) | $1.72_{\pm 0.03}$ | $\mathbf{76.8}_{\pm 0.0}$ | $69.1_{\pm 0.6}$ | $63.5_{\pm 0.8}$ | $69.9_{\pm 1.9}$ | 69.82 |
| CoMM (fine-tuned) | $\mathbf{1.34}_{\pm 0.01}$ | $68.18_{\pm 0.23}$ | $\mathbf{74.98}_{\pm 0.43}$ | $\mathbf{65.96}_{\pm 0.44}$ | $\mathbf{70.42}_{\pm 0.15}$ | 69.88 |

We consider two different experimental settings. In the self-supervised setting, we perform pre-training and evaluate with linear probing. We consider FactorCL, “Cross” and “Cross+self” methods for comparison. In the fine-tuning setting, CoMM is fully fine-tuned in a supervised way after pre-training. We compare it against SupCon and FactorCL-Sup as supervised methods.

Results for these experiments are in Table 1. In the self-supervised experiments with linear probing evaluation, CoMM surpasses FactorCL (second best) by large margins (16.3%, 2.6% and 8.1% of top-1 accuracy on MOSI, UR-FUNNY and MUsTARD, respectively) on three out of four classification datasets. On MIMIC, margins are considerably narrower, with FactorCL performing slightly better, and CoMM showing comparable results with “Cross” and “Cross+Self”. In the regression task of V&T, CoMM is considerably better than competitors ($3\times 10^{-4}$ lower MSE than second best). In the fine-tuning scenario, we observe the same pattern, with CoMM outperforming competitors on four datasets and FactorCL taking the lead on MIMIC.

These experiments not only show CoMM’s efficiency at learning multimodal representations, but also exhibit CoMM’s versatility to process different data domains (time series, audio, images, text, *etc*.) and to adapt to diverse backbones.

Multimodal IMDb (MM-IMDb) (Arevalo et al., ) is a dataset for movie genre prediction. We consider two modalities: images and text (the movie poster and its plot, respectively). Since each movie can be classified into one or more genres, it is a multi-label classification task, with 23 categories. MM-IMDb provides a suitable example of life-like multimodal task as the genre prediction cannot be performed accurately from the movie poster or the movie plot alone, while results significantly improve by considering both (Arevalo et al., ), suggesting that synergistic interactions are needed to fulfill this task.
We compare CoMM’s performance on MM-IMDb against important baselines under two different settings. First, in the self-supervised setting, we consider CLIP (Radford et al., ), representing “Cross” methods, SLIP (Mu et al., ) representing “Cross+Self” methods, SimCLR (Chen et al., ) for unimodal self-supervised methods, BLIP-2 (Li et al., ) as a recent powerful vision and language model, as baselines. All models were trained on unlabeled MM-IMDb. For CLIP, we also report results with the publicly released weights without further training. CoMM is initialized with pre-trained weights from CLIP and BLIP-2. Second, in the fine-tuning setting, we compare CoMM fine-tuned with state-of-the-art fully supervised baselines. We also include results
for LLaVA-NeXT (Li et al., ) representing new vision-and-language generative models, however these scores are not fully comparable to ours, since the model might have seen IMDb database during training. See Appendix B for implementation details.

**Table 2: Linear evaluation F1-score (weighted and macro) (in %) on MM-IMDb after 70 epochs. $\dagger$ indicates further training on unlabeled data. $\ddagger$ means supervised fine-tuning. Rows in color are supervised.**
| Model | Modalities | weighted-f1 | macro-f1 |
| --- | --- | --- | --- |
| SimCLR^† (Chen et al., ) | V | $40.35_{\pm 0.23}$ | $27.99_{\pm 0.33}$ |
| CLIP (Radford et al., ) | V | 51.5 | 40.8 |
| L | 51.0 | 43.0 |  |
| V+L | 58.9 | 50.9 |  |
| BLIP-2 (Li et al., ) | V+L | $57.4$ | $49.9$ |
| SLIP^† (Mu et al., ) | V+L | $56.54_{\pm 0.19}$ | $47.35_{\pm 0.27}$ |
| CLIP^† (Radford et al., ) | V+L | $54.49_{\pm 0.19}$ | $44.94_{\pm 0.30}$ |
| CoMM (ours, CLIP backbone) | V+L | $\underline{61.48}_{\pm 0.18}$ | $\underline{54.63}_{\pm 0.22}$ |
| CoMM (ours, BLIP-2 backbone) | V+L | $\mathbf{64.75}_{\pm 0.17}$ | $\mathbf{58.44}_{\pm 0.43}$ |
| MFAS (Pérez-Rúa et al., ) | V+L | 62.50 | 55.6 |
| ReFNet (Sankaran et al., ) | V+L | - | 56.7 |
| CoMM^‡ (ours, CLIP backbone) | V+L | $\underline{64.90}_{\pm 0.21}$ | $\underline{58.97}_{\pm 0.19}$ |
| CoMM^‡ (ours, BLIP-2 backbone) | V+L | $\mathbf{67.39}_{\pm 0.07}$ | $\mathbf{62.0}_{\pm 0.25}$ |
| LLaVA-NeXT (Li et al., ) | V+L | $64.28$ | $56.51$ |

Table 2 shows that CoMM outperforms all models in both settings. In the self-supervised setting, CoMM has a margin of $7.5\%$ and $5.8\%$ of macro and weighted F1-scores, respectively, with the second-best method. It is interesting to observe that CLIP performs better with the publicly released weights (probably because of the large-scale and diversity of the data it was trained on), than with further training on MM-IMDb. This result suggests that the genre movie prediction does not benefit from learning redundant information. Including uniqueness from the image modality allows for some improvement (SLIP). In the fine-tuning setting, CoMM fine-tuned outperforms existing fully supervised baselines even if MFAS has been designed to search for the best fusion strategy.

### 4.3 Experiments on real-world datasets with 3 modalities

**Table 3: Linear evaluation top-1 accuracy (%) on Vision&Touch (Lee et al., ) in the robotics domain and UR-FUNNY.**
| Model | #Mod. | V&T CP | UR-FUNNY |
| --- | --- | --- | --- |
| Cross | 2 | 84.4 | 50.1 |
| Cross+Self | 2 | 86.8 | 59.9 |
| CoMM (ours) | 2 | 88.1 | 63.1 |
| CMC (Tian et al., ) | 3 | 94.1 | 59.2 |
| CoMM (ours) | 3 | 94.2 | 64.6 |

We test CoMM’s abilities to learn multimodal interactions beyond 2 modalities.
We perform experiments on two large datasets including tri-modal data from MultiBench: Vision&Touch (contact prediction task) and UR-FUNNY.

In Table 3, we compare CoMM trained on the three modalities in a self-supervised way against CMC (Tian et al., ). We also compare with bi-modal models: CoMM, Cross and Cross+Self trained on image and proprioception data for Vision&Touch and image and text data for UR-FUNNY.

First, we observe a consistent improvement ($+6.1\%$ on V&T, $+1.5\%$ on UR-FUNNY) when adding a third modality with CoMM (compared to only using two), which demonstrates its versatility. Second, we improve the state-of-the-art for SSL methods on datasets with more than two modalities ($+0.1\%$ and $+5.4\%$ on V&T and UR-FUNNY respectively).

## 5 Ablation studies

We test three main components of our framework (the loss, fusion module and augmentation strategy) against important control baselines on bimodal Trifeature dataset (see Section 4.1).

Figure: Figure 5: Linear probing accuracy of redundancy $R$, uniqueness $U=\frac{1}{n}\sum_{i=1}^{n}U_{i}$ and synergy $S$ on bimodal Trifeature when optimizing each term separately in $\mathcal{L}_{\text{CoMM}}$. Minimizing $\mathcal{L}_{i}$ allows to learn $U_{i}$ and $R$, approximating $I(X_{i};Y)$ for $i\in\{1,...,n\}$. Optimizing $\mathcal{L}=-\hat{I}(Z^{\prime},Z^{\prime\prime})$ allows to slowly learn $R$, $U_{i}$ and $S$. CoMM quickly captures all information.
Refer to caption: extracted/5848597/images/ablation_study_trifeatures.png

Loss function.
First, we check our claim that optimizing both $\mathcal{L}$ and $\sum_{i=1}^{n}\mathcal{L}_{i}$ –as in Eq. 7– is required to accurately capture uniqueness, synergy, and redundancy. In  Fig. 5, we show that maximizing $\sum_{i=1}^{n}\mathcal{L}_{i}$ improves redundancy and uniqueness, as guaranteed by our Lemma 3, but fails for synergy. Conversely, maximizing $\mathcal{L}$ allows one to learn all information terms but very slowly. In particular, for synergy, we argue it is because the model has to learn modality-specific features first (phase 1) before learning their interactions (phase 2). The former is learned through $I(Z_{i},Z^{\prime})+I(Z_{i},Z^{\prime\prime})$ while the latter is captured with $I(Z^{\prime},Z^{\prime\prime})$. Hence, $\mathcal{L}_{\text{CoMM}}$ speeds up phase 1 and can learn synergy more efficiently in phase 2.

**Table 4: Linear fusion module versus attention-based fusion with latent converters (CoMM). Non-linearities are required to learn synergistic interactions between modalities.**
| Fusion | Redundancy | Uniqueness | Synergy | Average |
| --- | --- | --- | --- | --- |
| Concat + Linear | $99.71_{\pm 0.06}$ | $81.49_{\pm 2.88}$ | $50.0_{\pm 0.0}$ | $77.07$ |
| CoMM | $\mathbf{99.92}_{\pm 0.03}$ | $\mathbf{87.83}_{\pm 1.55}$ | $\mathbf{71.87}_{\pm 2.06}$ | $\mathbf{86.83}$ |

Fusion module.
We compare our attention-based latent fusion module with shallow linear fusion. We project modality-specific representations to the common latent space using only linear layers and remove latent converters. Table 4 shows that synergy is not captured with linear fusion. This is expected as the XOR gate (typical example of synergistic interactions) cannot be approximated by a linear function. Additionally, uniqueness accuracy is also degraded compared to CoMM ($-6\%$), suggesting that only modeling linear interactions limits the model’s representation power.

**Table 5: Effect of data augmentation on linear probing accuracy (%) of multimodal interactions. All refers to SimCLR augmentations (Chen et al., ). CoMM uses All for both modalities.**
| Augmentations | $R$ | $U_{1}$ | $U_{2}$ | $S$ | Average |  |
| --- | --- | --- | --- | --- | --- | --- |
| Modality 1 | Modality 2 |  |  |  |  |  |
| {All} | $\varnothing$ | $99.72_{\pm 0.04}$ | $79.92_{\pm 1.05}$ | $46.44_{\pm 3.74}$ | $50.0_{\pm 0.0}$ | $69.02$ |
| $\varnothing$ | {All} | $99.58_{\pm 0.13}$ | $53.89_{\pm 6.69}$ | $86.04_{\pm 2.01}$ | $50.0_{\pm 0.0}$ | $72.37$ |
| {All}$\setminus\{\text{crop}\}$ | {All} | $88.79_{\pm 0.96}$ | $25.65_{\pm 0.10}$ | $84.00_{\pm 4.43}$ | $50.0_{\pm 0.0}$ | $62.11$ |
| {All} | {All}$\setminus\{\text{crop}\}$ | $90.50_{\pm 2.07}$ | $83.22_{\pm 2.86}$ | $21.74_{\pm 1.36}$ | $50.0_{\pm 0.0}$ | $61.36$ |
| CoMM | $\mathbf{99.92}_{\pm 0.03}$ | $\mathbf{84.35}_{\pm 2.37}$ | $\mathbf{91.19}_{\pm 0.97}$ | $\mathbf{71.87}_{\pm 2.06}$ | $\mathbf{86.83}$ |  |

Data augmentation.
Finally, we show in Table 5 that strong data augmentation is crucial for learning multi-modal interactions, in line with the literature on unimodal contrastive methods. Contrary to FactorCL (Liang et al., ), applying strong augmentation on *both* modalities is beneficial for CoMM and we do not require task-dependent augmentations, highlighting the versatility of our framework.

## 6 Related work

Multimodal learning refers to methods that connect and integrate information from multiple sources of data (Baltrušaitis et al., ; Akkus et al., ). Early works focused on training separate encoders and studied different fusion mechanisms to blend the information from different inputs (Zeng et al., ; Pérez-Rúa et al., ). With the development of transformers and ViTs, the focus has shifted towards training a unique architecture and designing specific tasks and loss functions to integrate multimodal interactions (Xu et al., ; Lu et al., ; Sun et al., ; Chen et al., ; Lu et al., ). Today multimodal learning includes different research lines, from generative multimodal learning (Suzuki & Matsuo, ; Ramesh et al., ; Saharia et al., ; Wang et al., ; Alayrac et al., ) to multimodal representation learning (Zong et al., ; Tian et al., ). CoMM belongs to the latter category.

Self-supervised multimodal representation learning. Self-supervised learning aims to learn general representations through supervisory signals from the data itself (known as the pretext task) (Balestriero et al., ). In the multimodal context (Zong et al., ), self-supervised methods can be grouped according to their pretext task. Clustering-based methods (Alwassel et al., ; Asano et al., ), where cluster assignments are used as pseudo-labels to train the model and different modalities can be used as supervisory signals to each other; masking modeling (Mizrahi et al., ; Bachmann et al., ; Lu et al., ) methods that reconstruct pieces of information that have been masked from input data. In the multimodal case, masked modelling is used in a cross-modal way, by predicting missing information conditioned in other modalities. Contrastive methods (Radford et al., ; Jia et al., ) in the multimodal context have been mostly used to matched data from different modalities to obtain aligned representations. In this work, we take a different perspective for contrastive multimodal methods for learning multimodal representations and explore this family of approaches in the following paragraph.

Multimodal contrastive learning. Current methods in contrastive learning are inspired by the idea of multiview learning (Li et al., ) and have shown remarkable results in representation learning in unimodal tasks (Chen et al., ; He et al., ; Tian et al., ; Oord et al., ). The natural extension of these approaches to multimodal settings is to optimize a cross-modal contrastive loss (Radford et al., ; Alayrac et al., ; Jia et al., ). Other works have gone even further by introducing cross-modal and intra-modal contrastive objectives (Mu et al., ; Jain et al., ), or by adding other intra-modality regularization terms (Wang et al., ; Kim et al., ). However, these approaches are designed to learn redundant information, neglecting the contributions of uniqueness or synergy
. Recently, FactorCL (Liang et al., ) has proposed a solution to model shared and unique task-relevant information explicitly. Yet, the method relies heavily on assumptions that are hard to meet in practice; it proposes a factorized approximation of multimodal interactions that is prone to cumulative errors and does not model synergistic information. Alternatively, we propose CoMM, a contrastive multimodal approach that leverages simple ideas to capture unique, redundant and synergistic interactions.

## 7 Conclusions

Multisensory integration is at the core of human perception, allowing us to build coherent representations of our environment.
In this paper, we introduce CoMM, a contrastive multimodal method that enables the integration of multiple modalities in a single multimodal representation space. Unlike existing multimodal contrastive models, CoMM is designed to learn multimodal interactions beyond redundancy, through PID theory.
Our controlled experiments on the bimodal Trifeature dataset demonstrate that CoMM successfully learns redundant, unique and synergistic information. In real-life multimodal datasets from Multibench with two and three input modalities, CoMM outperforms existing methods by large margins in almost every case, showing the efficiency and versatility of CoMM to handle data across diverse domains (robotics, healthcare, affective computing, multimedia) and structures (time series, image, text, audio, tabular).

This work offers large avenues for future research on multimodal representation learning, in particular for crafting label-preserving multimodal augmentations not limited to unimodal augmentations. Limitations and perspectives for future research are further discussed in Appendix A. Overall, the simplicity and versatility of CoMM’s design make it a good candidate to learn deep representations of several modalities across domains. It offers promises in better solving real-world problems ranging from neuroscience (Preti & Van De Ville, ) and medical imaging (Boehm et al., ) to remote sensing (Gómez-Chova et al., ).

## Appendix A Limitations and future research

- •
CoMM’s theoretical analysis for more than two modalities is still unclear since PID theory is currently limited to $n=2$ modalities (see Section 2 in the main paper). While uniqueness can be easily defined for any $n\geq 2$, the number of interactions between modalities (e.g. redundancy and synergy) grows exponentially with $n$, making the analysis harder. Nonetheless, we should recall that CoMM performs empirically very well even for $n>2$ (see Section 4.3), and the number of loss terms increases only linearly with $n$ (see Eq. 7).
Another limitation is the additional computational cost associated with adding modality-specific encoders. A simple workaround is to use large pretrained (frozen) encoders (*e.g*. from CLIP (Radford et al., ) for vision and text or DINOv2 (Oquab et al., ) for vision) and to only tune a lightweight fusion transformer in CoMM, allowing a much faster training.
- •
CoMM’s computational cost for data augmentation is higher than in cross-modalities frameworks (such as CLIP). This is because for each data tuple $X=(X_{1},X_{2},\ldots,X_{n})$, CoMM needs to compute $X^{\prime}=t(X)$, while cross-modality methods feed $X$ directly into the neural network. A possible solution would be the implementation of a momentum encoder and a queue method, as in MoCo (He et al., ).
- •
Interpretability. Our experiments on the Trifeature dataset (see Section 4.1 in the main paper) show that CoMM can efficiently learn unique, redundant and synergistic information. However, it seems difficult to disentangle the contributions of these three interactions in the representation space. Disentanglement might be one direction for future work. However, other approaches to measure such quantities (given a dataset and a task) are emerging (Hu et al., ; Liang et al., ; ). Another interesting approach would be to use modality masking (already implemented and handled by CoMM) to analyze the contribution of each modality individually versus collectively.

We believe the above limitations are directions that were outside the scope of the current manuscript; however, they are exciting avenues for future research in multimodal representation learning.

## Appendix B Implementation details

For all experiments with CoMM, we use an attention-based fusion module that takes as input a sequence of embeddings. We use a 1-layer Transformer with 8 heads that applies self-attention to all inputs and we add a [CLS] learnable embedding at the beginning of the sequence. The embedding size depends on each dataset, detailed below.

### B.1 Encoder architectures by dataset

- •
Trifeature is composed of two visual modalities so we use AlexNet encoder for both modalities with $512$-d embedding space. For CoMM, we remove the last average pooling layer and we apply a linear patch embedding layer (Dosovitskiy et al., ) to the $6\times 6$ feature maps as latent converter. We then add fixed 2D sine-cosine positional embeddings (Dosovitskiy et al., ). For Cross, we use a linear projector for each encoder ($512$-d output space) and we optimize the CLIP loss (Radford et al., ). For Cross+Self, we apply a 3-layers MLP projection head to each encoder to optimize the SSL objectives as in SLIP (Mu et al., ) ($1024$-d hidden layers and $256$-d output space) and we use the same projectors and CLIP objective as for Cross.
- •
MIMIC contains tabular and time-series data, seen as two modalities. We use a 2-layers MLP ($10$-d hidden dimension, $10$-d output) as tabular encoder and a GRU ($512$-d hidden dimension) as time-series encoder (similarly to FactorCL (Liang et al., )). For tabular data, we use a feature tokenizer (Gorishniy et al., ) as latent converter with $512$-d embedding space and no converter for time-series.
- •
MOSI, UR-FUNNY and MUsTARD contain visual and textual modalities extracted from videos. Similarly to FactorCL (Liang et al., ), we use a 5-head Transformer with 5 layers for each modality with a $40$-d embedding space. We do not use latent converter in this case.
- •
MM-IMDb also contains visual and textual modalities but in their raw format. We use a ViT-B/32 (Dosovitskiy et al., ) image encoder pre-trained with CLIP (Radford et al., ) and a Sentence-BERT multilingual text encoder(^5^55[https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1)) pre-trained with CLIP and distilled with Sentence-BERT (Reimers & Gurevych, ). For CoMM, we consider the token embeddings given by the image and text encoders (frozen) and we do not use latent converters. For CLIP (Radford et al., ), we fine-tune the pre-trained encoders with their original architecture. For SLIP (Mu et al., ), we use the same pre-trained encoders as CLIP and we add a 3-layers visual projection MLP ($4096$-d hidden layers and $256$-d output space) to compute the SSL objectives.
- •
Vision&Touch has visual, force-torque and robot proprioception modalities available. For the binary contact prediction task, we only use visual and proprioception data for the experiments with two modalities and we encode images with a ResNet18 (He et al., ) ($512$-d output space) and proprioception data with a 5-layer MLP ($512$-d output space), as in the original paper (Lee et al., ). Force-torque data are encoded with a 5-layer causal convolutions network (512-d output space) in the experiments with three modalities. For CoMM, we remove the last average pooling layer and apply a patch embedding layer (Dosovitskiy et al., ) to the $4\times 4$ feature maps as latent converter. We consider the 1D feature vector of proprioception data as a 1-length sequence in the fusion module. For the end-effector regression task, we use visual and force-torque modalities. We also use ResNet18 as image encoder ($128$-d output space) and a 5-layer causal convolutions network ($128$-d output space) as force encoder (Lee et al., ). For CoMM, we add the same latent converter for images as in the previous task and we add a feature tokenizer (Gorishniy et al., ) for force-torque embeddings.

### B.2 Data augmentation by modality

For raw images (in Trifeature, MM-IMDb and Vision&Touch), we use the default SimCLR augmentations (Chen et al., ), which include RandomResizedCrop, ColorJitter, RandomGrayscale, GaussianBlur and RandomHorizontalFlip (from the PyTorch library).

For tabular data (in MIMIC and Vision&Touch), we add a random Gaussian noise to each component (assuming they are all continuous).

For time-series data (either extracted from videos as in MOSI, UR-FUNNY, MusTARD, from health recordings as in MIMIC or from force-torque readings as in Vision&Touch), we apply a random composition of Gaussian noise and random dropping between 0 and 80% of the sequence. We have compared several other strategies for this modality and we present the results in Appendix C.

For raw text (in MM-IMDb), we randomly mask 15% of input tokens by using a special [MASK] token as in BERT (Devlin et al., ).

### B.3 Latent converters by encoder architecture

- •
Transformer and GRU: the latent converter is the identity since the Transformer and GRU already output a sequence of embeddings;
- •
CNN: the latent converter is a patch embedding projection module originally defined in ViT (Dosovitskiy et al., ) that we apply to the features maps of the CNN;
- •
MLP: the latent converter is a feature tokenizer originally defined in (Gorishniy et al., ) for tabular data. The feature vector $h_{i}$ is transformed into sequential embeddings by applying feature-wise multiplication with a learnable matrix and we add a bias term. For proprioception and force-torque data, we simply consider them as a 1-length sequence in the fusion module.

### B.4 Experimental settings

We use AdamW optimizer (Loshchilov & Hutter, ) in all experiments and a learning rate $\alpha=3\times 10^{-4}$ for Trifeature (with weight decay $10^{-4}$), $\alpha=10^{-3}$ for MIMIC, MOSI, UR-FUNNY and MusTARD (with weight decay $10^{-2}$) and $\alpha=10^{-4}$ for MM-IMDb and Vision&Touch (with weight decay $10^{-2}$). For MM-IMDb, we also use a cosine scheduler with final value $10^{-6}$ and a warmup over 10 epochs. All models were optimized during 100 epochs. The critic in the InfoNCE losses in $\mathcal{L}_{\text{CoMM}}$ Eq. 7 is implemented as a 3-layer MLP ($512$-d hidden layers and $256$-d output space), similarly to the projection head in SimCLR (Chen et al., ). All experiments ran on a single V100 GPU with 32GB of memory.

#### Fine-tuning of CoMM.

For all downstream classification tasks, we use SupCon loss (Khosla et al., ) to fine-tune CoMM (with no additional parameters). In the case of multi-label classification with MM-IMDb, we use a linear head on top of CoMM and we optimize a cross-entropy loss for each label. For the regression task on Vision&Touch, we also use a linear head and we optimize the MSE loss. We systematically use early-stopping according to the validation accuracy in order to prevent over-fitting on the downstream tasks.

### B.5 Additional details for LLaVA-NeXT evaluation

We evaluate LLaVA-NeXT (Li et al., ) on MM-IMDb based on its answer to the following prompt: ”From the following plot: {…} and this poster image, give me all the movie genres it belongs to among the following list: {…}. Give me the answer as a list.” We formulate it as a close question with limited number of answers to be closer to the linear probing setting for representation learning models.

## Appendix C Additional experiments

In this section we present additional analysis of CoMM, including: a benchmark of several augmentation strategies for time-series data, ablation studies on our loss function on a real dataset, and supplementary results on the trifeatures dataset.

### C.1 Data augmentation strategy for time-series

Figure: Figure 6: Linear evaluation accuracy (%) on four datasets from MultiBench (Liang et al., ) when applying individual or composing data augmentations on the modalities. Diagonal entries correspond to one augmentation applied to all modalities while off-diagonal entries are a composition of two augmentations. Average is the average of all four matrices. Results are averaged over 5 independent runs (with different seeds) in each cell.
Refer to caption: extracted/5848597/images/matrix-perf-aug-multibenchv2.jpg

For time-series data like in MIMIC, MOSI, UR-FUNNY and MuSTARD, there is no consensus with respect to the best data augmentation strategy to apply for self-supervised contrastive learning. We benchmark several common augmentations for time-series data including random crop between 8% and 100% of the signal, random drop between 0% and 80% of the signal and adding Gaussian noise. We also designed two new multimodal augmentations: *multi-crop* (resp. *multi-drop*) consisting of cropping (resp. dropping) a signal across multiple modalities for time-aligned data (introducing consistency in the preserved multimodal signal).

We tested these augmentations along with their composition on four datasets from MultiBench (Liang et al., ) and we plot the results in Fig. 6. Overall, we find that composing Gaussian noise and random drop results in best performances across datasets and tasks. This is our default policy in our main experiments. Our proposed multi-drop and multi-crop augmentations can provide better results in some cases (for MIMIC and MuSTARD) but we select the same default augmentations for consistency across all datasets.

### C.2 Ablation studies on a real dataset

We performed the loss ablation study on MM-IMDb. Results follow the same tendency as in the Tri-features dataset (Fig. 5 main paper).

**Table 6: Ablation study of loss function contributions on MM-IMDb. $\mathcal{L}_{\text{CoMM}}$ allows to better capture multimodal interactions than each term separately.**
| Loss | weighted-f1 | macro-f1 |
| --- | --- | --- |
| $\sum_{i}\mathcal{L}_{i}$ | $60.71_{\pm 0.17}$ | $53.35_{\pm 0.37}$ |
| $\mathcal{L}$ | $54.94_{\pm 0.50}$ | $47.13_{\pm 0.56}$ |
| $\mathcal{L}_{\text{CoMM}}=\mathcal{L}+\sum_{i}\mathcal{L}_{i}$ | $\mathbf{61.48}_{\pm 0.18}$ | $\mathbf{54.63}_{\pm 0.22}$ |

### C.3 Design choices on Trifeatures experiments

Authors in (Hermann & Lampinen, ) experimented with Trifeatures on AlexNet and ResNet-50. Both architectures showed comparable results. Therefore, we chose to use AlexNet in our experiments. For completeness, we ran the same experiments on ResNet-50, which show that CoMM is the only model to learn all interactions, regardless of the architecture.

**Table 7: Linear probing accuracy of redundancy (shape), uniqueness (texture) and synergy (color and texture) on bimodal Trifeature dataset, with different backbone encoders. These results are complementary to Footnote 4.**
| Model | Redundancy | Uniqueness | Synergy |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Cross | C+S | CoMM | Cross | C+S | CoMM | Cross | C+S | CoMM |  |
| AlexNet | $100.0_{\pm 0.02}$ | $99.7_{\pm 0.2}$ | $99.9_{\pm 0.03}$ | $11.6_{\pm 0.9}$ | $86.9_{\pm 0.8}$ | $87.8_{\pm 1.6}$ | $50.0_{\pm 0.0}$ | $50.0_{\pm 0.03}$ | $71.9_{\pm 2.0}$ |
| ResNet | $100.0_{\pm 0.04}$ | $99.9_{\pm 0.04}$ | $99.9_{\pm 0.03}$ | $6.5_{\pm 0.7}$ | $96.2_{\pm 0.8}$ | $96.3_{\pm 1.3}$ | $50.0_{\pm 0.0}$ | $50.0_{\pm 0.0}$ | $75.0_{\pm 1.7}$ |

## Appendix D Pseudo-code

Algorithm 1 presents the pseudo-code for CoMM’s training. It is written in the general case when we have $n$ modalities. It is complementary to Fig. 3 (main paper), which depicts the case $n=2$. CoMM’s official implementation will be released upon acceptance of this work.

Figure: Algorithm 1 CoMM training algorithm

## Appendix E Datasets details

### E.1 Trifeature

The Trifeature dataset (Hermann & Lampinen, ) was introduced as a controlled environment to study the properties of vision neural networks and how they learn different features (shape, texture and color). Images contain one of ten shapes (triangle, square, circle, *etc*.), rendered in one of ten textures (solid, stripes, grid, *etc*.), in one of ten colors (red, green, blue, *etc*.). Shapes are rendered within a $128\times 128$ square, rotated at an angle drawn between $[-45^{\circ},45^{\circ}]$ and placed at a random position within a larger image ($224\times 224$), such that the shape is fully contained in the image. Then, an independently rotated texture and a color are applied.

In our experiments, we consider the bimodal version of this dataset as explained in Section 4.1 (main paper).

### E.2 MultiBench

All the following datasets are pre-processed as described in Liang et al. ().

- •
MIMIC (Medical Information Mart for Intensive Care) (Johnson et al., ) is a dataset comprising de-indentified clinical data related to patiens admitted to critical care units at a large Boston-area hospital between 2001 and 2012. It contains information about 53 423 hospital admissions, including 38 597 distinct patients. We use the data as provided by MultiBench (Liang et al., ), organized as in (Purushotham et al., ). There are two data modalities: first, a time series modality, composed by a set of medical measurements of a given patient taken every hour during 24 hours. Each measurement is a vector of size 12 (*i.e*., including 12 different measured numerical values). Second, a static modality, including medical information about the patient (age, gender, *etc*.), represented as a vector of size 5 (tabular data). As in (Liang et al., ), in our experiments we address the binary classification task of predicting whether a patient fits any disease in the ICD-9 code in group 7 (460-519). ICD-9 (International Statistical Classification of Diseases and Related Health Problems) codes are used to classify diseases and a variety of symptoms. Almost every health consition can be assigned a unique ICD-9 code group, where each group includes a set of similar diseases.
- •
MOSI (Zadeh et al., ) is a sentiment analysis dataset obtained from 2 199 YouTube video segments. Each sample consists of a video (visual frames), the corresponding audio and transcription (text). The original dataset evaluates sentiment intensities with continuous labels ranging from -3 to 3. We follow previous works (Liang et al., ) and consider the binary version of the labels (positive and negative), and the same data modalities for training: text and videos.
- •
UR-FUNNY (Hasan et al., ) is a dataset for humor detection in human speech. It was created from 1 866 TED talk videos, obtaining more than 16 000 samples (parts of the videos). Each sample in the dataset consists of videos (visual frames), audio and their transcrips (text). The task is binary classification (humor or non-humor sequence).
- •
MUsTARD (Castro et al., ) is a multimodal video dataset for automated sarcasm detection. It contains videos from popular television shows including Friends, The golden girls, The big bang theory and Sarcasmaholics anonymous. Each sample in the dataset correspond to an utterance composed of a video (visual frames), its corresponding audio and the transcription (text), labeled as sarcastic or non-sarcastic.
As previous works (Castro et al., ; Liang et al., ), we use the balanced partition consisting of 690 utterances. In our experiments with two modalities, we considered only text and vision.
- •
Vision&Touch (Lee et al., ) is a robot manipulation multimodal dataset including visual, force and proprioception data for a peg insertion task. The data is collected from a 7-DoF, torque-controlled Franka Panda robot, with a triangle peg attached to its end-effector. Its goal is to insert the peg into a triangle hole situated in a box attached to a table in front of the robot. By running a random policy (the robot takes random actions) and a heuristic policy (the robot attempts peg insertion), 150 trajectories are recorded, each of them consisting of 1 000 timesteps. These trajectories contain RGB images, depth maps, force, end-effector position and velocity. Following MultiBench (Liang et al., ), we consider two tasks on this dataset: (i) the binary task of predicting contact or no contact in the next timestep, and (ii) predicting the end-effector possition (measured in MSE).

### E.3 MM-IMDb

Multimodal IMDb (MM-IMDb) (Arevalo et al., ) is a multimodal dataset for movie genre prediction. It has been built from the Movielens 20M dataset (GroupLens research, ) by filtering out movies without poster image. Therefore, MM-IMDb comprises 25 959 movies along with their plot, poster, genres and additional metadata (*e.g*. year, language, writer, *etc*.). As in previous works (Arevalo et al., ; Liang et al., ; Pérez-Rúa et al., ), we consider posters (image) and plots (text) as input data modalities to perform the multilabel classification genre prediction task (23 categories, each movie can be assigned to several categories). Technically, MM-IMDb is part of the Multibench benchmark (Liang et al., ), however, since we used raw data instead of the proposed pre-processed features, we treat it as a separate dataset.

## Appendix F Details on processing times and model complexity

We include in Table 8 an analysis of the complexity of CoMM against CLIP and BLIP-2. As we can observe, the fusion module in CoMM adds a marginal computational cost to existing backbones without compromising speed.

**Table 8: Comparison of model complexity and processing times of different multimodal architectures.**
| Model | FLOPs | MACs | #Params | Fwd-latency |
| --- | --- | --- | --- | --- |
| CLIP | 251G | 126G | 222M | 488ms |
| CoMM (w/ CLIP) | 281G | 140G | 229M | 493ms |
| BLIP-2 | 9.22T | 4.61T | 1.17B | 14s |
| CoMM (w/ BLIP-2) | 9.48T | 4.74T | 1.17B | 15s |

## Appendix G Proofs

###### Proof 1 ( Lemma 1 )

Indeed, combining Assumption 1 (multiview redundancy) and equations for $I(X_{1};Y|X_{2})$ and $I(X_{2};Y|X_{1})$ from Eq. 2 (main paper) we obtain:

$$ $0<I(X_{1};Y|X_{2})+I(X_{2};Y|X_{1})=U_{1}+U_{2}+2S<2\varepsilon_{\text{info}}$ (8) $$

Since $\varepsilon_{\text{info}}$ is supposed to be small and all the terms $U_{1},U_{2},S\geq 0$, their contributions are negligible.

Thus, under the multiview redundancy assumption $I(X_{1},X_{2};Y)\approx R$.

$\blacksquare$

###### Proof 2 ( Lemma 2 )

Given data processing inequalities for the Markov chains $X\rightarrow X^{\prime}\rightarrow Z^{\prime}_{\theta}$ and $Z^{\prime}_{\theta}\rightarrow X\rightarrow Z_{\theta}$, we have:

$$ $I(Z_{\theta};Z^{\prime}_{\theta})\leq I(X,Z^{\prime}_{\theta})\leq I(X,X^{ \prime})$ (9) $$

The equality can be achieved, for example, by selecting $f_{\theta}(\cdot)=\text{Id}(\cdot)$, the identity function.

$\blacksquare$

###### Proof 3 ( Lemma 3 )

First, we prove that $I(Z^{\prime}_{\theta^{\star}};Y)=I(X^{\prime},Y)$.

Indeed, we have:

$$ $\displaystyle I(X^{\prime};Y)$ $\displaystyle=I(X^{\prime};Y;X)+I(X^{\prime};Y|X)$ $\displaystyle=I(Z^{\prime}_{\theta^{\star}};Y;X)$ (by lemma 1 in (Wang et al., 2022a ) ) $\displaystyle=I(Z^{\prime}_{\theta^{\star}};Y)-I(Z^{\prime}_{\theta^{\star}};Y |X)$ $\displaystyle=I(Z^{\prime}_{\theta^{\star}};Y)$ $\displaystyle\text{because }Z^{\prime}_{\theta^{\star}}=f_{\theta^{\star}}(t(X))$ (10) $$

Second, let $\mathcal{T}=\{t_{i}\}$ such that $X^{\prime}=t_{i}(X)=X_{i}$ and $Z^{\prime}_{\theta^{\star}}=f_{\theta^{\star}}(X_{i})=Z_{i}$ for $i\in\{1,2\}$ (with a slight abuse of notation). Thanks to the previous result (in 3) and by the consistency equations for $I(X_{i};Y)$ in Eq. 2 (main paper), the final result follows:

$$ $\displaystyle I(Z_{i};Y)$ $\displaystyle=I(Z^{\prime}_{\theta^{\star}};Y)$ $\displaystyle=I(X^{\prime};Y)$ because of 3 $\displaystyle=I(X_{i};Y)$ $\displaystyle=R+U_{i}$ because of consistency equations. (11) $$

$\blacksquare$

###### Proof 1 ( Lemma 1 )

###### Proof 2 ( Lemma 2 )

###### Proof 3 ( Lemma 3 )

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 1
presentation: 2
questions: See Weaknesses for questions about theoretical foundation of this paper.

In Lemma 2, it is shown that optimum is achieved assuming "an expressive enough network". Do we have any idea how big/complex the network needs to be to become an expressive enough network? How does performance scale with your network's size, etc.?
soundness: 1
strengths: This paper grounds multimodal representation learning in the theoretical framework of multimodal information theory. Lemma 2 & 3 offer insight into novel methods that can be applied to learn multimodal representations with beyond redundancy.
summary: The authors present a multimodal representation learning method that uses multimodal augmentations to learn all four multimodal interaction terms. The proposed optimization terms are designed to optimize for $R + U_i$ and $U_1 + U_2 + R + S$. The authors design a multimodal encoder+fusion architecture that achieve state-of-the-art performance on several MM benchmarks.
weaknesses: I have strong doubts about the soundness of the theoretical foundation of this paper:
1. Assumption 1 assumes the existence of a multimodal augmentation $t$ such that $I(X; t(X)) = I(X; Y)$. This means that if $X$ contains more information than the label $Y$, $t(X)$ has to "reduce" the information in $X$ down to $Y$. This is different from just "label-preserving multimodal augmentation". The experiments section also offers little insight about how exactly this augmentation is carried out. The augmentations mentioned for real-world datasets, including cropping, jitter, etc., looks like they certainly would lead to $I(X; t(X)) \gg I(X; Y)$. This would be a direct contradiction of Assumption 1.
2. Moreover, for the pretraining paradigm used on real-world datasets, we don't even know at pretraining time what the task $Y$ is yet. Correct me if I'm wrong--it's not clear to me how it is possible for this assumption to hold.
3. The real gains from this method seems to be the aggressive mutual information optimization between all pairs of single modalities and all modalities as well as all data augmentations.

It would be helpful if the authors could provide more rigorous justification for Assumption 1. For example, why does Assumption 1 hold in the real world?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Regard weakness 1, answering the following questions can help differentiate CoMM from existing contrastive methods and highlight the necessity for going beyond modeling redundancy:
1. How applicable is the Multi-view redundancy assumption in the realistic settings?
2. What makes it necessary to go beyond learning redundancy under the Multi-view redundancy assumption? Why are existing contrastive modeling approaches that focus on primary redundancy interaction insufficient?
3. Without the Multi-view redundancy assumption, i.e. for other tasks, do those existing contrastive learning framework still capture primarily redundancy?

Regard weakness 2, why is the Label-preserving multimodal augmentations (Assumption 1) considered a weaker and more realistic assumption? How can the mentioned mutual information quantity be measured in a realistic setting in order to choose the right augmentations?

Regard weakness 3, the reviewer expects to see more direct evidence that shows the superior modeling of synergistic interaction of CoMM compared to the existing strong methods (e.g. FactorCL) on real-world datasets.

Regard weakness 4, answering the following questions can provide additional insights into how the proposed framework better models synergy:
1. How should a 50% probing accuracy for synergy be interpreted? Is it only based on presence/absence?
2. What is the intuition for the crop augmentation being the key to learning synergy?

Regard weakness 5, please add more details on synthetic experiment setup as previously suggested.
soundness: 2
strengths: 1. **Novel multimodal interaction learning under a contrastive learning framework**: the proposed CoMM framework is both theoretically and empirically shown to be able to capture uniqueness and (especially) synergy better than prior work.
2. **Comprehensive evaluation and careful ablation**: the proposed method is evaluated on a wide range of multimodal benchmarks, which involves diverse tasks, all requiring different level modeling of interactions. The paper conducts careful ablation in analyzing the importance of each part of the loss in capturing different interactions, which has provided insights in how those interactions are learnt with the proposed framework.
summary: This paper proposed CoMM, a Contrastive Multimodal learning framework, that captures multimodal interactions and aligns multimodal representations by maximizing the mutual information between augmented versions of these multimodal features. The paper shows that, under certain data augmentation, this framework can capture uniqueness and synergy better than existing work. Empirically, it is also shown that such framework facilitates multimodal learning on various multimodal benchmarks.
weaknesses: 1. **Need for clarification about the Multi-view redundancy assumption**: the paper introduces the Multi-view redundancy assumption (Definition 1) to highlight the insufficiency of several existing works that propose cross-modal contrastive learning. Specifically, the paper shows that under this assumption, which states that "most task-relevant information is shared across modalities", cross-modal contrastive learning is primarily learning the redundancy interaction while ignoring the others interactions (uniqueness and synergy). The reviewer has the following questions:

    1. it is unclear how applicable this assumption is in the realistic settings: in fact, as many prior work estimating multimodal interaction in real-world multimodal datasets show, multimodal tasks can vary by a diverse range of combinations of those interactions;

    2. given this assumption, it seems that capturing redundancy would be sufficient already to learn task-relevant information, what makes it necessary to go beyond learning redundancy;

    3. do those existing contrastive learning frameworks still capture primarily redundancy, i.e. does Lemma 1 still hold, without the multi-view redundancy assumption?

2. **Strong assumption of Label-preserving multimodal augmentations (Assumption 1)**: the paper points out that existing methods like FactorCL requires strong and often unrealistic assumption about optimal multimodal augmentations based on conditional augmentations. Instead, the paper relaxes the assumption to label-preserving multimodal augmentations, which is still considered strong, as it is hard to exactly measure the mutual information between the original data and its augmented version $I(X,X')$, as well as between the data and its label $I(X,Y)$ ensuring $I(X,X')=I(X,Y)$ holds in a realistic setting. In fact, in the evaluation, the reviewer does not find any evidence to support that when evaluating on the real-world datasets, the augmentation indeed preserves the label. This not only makes the theory and the experiments detached, but also makes the choice of the "right" augmentation unclear, which is essential to learn the correct objective.
3. **Indirect evidence of CoMM's superior learning of synergy**: the paper highlights the novelty of the proposed framework, CoMM, in better capturing the synergistic interaction than existing methods. However, except Figure 4 (which does not include comparison with the most similar work FactorCL), which is shown on synthetic dataset, the paper does not provide any other direct evidence for showing the claimed superior learning of synergy when evaluating on real-world datasets. For example, a large margin of performance when evaluating on datasets with primary synergistic interaction (and minimal other types of interaction), for example, VQA, will constitute strong evidence to support the claim. Also, it'll be useful to provide estimates of the proportion of different types of interaction within each real-world datasets for evaluation to better understand the advantages of CoMM for modeling different combinations of interactions.
4. **Mysterious number of 50% accuracy for probing synergy**: except for the probing accuracy of synergy for CoMM being 71.87% (which gets round up to 71.9% in Figure 4), the probing accuracy of synergy for all other methods are all 50%. Does this number represent random chance? In other words, how is the probing accuracy of synergy evaluated? Is it only based on presence/absence? In Table 5, it is shown that without the crop augmentation for either modality 1 or modality 2, the synergy interaction seems not to be learnt. What is the intuition for the crop augmentation being the key to learning synergy? Overall, the reviewer feels an insufficient exploration and understanding of modeling synergy (the highlighted novelty of the paper) in the proposed framework, except for a little bit of discussion in the Ablation section (loss function), which only provides a hypothetical explanation.
5. **Missing details about the synthetic experiment setups**: the reviewer does not find documentation about how redundancy, uniqueness, and synergy are defined for the Trifeature dataset or how to control the amount of each type of interaction, in the main text or in the appendix. The paper should provide more details about the synthetic experiment setups, since the original Trifeature dataset is not designed for multimodal learning but rather for feature learning.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - **1**: The authors claim in the introduction that they evaluate CoMM across "diverse data types (images, text and audio)". However, from my understanding, the datasets used in the MultiBench benchmark are composed of pre-processed features extracted from the original, high-dimensional modalities. Am I correct? Given the use of pre-processed, low-dimensional, features, instead of raw data, it might be a stretch that claiming that the authors evaluate CoMM on audio data. Furthermore, how do you apply label-preserving multimodal transformations to these feature features?
- **2**: How to define the set of label-preserving multimodal transformations? Is it task-dependent? How large does this set need to be to work for your framework. Perhaps a section, even if in appendix, discussing these questions would be quite interesting, especially for practitioners.
- **3**: In Section 4.1., I found it interesting that there is no comparison with FactorCL, since this comparison would emphasize the relevance of the contribution (since, to my understanding, FactorCL cannot capture the synergetic interactions in the dataset). How does FactorCL perform in this controlled experiment?
- **4**: In Table 1, CoMM outperforms the other approaches across most datasets except for MIMIC. Any reason for this? Is there any special feature of the MIMIC dataset?
- **5**: In Section 5, Fusion module, did you also compare to a Concat+Non-linear fusion mechanism?
soundness: 4
strengths: - **Originality**:
  - As highlighted by the authors, the use of self-supervision contrastive objectives to learn multimodal representations is not particularly novel (see, for example [1-4] for examples in fully-training and fine-tuning settings). However, the authors focus on the problem of learning not only redundant interactions between the modalities, but also the unique and synergetic interactions. This work seems to follow in line with previous work (FactorCL [6]), extending it to account for synergetic interactions and relaxing the factorization assumption and the need for the use of optimal multimodal augmentations. The use of the Trifeature dataset to quantitively evaluate the effects of redundant, unique and synergetic interactions is quite ingenious and, to the best of my knowledge, novel.

- **Quality**:
   - The work presented here is undoubtedly of high quality. The authors present in depth their main contribution, CoMM, and ground it in the theory of partial information decomposition. The theoretical results, even if not particularly extensive, give a strong support to the proposed method. Furthermore, the evaluation shown in this paper is also of high-quality: the authors evaluate their approach in both controlled and more complex environments, against a wide range of baselines, and the results mostly support the claims of the authors (see "Questions" for more details and experimental clarifications). 

- **Clarity**:
   - The current version of the work is well-written, with no major typos (that I could detect). The authors employ a clear notation throughout the document. Furthermore, the document is easy to read, also due to the intuitive figures created by the authors (the only exception being Figure 1, where it is not clear what does $\mathcal{M}$ refers to). I would, however, recommend that the authors place the "Limitations and Future Work" section of the Appendix in the main document, as it clarifies the current limitations of the method (which is not something necessarly negative) for the reader.

- **Significance**:
   - This work proposes a novel multimodal learning objective that is able to capture a wide range of interactions between multimodal data for downstream tasks. Given the relevance of multimodal representation learning for the current field of AI, the work can have significant impact in both researchers and practitioners.


**References**: 

- [1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.
- [2] Zhai, Xiaohua, et al. "Sigmoid loss for language image pre-training." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
- [3] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International conference on machine learning. PMLR, 2022.
- [4] Poklukar, Petra, et al. "Geometric multimodal contrastive representation learning." International Conference on Machine Learning. PMLR, 2022.
- [5] Liang, Paul Pu, et al. "Factorized contrastive learning: Going beyond multi-view redundancy." Advances in Neural Information Processing Systems 36 (2024).
summary: This work introduces a novel self-supervised learning objective for multimodal representation learning that captures the redundant, unique and synergetic interactions between different modalities. Their proposed approach (CoMM) is composed of a multimodal architecture, with modality-specific encoders and an attention-based fusion mechanism, that encodes a multimodal representation and a novel contrastive learning objective, grounded on the theory of partial information decomposition. The authors extensively evaluate CoMM against other self-supervised objectives, considering datasets of different complexities and number of modalities. The results highlight that the proposed method: (i) is able to capture the proposed interactions; (ii) outperforms other learning objectives in both self-supervised and fine-tuning settings; (iii) can be employed on scenarios with complex modalities (such as images and text) and with more than 2 modalities. Finally, the authors present an ablation study focused on the loss function, fusion module and data augmentation.
weaknesses: My two main concerns with the current version of the work are the following:
- From an architectural point-of-view, the work presents little novelty: the multimodal fusion mechanism, based on concatenation and a transformer block, is also not particularly novel (see, for example, [1], where the authors discuss several works that employ concatenation for multimodal inputs in transformer architectures). Also, despite introducing some novel ideas, the work is still a continuation of FactorCL [2], where the use of partial information decomposition theory was also explored to derive a suitable learning objective (even if a different one from the one proposed).
- The majority of evaluations presented by the authors are in controlled datasets (Section 4.1) or simplified environments (Section 4.2 - Multibench, Section 4.3), where the high-dimensional raw data are preprocessed into low-dimensional feature vectors (as described by the authors on Section 4.2). As such, it is not clear how the learned representations by the proposed method would work on more complex, real-world tasks (beyond the Multimodal IMDB dataset), for example on the ones proposed in [3].


**References**:

[1] - Xu, Peng, Xiatian Zhu, and David A. Clifton. "Multimodal learning with transformers: A survey." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.10 (2023): 12113-12132.

[2] - Liang, Paul Pu, et al. "Factorized contrastive learning: Going beyond multi-view redundancy." Advances in Neural Information Processing Systems 36 (2024).

[3] - Ying, Kaining, et al. "MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI." Forty-first International Conference on Machine Learning.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: (Not important, just out of curiosity) Why is MM-IMDB not listed as part of MultiBench? (Since MM-IMDB is included in MultiBench)
soundness: 4
strengths: 1. The authors proposed a novel method for an important problem (self-supervised multimodal representation learning), CoMM, that outperformed all baselines in almost all tasks. CoMM also only adds a marginal computation overhead compared to baselines.

2. The authors theoretically justified why CoMM works, and confirmed their theory with experiments and analyses on the synthetic dataset.

3. The paper's presentation quality is generally good. No significant grammar errors and the figures/tables are well-made.

4. The authors justified their design choices through ablation studies on both synthetic and real datasets.
summary: In this paper, the authors proposed CoMM, a novel multimodal contrastive representation learning method that performs multiple InfoNCE contrastive losses between unimodal representations and augmented multimodal representations in the same space. The authors first theoretically justified their approach through information theory and breaking down multimodal interactions into Redundancy, Uniqueness and Synergy. The authors evaluated their proposed method first on a synthetic dataset to show that CoMM can better capture the interactions, and then evaluated CoMM against other self-supervised representation learning methods on many datasets from MultiBench and MM-IMDB with linear evaluation classification accuracy, and CoMM outperformed all baselines. The authors also conducted ablation studies on the training objective and architecture of CoMM on the synthetic dataset.
weaknesses: Minor presentation issue: In section 4.2, the title claims that all tasks in this subsection should be 2-modal, but in lines 356-358 some of the video-affect tasks are 3-modal. It is unclear which 2 modalities out of 3 were used for this experiment, and it can be quite confusing (especially on UR-funny which occurred in both the 2-modal experiments and the 3-modal experiments).

Meta-review: This paper proposes CoMM, a Contrastive Multimodal learning framework, that learns multimodal representations by maximizing the mutual information between augmented versions of these multimodal features. They prove that this framework can capture uniqueness and synergy better than existing work, and also show empirically that it leads to strong multimodal learning performance on several benchmarks.

After the discussion period, 3 reviewers voted to accept the paper, citing its strengths in theory and practice, strong results across several datasets, and broad applicability for many multimodal scenarios where capturing redundant, unique, and synergistic features are important.

The only concern blocking the paper was raised by J1jX and ZXRV regarding the feasibility of Assumption 1. In response, the authors provided empirical evidence supporting this assumption, and added detailed discussions about it in the paper. Reviewer J1jX still recommended rejection after the discussion period, but I believe that the paper's strong methodological and empirical results still warrant presentation at ICLR. Furthermore, as reviewer J1jX pointed out, I encourage the authors to significantly rephrase/relax/clarify Assumption 1, since the rest of the paper actually does not depend on it - the assumption is technically true as an existential statement, the only issue is with practically reaching such a multimodal representation using neural network approximation. I therefore lean with the 3 positive reviews and recommend acceptance.

Decision: Accept (Poster)