Title: Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
Venue: NeurIPS 2023
Accepted: yes
Keywords: multimodal learning

contrastive learning

self-supervised learning

information theory
Abstract: In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data with only pairing information (e.g., image-caption or video-audio pairs). Underpinning these approaches is the assumption of multi-view redundancy - that shared information between modalities is necessary and sufficient for downstream tasks. However, in many real-world settings, task-relevant information is also contained in modality-unique regions: information that is only present in one modality but still relevant to the task. How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks? This paper proposes FactorCL, a new multimodal representation learning method to go beyond multi-view redundancy. FactorCL is built from three new contributions: (1) factorizing task-relevant information into shared and unique representations, (2) capturing task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bounds, and (3) multimodal data augmentations to approximate task relevance without labels. On large-scale real-world datasets, FactorCL captures both shared and unique information and achieves state-of-the-art results on six benchmarks.

FACTORIZED CONTRASTIVE LEARNING: Going Beyond Multi-view Redundancy

Abstract

In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data with only pairing information (e.g., image-caption or video-audio pairs). Underpinning these approaches is the assumption of multi-view redundancy - that shared information between modalities is necessary and sufficient for downstream tasks. However, in many real-world settings, task-relevant information is also contained in modality-unique regions: information that is only present in one modality but still relevant to the task. How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks? This paper proposes FACTORCL, a new multimodal representation learning method to go beyond multi-view redundancy. FACTORCL is built from three new contributions: (1) factorizing task-relevant information into shared and unique representations, (2) capturing task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bounds, and (3) multimodal data augmentations to approximate task relevance without labels. On large-scale real-world datasets, FACTORCL captures both shared and unique information and achieves state-of-the-art results on six benchmarks.

1 Introduction

Learning representations from different modalities is a central paradigm in machine learning [48]. Today, a popular learning method is to first pre-train general representations on unlabeled multimodal data before fine-tuning on task-specific labels [10, 40, 47, 48, 50]. These current multimodal pretraining approaches have largely been inherited from prior work in multi-view learning [13, 58] that exploit a critical assumption of multi-view redundancy : the property that shared information between modalities is almost exactly what is relevant for downstream tasks [70, 73, 76]. When this assumption holds, approaches based on contrastive pre-training to capture shared information [13, 41, 61, 72], followed by fine-tuning to keep task-relevant shared information [76], have seen successful applications in learning from images and captions [61], video and audio [3], speech and transcribed text [58], and instructions and actions [21]. However, our paper studies two fundamental limitations in the application of contrastive learning (CL) to broader real-world multimodal settings (see Figure 1 for a visual depiction and experimental results showing the performance drop of CL):

Low shared information relevant to tasks: There exists a wide range of multimodal tasks involving small amounts of shared information, such as between cartoon images and figurative captions (i.e., not literal but metaphoric or idiomatic descriptions of the images [52, 88]). In these situations, standard multimodal CL will only receive a small percentage of information from the learned representations and struggle to learn the desired task-relevant information.

High unique information relevant to tasks: Many real-world modalities can provide unique information not present in other modalities. Examples include healthcare with medical sensors or robotics with force sensors [45, 49]. Standard CL will discard task-relevant unique information, leading to poor downstream performance.

1.00

Task-relevant

sharedinformation

0.98

S = I(X1;X2; Y)

0.96

X1

(modality 1)

(modality 2)

0.94

Ours

SupCon

0.92

SimCLR

U = I(X; Y|X2)

U2 = I(X2; Y|X1)

Cross+Self

Task-relevant

(task)

Task-relevant

0.90

0.0

0.2

0.4

0.6

0.8

1.0

uniqueinformation

uniqueinformation

(U1 + U2)/(S + U1 + U2)

In light of these limitations, how can we design suitable multimodal learning objectives that work beyond multi-view redundancy? In this paper, starting from the first principles in information theory, we provide formal definitions of shared and unique information via conditional mutual information and propose an approach, FACTORIZED CONTRASTIVE LEARNING (FACTORCL for short), to learn these multimodal representations beyond multi-view redundancy using three key ideas. The first idea is to explicitly factorize shared and unique representations. The second idea is to capture task-relevant information via maximizing lower bounds on MI and remove task-irrelevant information via minimizing upper bounds on MI, resulting in representations with sufficient and necessary information content. Finally, a notion of task relevance without explicit labels in the selfsupervised setting is achieved by leveraging multimodal augmentations . Experimentally, we evaluate the effectiveness of FACTORCL on a suite of synthetic datasets and large-scale real-world multimodal benchmarks involving images and figurative language [88], prediction of human sentiment [91], emotions [93], humor [27], and sarcasm [12], as well as patient disease and mortality prediction from health indicators and sensor readings [38], achieving new state-of-the-art performance on six datasets. Overall, we summarize our key technical contributions here:

A new analysis of contrastive learning performance showing that standard multimodal CL fails to capture task-relevant unique information under low shared or high unique information cases.

A new contrastive learning algorithm called FACTORCL:

(a) FACTORCL factorizes task-relevant information into shared and unique information, expanding contrastive learning to better handle low shared or high unique information.

(b) FACTORCL optimizes shared and unique information separately, by removing task-irrelevant information via MI upper bounds and capturing task-relevant information via lower bounds, yielding optimal task-relevant representations.

(c) FACTORCL leverages multimodal augmentations to approximate task-relevant information, enabling self-supervised learning from our proposed FACTORCL.

2 Analysis of Multi-view Contrastive Learning

We begin by formalizing definitions of four types of information: shared, unique, task-relevant, and task-irrelevant information in multimodal data. To formalize the learning setting, we assume there exist two modalities expressed as random variables X 1 and X 2 with outcomes x 1 and x 2 , and a task with the random variable Y and outcome y . We denote X -i as the other modality where appropriate.

Shared and unique information : We formalize shared and unique information by decomposing the total multimodal information I ( X 1 , X 2 ; Y ) into three conditional mutual information (MI) terms:

where I ( X 1 , X 2 ; Y ) = ∫ p ( x 1 , x 2 , y ) log p ( x 1 ,x 2 ,y ) p ( x 1 ,x 2 ) p ( y ) dx 1 dx 2 dy is the total MI between the joint random variable X 1 , X 2 and the task Y , S = I ( X 1 ; X 2 ; Y ) = I ( X 1 ; X 2 ) -I ( X 1 ; X 2 ∣ Y ) = ∫ p ( x 1 , x 2 ) log p ( x 1 ,x 2 ) p ( x 1 ) p ( x 2 ) dx 1 dx 2 -I ( X 1 ; X 2 ∣ Y ) is the task-relevant shared in-

formation, I ( X 1 ; X 2 ∣ Y ) = ∫ p ( x 1 , x 2 , y ) log p ( x 1 ,x 2 ∣ y ) p ( x 1 ∣ y ) p ( x 2 ∣ y ) dx 1 dx 2 dy is the task-irrelevant shared information, and U 1 = I ( X 1 ; Y ∣ X 2 ) , U 2 = I ( X 2 ; Y ∣ X 1 ) denote unique task-relevant information.

Limitations of CL : Current approaches for CL maximize mutual information I ( X 1 ; X 2 ) (and subsequently task-relevant shared information I ( X 1 ; X 2 ; Y ) during supervised fine-tuning), without modeling unique information. These methods generally learn a pair of representations [73, 76],

For example, Z 1 could encode images X 1 and Z 2 encodes text X 2 via maximizing a lower bound on I ( X 1 ; X 2 ) using the NCE objective [58]. The NCE objective falls into a broader class of contrastive learning methods [13, 15, 28, 41, 61] that model the ratio between joint densities p ( x 1 , x 2 ) and product of marginal densities p ( x 1 ) p ( x 2 ) using positive and negative samples [57, 59, 60, 79, 84] or probabilistic classifiers [55, 77]. It has been shown that contrastive learning works well under the assumption of multi-view redundancy [4, 31, 70, 71, 76]:

Definition 1. (Multi-view redundancy) ∃ ϵ > 0 such that I ( X 1 ; Y ∣ X 2 ) ≤ ϵ and I ( X 2 ; Y ∣ X 1 ) ≤ ϵ .

In other words, the task-relevant information in data is mostly shared across both views and the unique information is at most a small ϵ . From a representation perspective, Tian et al. [72] further introduces the assumption that the optimal representation is minimal and sufficient, where all learned task-relevant information is shared information: I ( Z 1 ; Y ∣ X 2 ) = I ( Z 2 ; Y ∣ X 1 ) = 0 . While the multiview redundancy is certainly true for particular types of multimodal distributions, it crucially ignores settings that display multi-view non-redundancy and unique information can be important, such as when health indicators, medical sensors, and robotic visual or force sensors each provide unique information not present in other modalities [45, 49].

Definition 2. (Multi-view non-redundancy) ∃ ϵ > 0 such that I ( X 1 ; Y ∣ X 2 ) > ϵ or I ( X 2 ; Y ∣ X 1 ) > ϵ .

Under multi-view non-redundancy, we show that standard CL only receives a weak training signal since it can only maximize a lower bound on shared information I ( X 1 ; X 2 ) , and struggles to learn task-relevant unique information. We formalize this intuition with the following statement:

Theorem 1. (Suboptimality of standard CL) When there is multi-view non-redundancy as in Definition 2, given optimal representations { Z 1 , Z 2 } that satisfy Eq.(2 and I ( Z 1 ; Y ∣ X 2 ) = I ( Z 2 ; Y ∣ X 1 ) = 0 [72], we have that

Correspondingly, the Bayes error rate P e ( Z 1 , Z 2 ) ∶= 1 -E p ( z 1 ,z 2 ) [ max y ∈ Y P ( ˆ Y = y ∣ z 1 , z 2 )] of contrastive representations { Z 1 , Z 2 } for a downstream task Y is given by:

We include proofs and a detailed discussion of the assumptions in Appendix B. Based on Eq.(3), I ( Z 1 , Z 2 ; Y ) decreases with higher task-relevant unique information I ( X 1 ; Y ∣ X 2 ) and I ( X 2 ; Y ∣ X 1 ) ; we call this the difference I ( X 1 , X 2 ; Y ) -I ( Z 1 , Z 2 ; Y ) the uniqueness gap . The uniqueness gap measures the loss in task-relevant information between the input and encoded representation: as task-relevant unique information grows, the uniqueness gap increases. In addition, I ( Z 1 , Z 2 ; Y ) also drops with lower I ( X 1 ; X 2 ) (i.e., two modalities sharing little information to begin with), or with higher I ( X 1 ; X 2 ∣ Y ) (i.e., when the shared information is mostly task-irrelevant). Similarly, in Eq.(5), the Bayes error rate of using { Z 1 , Z 2 } for prediction is directly related to the task-relevant information in { Z 1 , Z 2 } : error on the downstream task increases with higher unique information and lower shared information.

3 FACTORIZED CONTRASTIVE LEARNING

We now present a suite of new CL objectives that alleviate the challenges above and work at all ranges of shared and unique information. At a high level, we aim to learn a set of factorized representations Z S 1 , Z S 2 , Z U 1 , Z U 2 representing task-relevant information in X 1 shared with X 2 , in X 2 shared with X 1 , unique to X 1 , and unique to X 2 respectively. As common in practice [61, 72], we define

X

X

X

X

Augmentation

X1

Factorized Contrastive Learning

ZU

arg max I(Z1; Y|X2) = INcE(X1; X) - INcE-CLUB(X, X2) + INcE(X; X2|X), X2)

Y

Y

X1

Z1

<-

I

X

X2

X

X2

X

X2

Lowerbound

Upperbound

Augmentation

X2

Zs,Zs

=

arg max I(Z1; Z2; Y) = INce(X; X2)

Y

)- INCE-CLUB(X;X2IX1,X2)

Unique

Z1.Z2

X

X2

X

X2

X

X2

X1

X2

X2

Y

Y

Y

=

arg max I(Z2; Y|X1) = INcE(X2;X2) -

INCE-CLuB(X1, X2) + INcE(X; X2IX,X2)

neural networks f θ with trainable parameters θ to extract representations from inputs X 1 and X 2 . Learning these parameters requires optimizing differentiable and scalable training objectives to capture task-relevant shared and unique information (see overview in Figure 2):

where I ( Z 1 ; X 2 ; Y ) = I ( Z 1 ; X 2 ) -I ( Z 1 ; X 2 ∣ Y ) is the shared information and I ( Z 2 ; X 1 ; Y ) = I ( Z 2 ; X 2 ) -I ( Z 2 ; X 1 ∣ Y ) is the unique information. One important characteristic of our framework is that when unique information is zero: I ( X 1 ; Y ∣ X 2 ) = 0 and I ( X 2 ; Y ∣ X 1 ) = 0 , or all shared information is task-relevant: I ( X 1 ; X 2 ; Y ) = I ( X 1 ; X 2 ) , our framework recovers standard CL as in Eq.(2). However, as we have previously indicated and will show empirically, these assumptions can easily be violated, and our framework enlarges Eq.(2) to cases where unique information is present.

The learned Z s can then be used as input to a linear classifier and fine-tuned to predict the label for multimodal classification or retrieval tasks. However, the shared and unique MI terms above are often intractable in practice. In the next section, we will build up our method step by step, eventually showing that each term in Eqs.(6- 7) can be approximated as follows:

where I NCE and I NCE-CLUB are scalable contrastive estimators (Section 3.1) and X ′ 1 , X ′ 2 are suitable data augmentations (Section 3.2) on each modality. Overall, these equations can be interpreted as both positive and negative signals to learn representations for S and U . For shared information S , the estimator maximizes task-relevant shared information via I NCE ( X 1 ; X 2 ) while removing task-irrelevant shared information via a novel upper bound -I NCE-CLUB ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) . For unique information U i , we capture task-relevant uniqueness via + I NCE ( X i ; X ′ i ) while non-unique information is removed via -( I NCE-CLUB ( X 1 ; X 2 ) -I NCE ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 )) . In the following sections, we derive this final objective step-by-step: (1) approximating the MI objectives in S and U with CL estimators, (2) relaxing the dependence on labels Y with self-supervised data augmentations, finally (3) discussing overall training and implementation details of end-to-end self-supervised learning.

3.1 Supervised FACTORCL with shared and unique information

To capture shared and unique information via an objective function, we will need to maximize lower bounds for all terms with a positive sign in Eq.(8) and (9) ( I ( X 1 ; X 2 ) , I ( X i ; Y ) , I ( X 1 ; X 2 ∣ Y )) and minimize upper bounds for all terms with a negative sign ( I ( X 1 ; X 2 ) , I ( X 1 ; X 2 ∣ Y )) . Our first theorem derives general lower and upper bounds for MI terms as variants of contrastive estimation:

0

2

4

6

8

10

12

14

16

Mutual Information

True MI

NCE

CLUB

NCE

CLUB

Gaussian, dim=50

Gaussian, dim=100

Gaussian, dim=200

Theorem 2. (Contrastive estimators for I ( X 1 ; X 2 ) ) Defining the NCE and NCE-CLUB estimators,

where f ∗ ( x 1 , x 2 ) is the optimal critic from I NCE plugged into the I CLUB objective [16]. We call the proposed plug-in objective Eq.(11) I NCE-CLUB , and obtain lower and upper bounds on I ( X 1 ; X 2 ) :

Proof. The lower bound I NCE ( X 1 ; X 2 ) ≤ I ( X 1 ; X 2 ) follows from Oord et al. [58]: optimizing the objective leads to an optimal critic [60] f ∗ = log p ( x 1 ∣ x 2 ) + c ( x 1 ) , with a deterministic function c (⋅) . Plugging optimal critic f ∗ into I NCE-CLUB ( X 1 ; X 2 ) cancels out the c ( x 1 ) term and yields I NCE-CLUB ( X 1 ; X 2 ) and I ( X 1 ; X 2 ) ≤ I NCE-CLUB. We include a detailed proof in Appendix C.1.

I NCE-CLUB ( X 1 ; X 2 ) gives a desired upper bound of I ( X 1 ; X 2 ) 'for free' while avoiding separately optimizing lower bound and upper bounds. In Figure 3, we show these two bounds in practice across two Gaussian distributions X 1 and X 2 with varying amounts of MI I ( X 1 ; X 2 ) . We use the second formulation of I CLUB [16], which assumes p ( x 1 ∣ x 2 ) to be unknown. Our upper bound is empirically tighter (see Figure 3) and comes for 'free' via jointly maximizing the lower bound I NCE. These lower and upper bounds can be seen as new contrastive objectives over positive and negative ( x 1 , x 2 ) pairs, enabling a close integration with existing pre-training paradigms. Finally, we can similarly obtain bounds for the conditional MI I NCE ( X 1 ; X 2 ∣ Y ) ≤ I ( X 1 ; X 2 ∣ Y ) ≤ I NCE-CLUB ( X 1 ; X 2 ∣ Y ) :

These two bounds result in conditional CL objectives [51, 74, 78] - they differ critically from standard CL methods since they capture task-irrelevant shared information that remains between X 1 and X 2 after observing Y . This task-irrelevant shared information is removed by minimizing its upper bound. Note that f ( x 1 , x 2 , y ) here denotes a different function from f ( x 1 , x 2 ) in Eq.(10), as the general forms are different (taking in x 1 , x 2 versus x 1 , x 2 , y ). f ( x 1 , x 2 , y ) can be implemented in different ways, e.g., g ([ x 1 , y ]) T h ( x 2 ) where g () , h () are trainable encoders and [ x 1 , y ] denotes concatenation [69].

3.2 Self-supervised FACTORCL via multimodal augmentations

The derivations above bring about supervised CL objectives with access to Y [41]. For unsupervised CL [58, 72], we derive similar objectives without access to Y by leveraging semantic augmentations on each modality. Denote X ′ as some augmentation of X (e.g., rotating, shifting, or cropping). Under

the optimal augmentation assumption from Tian et al. [72] (restated below), replacing Y with X ′ in our formulations enables learning of task-relevant information without access to labels:

Definition 3. (Optimal unimodal augmentation) [72] X ′ 1 is an optimal unimodal augmentation for X 1 when I ( X ; X ′ ) = I ( X ; Y ) , which implies that the only information shared between X and X ′ is task-relevant with no irrelevant noise.

This assumption is satisfied when all information shared between X and X ′ is task-relevant, which implies that the augmentation keeps task-relevant information constant while changing task-irrelevant information. In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background. By performing two separate unimodal augmentations giving X ′ 1 and X ′ 2 , we can substitute contrastive estimators in Eqs.(13) and (14), by replacing I ( X i ; Y ) terms with I ( X i ; X ′ i ) and replacing I ( X 1 ; X 2 ∣ Y ) terms with I ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) :

The objectives can be seen as conditional contrastive learning on augmentations ( X ′ 1 , X ′ 2 ). Here again f ( x 1 , x 2 , x ′ 1 , x ′ 2 ) is different from the critics in Eqs.(13 because of the different general forms. We implement f () here as g ([ x 1 , x ′ 1 ]) T h ([ x 2 , x ′ 2 ]) where g () , h () are trainable encoders specific for each modality and [ x 1 , x ′ 1 ] denotes concatenation. This concatenation is justified by the CMI estimators in Sordoni et al. [69], who show that concatenating the conditioning variable with the input in the critic f ( x 1 , x 2 , x ′ 1 , x ′ 2 ) yields a Conditional InfoNCE estimator (Eq.(15)) that is a lower bound for CMI. However, the exact Conditional InfoNCE estimator learns a different conditional distribution p ( x 1 , x 2 ∣ x ′ 1 , x ′ 2 ) for each augmented pair x ′ 1 , x ′ 2 , which can be prohibitively expensive. We could approximate this by creating multiple augmentations of a single paired x 1 , x 2 . Our code uses one augmented pair x ′ 1 , x ′ 2 for each x 1 , x 2 but could be extended to multiple pairs, and we find this simple approach yields consistent CMI lower and upper bounds that are empirically comparable to existing CMI estimators [55, 69]. We include full comparisons and implementation details in Appendix D.1, and in Appendix C.2 we discuss an alternative interpretation based on viewing CL as kernel learning which permits using conditional kernel estimation for our objectives.

Although we find this method to work well in practice, a more careful analysis reveals that 2 separate unimodal augmentations X ′ 1 and X ′ 2 each satisfying I ( X i ; X ′ i ) = I ( X i ; Y ) do not together satisfy I ( X 1 ; X 2 ∣ Y ) = I ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) needed for the substitution in Eqs.(15) and (16) to hold with equality. To satisfy this property exactly, we define optimal multimodal augmentations:

Definition 4. (Optimal multimodal augmentation) X ′ 1 and X ′ 2 are optimal multimodal augmentation for X 1 and X 2 when I ( X 1 , X 2 ; X ′ 1 , X ′ 2 ) = I ( X 1 , X 2 ; Y ) , which implies that the only information shared between X 1 , X 2 and X ′ 1 , X ′ 2 is task-relevant with no irrelevant noise.

We satisfy I ( X 1 , X 2 ; X ′ 1 , X ′ 2 ) = I ( X 1 , X 2 ; Y ) using two steps:

We call the second step unique augmentation : after observing X 1 , we create augmented X ′ 2 from X 2 to keep task-relevant information not already in X 1 . To empirically satisfy optimal multimodal augmentations, we avoid augmentations in one modality that will remove or strongly destroy information shared with the other modality. For example, in image captioning, we should avoid image augmentations such as cropping that destroy information from the caption (e.g., cropping object parts referred to by the caption), and instead, only augment images via flipping or color jittering which retains all caption information. Figure 4 shows an example of unique augmentation that satisfies these conditions. In our experiments, we will show that our augmentations consistently perform better than standard augmentations (Table 3), suggesting that approximately satisfying Eqs.(17) and (18) can be empirically sufficient, which is simple and straightforward to implement on real-world datasets.

3.3 Overall method and implementation

Algorithm 1 Standard multimodal CL.

Task-relevant

Input

information

X1

Thecaris as

Unique in X1:

fast as a cheetah.

"cheetah"

(xL))

Unique in X2:

X2

highway pixels

(Image)

Shared between

1

X, and X2:

Y

(image and caption

are matched)

fastcar

Unique task information

Standard

Aug for X2

in X2 destroyed:

highway pixels

Crop + Flip

Unique

Unique task information

in X2 preserved:

Aug for X2

highway pixels

Figure 4: Standard vs. unique augmentations for the figurative language [88] dataset. After augmenting text modality X 1 independently (same for both augmentation types), we illustrate their differences for image augmentation: unique augmentation on images should avoid removing information referred to by X 1 (the text). The text mentions that the car is fast so unique augmentation for images should not remove the highway pixels of the image which can suggest the car is fast.

4 Experiments

We run comprehensive experiments on a suite of synthetic and large-scale real-world datasets with varying requirements of shared and unique task-relevant information, comparing our FACTORCL method to key baselines:

SimCLR [13]: the straightforward method of cross-modal ( X 1 , X 2 ) contrastive learning.

Cross+Self [33, 36, 44, 64, 85, 89]: captures a range of methods combining cross-modal ( X 1 , X 2 ) CL with additional unimodal ( X i , X ′ i ) CL objectives. This category also includes other ways of preserving unique information, such as through (variational) autoencoder reconstructions [81].

Cross+Self+Fact [86, 89]: A factorized extension of Cross+Self, which is approximately done in prior work that adds separate (typically pre-trained) unimodal encoders for each modality.

SupCon [41], which learns I ( X 1 ; X 2 ∣ Y ) using CL conditioned on Y from labeled data.

We also carefully ablate each component of our method and investigate factors, including training data size and choice of augmentations. The intermediate ablations that emerge include:

FACTORCL-SUP: The supervised CL version which uses labels Y in Eqs.(13) and (14).

Algorithm 2 FACTORCL

.

The final algorithm sketch is in Algorithm 2, which we compare against standard CL in Algorithm 1. It can be shown that FACTORCL learns all the taskrelevant information from both modalities:

Theorem 3. (Optimality of FACTORCL ) If Z S 1 , Z S 2 , Z U 1 , Z U 2 perfectly maximize Eqs.(6-7) and the estimations in Eqs.(8) and (9) are tight, we obtain I ( X 1 , X 2 ; Y ) = I ( Z S 1 ; Z S 2 ; Y ) + I ( Z U 1 ; Y ∣ Z S 2 ) + I ( Z U 2 ; Y ∣ Z S 1 ) , suggesting that FACTORCL learns both shared and unique task-relevant information.

We include the full proof in Appendix C.3. In practice, while we do not expect perfect estimation of MI quantities and maximization with respect to MI objectives, we include implementation details regarding architectures and contrastive objectives that improve empirical performance in Appendix D.1.

Complexity : Compared to heuristic combinations of cross-modal and single-modality CL [33, 36, 44, 64, 81, 85, 89], our approach does not significantly increase complexity: (1) upper bounds on MI can be estimated 'for free' by directly plugging in the optimal critic from I NCE, (2) removal of task-irrelevant information via I ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) shares encoders with I NCE, and (3) separate unimodal augmentations perform empirically well. We describe some extensions of other self-supervised methods in Appendix C.4.

FACTORCL-SSL: The fully self-supervised version of our approach replacing Y with multimodal augmentations X ′ 1 and X ′ 2 to approximate the task.

OurCL-SUP: FACTORCL-SUP but removing the factorization so only two features Z 1 is optimized for both I ( X 1 ; X 2 ; Y ) and I ( X 1 ; Y ∣ X 2 ) , Z 2 optimized for both I ( X 1 ; X 2 ; Y ) and I ( X 2 ; Y ∣ X 1 ) .

OurCL-SSL: FACTORCL-SSL but also removing the factorization in the self-supervised setting.

The formulation of each ablation and implementation can be found in Appendix D.1.

4.1 Controlled experiments on synthetic datasets

Synthetic data generation : We begin by generating data with controllable ratios of task-relevant shared and unique information. Starting with a set of latent vectors w 1 , w 2 , w s ∼ N( 0 d , Σ 2 d ) , d = 50 representing information unique to X 1 , X 2 and common to both respectively, the concatenated vector [ w 1 , w s ] is transformed into high-dimensional x 1 using a fixed transformation T 1 and likewise [ w 2 , w s ] to x 2 via T 2 . The label y is generated as a function (with nonlinearity and noise) of varying ratios of w s , w 1 , and w 2 to represent shared and unique task-relevant information.

Results : In Figure 1, we show our main result on synthetic data comparing FACTORCL with existing CL baselines. FACTORCL consistently maintains the best performance, whereas SimCLR [13] and SupCon [41] see performance drops as unique information increases. Cross+Self [33, 36, 44, 89] recovers in fully unique settings (x-axis = 1 . 0 ) but suffers at other ratios.

Representation probing information : We run a probing experiment to compute how well different contrastive representations capture shared and unique information. In Table 1, for the Z i 's learned by each method, we approximately compute I ( Z i ; w 1 ) , I ( Z i ; w 2 ) , and I ( Z i ; w s ) with respect to ground truth generative variables w s , w 1 , and w 2 . As expected, existing methods such as SimCLR capture smaller amounts of unique information (roughly 4 bits in I ( Z i ; w 1 ) and I ( Z i ; w 2 ) ), focusing instead on learning I ( Z i ; w s ) (12 bits). Cross+self captures slightly larger I ( Z i ; w 2 ) = 4 . 26 , and SupCon with labeled data captures up to 5 bits of unique information. Our FACTORCL approach captures 7 bits of unique information and maintains 10 bits of shared information, with total information captured higher than the other approaches. Furthermore, { Z S 1 , Z S 2 } capture more information about w s , Z U 1 about w 1 , and Z U 2 about w 2 , indicating that factorization in our approach is successful.

4.2 Self-supervised multimodal learning with low redundancy and high uniqueness

Multimodal fusion datasets : We use a large collection of real-world datasets provided in MultiBench [45], where we expect varying ratios of shared and unique information important for the task, to compare FACTORCL with other CL baselines:

MIMIC [38]: mortality and disease prediction from 36 , 212 medical records (tabular patient data and medical time-series sensors from ICU).

MOSEI [93]: multimodal sentiment and emotion benchmark with 23 , 000 monologue videos.

MOSI [91]: multimodal sentiment analysis from 2 , 199 YouTube videos.

UR-FUNNY [27]: a dataset of humor detection from more than 16 , 000 TED talk videos.

MUSTARD [12]: a corpus of 690 videos for research in sarcasm detection from TV shows.

IRFL [88]: 6 , 697 matching images and figurative captions (rather than literal captions).

Together, these datasets cover seven different modalities from the healthcare, affective computing, and multimedia research areas and total more than 84 , 000 data points. For MIMIC with tabular and medical sensor inputs, we train self-supervised CL models on top of raw modality inputs. For IRFL with image and caption inputs, we start with a pretrained CLIP model [61] and perform continued pre-training to update CLIP weights with our FACTORCL objectives, before linear classifier testing. For the remaining four video datasets, we train self-supervised CL models starting from standard pre-extracted text, video, and audio features [45]. Please refer to Appendix D.2 for experimental details. We release our code and models at https://github.com/pliang279/FactorCL .

Multimodal fusion results : From Table 2, FACTORCL significantly outperforms the baselines that do not capture both shared and unique information in both supervised and self-supervised settings, particularly on MUSTARD (where unique information expresses sarcasm, such as sardonic facial expressions or ironic tone of voice), and on MIMIC (with unique health indicators and sensor readings). In Table 3, we also show that FACTORCL substantially improves the state-of-the-art in classifying images and figurative captions which are not literally descriptive of the image on IRFL, outperforming zero-shot and fine-tuned CLIP [61] as well as continued pre-training baselines on top of CLIP.

Modeling ablations : In Table 2, we also carefully ablate each component in our method and indicate either existing baselines or newly-run ablation models.

Factorized representations : In comparing FACTORCL-SSL with OurCL-SSL, and also FACTORCL-SUP with OurCL-SUP, we find that factorization is critical: without it, performance drops on average 6 . 1% , with performance drop as high as 8 . 6% for MIMIC.

Information removal via upper bound : By comparing FACTORCL with SimCLR, Cross+Self, and Cross+Self+Fact, and SupCon that only seek to capture task-relevant information via contrastive lower bounds on MI, we find that separately modeling the task-relevant information (to be captured) and task-irrelevant information (to be removed) is helpful. Without removing task-irrelevant information via the upper-bound objective, performance drops on average 13 . 6% , with performance drops as high as 23 . 5% for the MOSI dataset. We also found that training was more difficult without this objective, which is expected due to overwhelming superfluous information from the dataset [93].

Multimodal augmentations : Finally, we investigate the differences between separate unimodal augmentations (FACTORCL-IndAug in Table 3) versus a joint multimodal augmentation (FACTORCL-SSL) on the IRFL dataset. We choose this dataset since its images and captions are the easiest to visualize (see Figure 4 for augmentations from both strategies). In the self-supervised setting, we find that multimodal augmentations achieve 95% performance, higher than the 92% for separate unimodal augmentations, and both outperform baselines SimCLR and Cross+Self.

Ablations on S, U 1 and U 2 : In Table 4, we also test FACTORCL when training linear classifiers on top of only shared { Z S 1 , Z S 2 } and unique Z U 1 , Z U 2 separately. We call these models FACTORCLS , FACTORCLU 1 , and FACTORCLU 2 . Immediately, we observe that performance drops as compared to the full FACTORCL model, indicating that both shared and unique information are critical in real-world multimodal tasks. As expected, the best-performing submodel is the one that captures the region with the largest amount of task-relevant information: MOSEI and MOSI are known to include a lot of redundancy and unique information since language is very important for detecting sentiment [93], so FACTORCLS and FACTORCLU 2 perform best. For sarcasm detection on MUSTARD, video information is most important with FACTORCL-

Table 3: Continued pre-training on CLIP with our FACTORCL objectives on classifying images and figurative language.

U 1 performing best ( 59 . 4% ), and ablation models are also the furthest away from full multimodal performance ( 69 . 9% ). This is aligned with intuition where sarcasm is expressed through tone of voice and visual gestures (high U 1 ), as well as from contradictions between language and video (higher multimodal performance).

Additional results : In Appendix D.3, we also verify FACTORCL in settings with abundant shared information, where we expect to recover the same performance as standard CL [13, 58, 72].

5 Related Work

Contrastive learning is a successful self-supervised learning paradigm for computer vision [11, 13, 14, 25, 28, 58], natural language [24, 54, 56], speech [5, 58, 63], and multimodal tasks [1, 37, 61]. Its foundational underpinnings are inspired by work in multiview information theory [23, 41, 70, 72, 76] studying the shared information between two views and whether they are necessary or sufficient in predicting the label. Recently, Wang et al. [81] and Kahana and Hoshen [39] discuss the limitations of assuming multiview redundancy and propose autoencoder reconstruction or unimodal contrastive learning to retain unique information, which resembles the Cross+self baselines in our experiments. We refer the reader to Shwartz-Ziv and LeCun [67] for a comprehensive review on multiview and contrastive learning. Our work also relates to conditional contrastive learning [17, 51, 78, 87], where positive or negative pairs are supposed to sample from conditional distributions.

Multimodal contrastive learning aims to align related data from different modalities, typically provided as positive pairs. This could be done via optimizing a contrastive objective for intermodality pairs [1, 2, 37, 61], or both intra- and inter-modality data pairs [33, 36, 42, 44, 89]. Our work also relates to factorized representation learning, which primarily studies how to capture modality-specific information primarily in each modality and multimodal information redundant in both modalities [32, 75]. Prior work has used disentangled latent variable models [8, 30, 32, 75], mixture-of-experts [66], or product-of-experts [83] layer to explain factors in multimodal data.

Information theory [18, 65] has been used to study several phenomena in multimodal learning, including co-learning [62, 92] and multi-view learning [34, 76]. Due to its theoretical importance, several lower and upper bounds have been proposed for practical estimation [58-60, 84]. We build on the CLUB upper bound [16] to create a more accurate and stable bound. Our characterizations of shared and unique information are also related to partial information decomposition [82], coinformation [7, 80], interaction information [53], and cross-domain disentanglement [35] research.

6 Conclusion

This paper studied how standard CL methods suffer when task-relevant information lies in regions unique to each modality, which is extremely common in real-world applications such as sensor placement, medical testing, and multimodal interaction. In response, we proposed FACTORCL, a new method expanding CL techniques through the use of factorized representations, removing task-irrelevant information via upper bounds on MI, and multimodal data augmentations suitable for approximating the unobserved task. Based on FACTORCL's strong performance, there are several exciting directions in extending these ideas for masked and non-contrastive pre-training; we further discuss broader impacts and limitations of this line of work in Appendix A.

References

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems , 34:24206-24221, 2021.

Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovi´ c, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems , 33:25-37, 2020.

Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision , pages 609-617, 2017.

Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems , 32, 2019.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems , 33: 12449-12460, 2020.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations , 2021.

Anthony J Bell. The co-information lattice. In Proceedings of the fifth international workshop on independent component analysis and blind signal separation: ICA , volume 2003, 2003.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI , 35(8), August 2013.

Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems , 29, 2016.

Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language berts. Transactions of the Association for Computational Linguistics , 9:978-994, 2021.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously_ perfect paper). arXiv preprint arXiv:1906.01815 , 2019.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning , pages 1779-1788. PMLR, 2020.

Jianfeng Chi, William Shand, Yaodong Yu, Kai-Wei Chang, Han Zhao, and Yuan Tian. Conditional supervised contrastive learning for fair text classification. arXiv preprint arXiv:2205.11485 , 2022.

Thomas M Cover and Joy A Thomas. Information theory and statistics. Elements of information theory , 1 (1):279-335, 1991.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems , 35: 35603-35620, 2022.

Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transactions on Information theory , 40(1):259-266, 1994.

Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017 , 2020.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 , 2021.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Jing Huang, Larry Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. Advances in Neural Information Processing Systems , 35:28319-28334, 2022.

Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed Ehsan Hoque. Ur-funny: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2046-2056, 2019.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations , 2018.

Wei-Ning Hsu and James Glass. Disentangling by partitioning: A representation learning framework for multimodal sensory data. arXiv preprint arXiv:1805.11264 , 2018.

Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and Alexander Hauptmann. Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision-language models. arXiv preprint arXiv:2103.08849 , 2021.

Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems , 34:10944-10956, 2021.

HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, and Kee-Eung Kim. Variational interaction information maximization for cross-domain disentanglement. Advances in Neural Information Processing Systems , 33:22479-22491, 2020.

Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. Mural: multimodal, multitask retrieval across languages. arXiv preprint arXiv:2109.05125 , 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Jonathan Kahana and Yedid Hoshen. A contrastive objective for learning disentangled representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 579-595. Springer, 2022.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

Byoungjip Kim, Sungik Choi, Dasol Hwang, Moontae Lee, and Honglak Lee. Transferring pre-trained multimodal representations with cross-modal similarity matching. Advances in Neural Information Processing Systems , 35:30826-30839, 2022.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www.cs.toronto.edu/~kriz/cifar.html .

Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parameter efficient multimodal transformers for video representation learning. arXiv preprint arXiv:2012.04124 , 2020.

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multibench: Multiscale benchmarks for multimodal representation learning. NeurIPS Datasets and Benchmarks Track , 2021.

Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning , pages 6565-6576. PMLR, 2021.

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Shengtong Mo, Dani Yogatama, et al. Highmmt: Towards modality and task generalization for high-modality representation learning. arXiv preprint arXiv:2203.01311 , 2022.

Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430 , 2022.

Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. Quantifying & modeling feature interactions: An information decomposition framework. arXiv preprint arXiv:2302.12247 , 2023.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , pages 13-23, 2019.

Martin Q Ma, Yao-Hung Hubert Tsai, Paul Pu Liang, Han Zhao, Kun Zhang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Conditional contrastive learning for improving fairness in self-supervised learning. arXiv preprint arXiv:2106.02866 , 2021.

Emily E Marsh and Marilyn Domas White. A taxonomy of relationships between images and text. Journal of documentation , 2003.

William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory , 4(4):93-111, 1954.

Yu Meng, Chenyan Xiong, Payal Bajaj, Paul Bennett, Jiawei Han, Xia Song, et al. Coco-lm: Correcting and contrasting text sequences for language model pretraining. Advances in Neural Information Processing Systems , 34:23102-23114, 2021.

Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Ccmi: Classifier based conditional mutual information estimation. In Uncertainty in artificial intelligence , pages 1083-1093. PMLR, 2020.

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005 , 2022.

XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory , 56(11): 5847-5861, 2010.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems , 32, 2019.

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning , pages 5171-5180. PMLR, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Anil Rahate, Rahee Walambe, Sheela Ramanna, and Ketan Kotecha. Multimodal co-learning: challenges, applications with datasets, recent advances and future directions. Information Fusion , 81:203-239, 2022.

Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pretraining for speech recognition. arXiv preprint arXiv:1904.05862 , 2019.

Bin Shan, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil 2.0: Multi-view contrastive learning for image-text pre-training, 2022.

Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal , 27 (3):379-423, 1948.

Yuge Shi, Brooks Paige, Philip Torr, et al. Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in Neural Information Processing Systems , 32, 2019.

Ravid Shwartz-Ziv and Yann LeCun. To compress or not to compress-self-supervised learning and information theory: A review. arXiv preprint arXiv:2304.09355 , 2023.

Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. CoRR , abs/1910.06222, 2019. URL http://arxiv.org/abs/1910.06222 .

Alessandro Sordoni, Nouha Dziri, Hannes Schulz, Geoff Gordon, Philip Bachman, and Remi Tachet Des Combes. Decomposed mutual information estimation for contrastive representation learning. In International Conference on Machine Learning , pages 9859-9869. PMLR, 2021.

Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. In Conference on Learning Theory , 2008.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. ECCV , 2020.

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems , 33:6827-6839, 2020.

Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory , pages 1179-1206. PMLR, 2021.

Yao-Hung Hubert Tsai, Tianqin Li, Weixin Liu, Peiyuan Liao, Ruslan Salakhutdinov, and Louis-Philippe Morency. Learning weakly-supervised contrastive representations. In International Conference on Learning Representations .

Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. Learning factorized multimodal representations. ICLR , 2019.

Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self-supervised learning from a multi-view perspective. In International Conference on Learning Representations , 2020.

Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Russ R Salakhutdinov. Neural methods for point-wise dependency estimation. Advances in Neural Information Processing Systems , 33:62-72, 2020.

Yao-Hung Hubert Tsai, Tianqin Li, Martin Q Ma, Han Zhao, Kun Zhang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning with kernel. arXiv preprint arXiv:2202.05458 , 2022.

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations , 2019.

Jorge R Vergara and Pablo A Estévez. A review of feature selection methods based on mutual information. Neural computing and applications , 24:175-186, 2014.

Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16041-16050, 2022.

Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 , 2010.

Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. Advances in Neural Information Processing Systems , 31, 2018.

Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual information in contrastive learning for visual representations. arXiv preprint arXiv:2005.13149 , 2020.

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space, 2022.

Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning, 2022.

Zesheng Ye and Lina Yao. Contrastive conditional neural processes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9687-9696, 2022.

Ron Yosef, Yonatan Bitton, and Dafna Shahaf. Irfl: Image recognition of figurative language. arXiv preprint arXiv:2303.15445 , 2023.

Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6995-7004, 2021.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 , 2016.

Amir Zadeh, Paul Pu Liang, and Louis-Philippe Morency. Foundations of multimodal co-learning. Information Fusion , 64:188-193, 2020.

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2236-2246, 2018.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Appendix

A Broader Impact

Multimodal data and self-supervised models are ubiquitous in a range of real-world applications. This paper is our attempt at broadening the applicability of self-supervised contrastive methods to a wider range of multimodal tasks beyond those that exhibit multi-view redundancy. We believe that special care must be taken to ensure that these models are safely deployed for real-world benefit:

Time and space complexity : Modern multimodal models are large and take up a significant amount of carbon footprint during training and testing. As compared to heuristic combinations of cross-modal and single-modality CL [33, 36, 44, 64, 81, 85, 89], we believe that FACTORCL does not significantly increase complexity: (1) upper bounds on MI can be estimated 'for free' by directly plugging in the optimal critic from I NCE, and (2) removal of task-irrelevant information via I ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) shares encoders with I NCE, and (3) separate unimodal augmentations perform well enough in practice. We also release our code and models so that they can be evaluated quickly on new tasks, which can amortize complexity costs.

Privacy and security : There may be privacy risks associated with making predictions from multimodal data of recorded human behaviors and medical data (i.e., the datasets used in our experiments for analysis of sentiment, emotions, personality, sarcasm, and humor, as well as disease prediction from medical data). We have followed best practices in maintaining the privacy and safety of these datasets: (1) the creators of these video datasets have taken the appropriate steps to only access public data that participants or content creators have consented for public release (creative commons license and following fair use guidelines of YouTube) [12, 27, 93], (2) MIMIC has been rigorously de-identified in accordance with Health Insurance Portability and Accountability Act (HIPAA) such that all possible personal information has been removed from the dataset [38], (3) all video data was also anonymized and stripped of all personal (e.g., personally identifiable information) and protected attributes (e.g., race, gender), (4) all models trained on affect recognition datasets use only pre-extracted non-invertible features that rely on general visual or audio features such as the presence of a smile or magnitude of voice which cannot be used to identify the speaker [91, 93], and (5) we studied the videos collected in these affective computing datasets and found no offensive words used or personal attacks recorded in the video. Finally, we only use these datasets for research purposes and emphasize that any multimodal models trained to perform prediction should only be used for scientific study and should not in any way be used for real-world harm.

Social biases : We acknowledge risks of social bias due to imbalanced datasets, resulting in potential biases surrounding gender, race, and ethnicity, among others [9, 46]. We note that our FACTORCL approach has a close link with conditional CL [51], which can also be adapted to condition on sensitive attributes and therefore reduce bias. Studying these research questions is an important direction for future work.

Future work : We discuss more limitations and potential future work in this direction. Firstly, optimizing our objectives using better MI lower and upper bounds such as in Guo et al. [26] and Sordoni et al. [69], could improve the performance for inputs of higher dimension and complex modality. Next, the current data augmentation method requires one to pick augmentations to approximately satisfy Definition 4; future work could extend InfoMin [72] to automatically generate data augmentations to satisfy Definition 4, or leverage future progress in multimodal generative models for data augmentation. Lastly, future work could quantify whether shared or unique information is more important for different tasks and reweight the terms in the FACTORCL objective to suit the tasks.

B Analysis of Multi-view Contrastive Learning

Multi-view shared information describes the extent and dimensions in which information can be shared across different views. The presence of shared information is often in contrast to unique information that exists solely in a single modality, and can be formalized via information theory:

Definition 5. (Shared information) Given X 1 and X 2 , I ( X 1 ; X 2 ) = ∫ p ( x 1 , x 2 ) log p ( x 1 ,x 2 ) p ( x 1 ) p ( x 2 ) measures the degree of information-theoretic shared information between X 1 and X 2 .

Definition 6. (Task-relevant shared information) Given X 1 , X 2 , and a target Y , I ( X 1 ; X 2 ; Y ) = I ( X 1 ; X 2 ) -I ( X 1 ; X 2 ∣ Y ) = ∫ p ( x 1 , x 2 ) log p ( x 1 ,x 2 ) p ( x 1 ) p ( x 2 ) - ∫ p ( x 1 , x 2 ∣ y ) log p ( x 1 ,x 2 ∣ y ) p ( x 1 ∣ y ) p ( x 2 ∣ y ) measures

the amount of task-relevant shared information between X 1 and X 2 for predicting Y . I ( X 1 ; X 2 ∣ Y ) represents the task-irrelevant shared information.

Learning shared information via contrastive learning : Current approaches for multi-view contrastive learning model shared information I ( X 1 ; X 2 ) (and subsequently task-relevant shared information I ( X 1 ; X 2 ; Y ) during downstream task fine-tuning), without modeling unique information.

Optimizing for I ( X 1 ; X 2 ) is performed via a surrogate loss during self-supervised pre-training (where we do not have access to the label Y ) by maximizing the InfoNCE objective:

Oord et al. [58] show that I ( X 1 ; X 2 ) ≥ log k - L NCE ( X 1 ; X 2 ) where L NCE ( X 1 ; X 2 ) is negative of INFONCE and is the loss to minimize (rather than maximize) in training. NCE falls into a broader class of contrastive learning methods [13, 15, 28, 41, 61] that model the ratio between joint densities p ( x 1 , x 2 ) and product of marginal densities p ( x 1 ) p ( x 2 ) using positive and negative samples [57, 59, 60, 79, 84] or probabilistic classifiers [55, 77], all of which can also be used to capture shared information.

Tian et al. [71] argues that the optimal view of contrastive learning is also minimal: the minimal representations only extract relevant information of the contrastive task (maximizing the shared part) and throw away other information. Therefore, from this minimal assumption, we have I ( Z 1 ; Y ∣ X 2 ) = 0 and I ( Z 2 ; Y ∣ X 1 ) = 0 as minimal Z 1 and Z 2 only captures task-relevant information from the shared part. By conditioning on X 1 or X 2 , the shared part is removed, and Z 1 and Y (or Z 2 and Y ) do not share information.

Lastly, we restate the multi-view non-redundancy from Definition 2:

Definition 7. (Multi-view non-redundancy) ∃ ϵ > 0 such that I ( X 1 ; Y ∣ X 2 ) > ϵ or I ( X 2 ; Y ∣ X 1 ) > ϵ .

We would like to compare and clarify the differences between the multiview redundancy assumption in Eq.(1) and the multi-view nonredundancy in Def. 7. The multiview redundancy assumption in Eq.(1) states that the task-relevant information from the unique part is minimal ( ≤ ϵ ). The multiview non-redundancy states the opposite: the task-relevant information from the unique part is nonzero and nonminimal, as it is not bounded by ϵ . Next we briefly clarify the relationship between these two assumptions and the InfoMin assumption: I ( Z 1 ; Y ∣ X 2 ) = I ( Z 2 ; Y ∣ X 1 ) = 0 . InfoMin is about representation Z while the redundancy assumptions are only about data X . InfoMin states that the optimal (sufficient and minimal) representation learns task-relevant information only from the shared part, as we discussed in the paragraph above. We empirically checked the two assumptions: Tables 1 and 4 in the main text show that the multiview non-redundancy assumption holds empirically, and Table 11 shows that the InfoMin assumption holds empirically.

We now show the limitations of CL methods, first restating the Theorem here:

Theorem 4. (Suboptimality of standard CL) When there is multi-view non-redundancy as in Definition 7, given optimal representations { Z 1 , Z 2 } that satisfy Eq.(19 and I ( Z 1 ; Y ∣ X 2 ) = I ( Z 2 ; Y ∣ X 1 ) = 0 [72], we have that

Proof. Since Z 1 and Z 2 maximize I ( X 1 ; X 2 ) we have that I ( Z 1 ; X 2 ) = I ( X 1 ; Z 2 ) = I ( X 1 ; X 2 ) so I ( Z 1 ; X 2 ; Y ) = I ( X 1 ; Z 2 ; Y ) = I ( X 1 ; X 2 ; Y ) and I ( Z 1 ; X 2 ∣ Y ) = I ( X 1 ; Z 2 ∣ Y ) = I ( X 1 ; X 2 ∣ Y ) .

We now show the relationship between I ( X 1 , X 2 ; Y ) , which is the total information X 1 , X 2 contributes towards predicting Y in classic supervised learning, with I ( Z 1 , Z 2 ; Y ) , which is the information that our learned self-supervised representations can contribute towards Y during supervised fine-tuning. We first derive the relationship between I ( Z 1 ; Y ) and I ( X 1 ; Y ) :

Given X 1 , we further derive a relationship between I ( Z 2 ; Y ∣ Z 1 ) and I ( X 2 ; Y ∣ X 1 ) :

In Eqs.(28) and (30), we use the fact that conditioning on Z 1 and X 1 jointly reduces to conditioning on X 1 since Z 1 is deterministically obtained from X 1 , and in Eq.(32) we use the definition of learning Z s to maximize I ( X 1 ; X 2 ) so I ( Z 1 ; X 2 ; Y ) = I ( X 1 ; Z 2 ; Y ) . Finally, adding both terms up,

gives the desired result.

Bayes error rate. The Bayes error rate P e ( Z 1 , Z 2 ) ∶= 1 -E P Z 1 ,Z 2 [ max y ∈ Y P ( ˆ Y = y ∣ z 1 , z 2 )] of contrastive representations { Z 1 , Z 2 } is given by:

Proof. We use the inequality between P e and H ( Y ∣ Z ) [22, 76, 81]:

If we regard Z as the joint of Z 1 and Z 2 , then we have

We further expand H ( Y ∣ Z 1 , Z 2 ) by definition of mutual information, I ( X ; Y ) = H ( X ) -H ( X ∣ Y ) , Theorem 4, and the I ( X 1 ; X 2 ; Y ) = I ( X 1 ; X 2 ) -I ( X 1 ; X 2 ∣ Y ) :

Plugging in Eq.(42), we have

resulting in the Bayes error rate as desired.

and

C FACTORIZED CONTRASTIVE LEARNING

C.1 Contrastive estimators

Theorem 5. (Contrastive estimators for I ( X 1 ; X 2 ) ) Defining the NCE estimator and NCE-CLUB estimator as follows,

where f ∗ ( x 1 , x 2 ) is the optimal critic from I NCE plugged into the I CLUB objective [16]. We call the proposed plug-in objective Eq.(11) I NCE-CLUB , and obtain lower and upper bounds on I ( X 1 ; X 2 ) :

Proof. The lower bound I NCE ( X 1 ; X 2 ) ≤ I ( X 1 ; X 2 ) follows from Oord et al. [58]: optimizing the objective leads to an optimal critic f ∗ = log p ( x 2 ∣ x 1 ) + c ( x 2 ) [60] or without loss of generality f ∗ = log p ( x 1 ∣ x 2 ) + c ( x 1 ) , where c (⋅) is an arbitrary deterministic function. Plugging the optimal critic into the I NCE ( X 1 ; X 2 ) gives the result: I NCE ( X 1 ; X 2 ) + log N ≤ I ( X 1 ; X 2 ) [58, 60].

Next, the original I CLUB ( X 1 ; X 2 ) [16] is defined as:

As mutual information is symmetric w.r.t x 1 and x 2 : I ( X 1 ; X 2 ) = I ( X 2 ; X 1 ) , without loss of generality, we have:

The formulation above assumes p ( x 1 ∣ x 2 ) is known, which is satisfied when we use the optimal critic f ∗ = log p ( x 1 ∣ x 2 ) + c ( x 1 ) from I NCE ( X 1 ; X 2 ) . Plugging the optimal critic f ∗ into I CLUB ( X 1 ; X 2 ) , we obtain a desired upper bound I NCE-CLUB ( X 1 ; X 2 ) of I ( X 1 ; X 2 ) :

Eq.(59) is from the linearity of expectation, Eq.(60) is from the fact that c ( x 1 ) is not a function of x 2 and therefore E p ( x 1 ,x 2 ) [ c ( x 1 )] = E p ( x 1 ) p ( x 2 ) [ c ( x 1 )] = E p ( x 1 ) [ c ( x 1 )] , and Eq.(61) is directly from the original I CLUB ( X 1 ; X 2 ) paper [16].

C.2 Unimodal and multimodal augmentations

We first restate the definitions of optimal single-view and multi-view augmentation:

Definition 8. (Optimal unimodal augmentation) X ′ 1 is an optimal unimodal augmentation for X 1 when I ( X ; X ′ ) = I ( X ; Y ) , which implies that the only information shared between X and X ′ is task-relevant with no irrelevant noise.

Definition 9. (Optimal multimodal augmentation) X ′ 1 and X ′ 2 are optimal multimodal augmentation for X 1 and X 2 when I ( X 1 , X 2 ; X ′ 1 , X ′ 2 ) = I ( X 1 , X 2 ; Y ) , which implies that the only information shared between X 1 , X 2 and X ′ 1 , X ′ 2 is task-relevant with no irrelevant noise.

When are these assumptions satisfied? I ( X ; X ′ ) = I ( X ; Y ) holds when all information shared between X and X ′ is task-relevant, which implies that the augmentation keeps task-relevant information constant while changing task-irrelevant information. In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background. To satisfy I ( X 1 , X 2 ; X ′ 1 , X ′ 2 ) = I ( X 1 , X 2 ; Y ) , by the chain rule of MI, we augment in two steps:

the second step is the unique augmentation : after observing X 1 , we create augmented X ′ 2 from X 2 to keep the task-relevant information but meanwhile do not affect any information from X 1 . In Table 5, we include some more examples of how unique augmentations could be designed across different datasets.

Final objectives : If Definitions 8 and 9 are both satisfied, we can substitute contrastive estimators in the following equations:

by replacing I ( X i ; Y ) terms with I ( X i ; X ′ i ) and replacing I ( X 1 ; X 2 ∣ Y ) terms with I ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) :

C.2.1 Implementing conditional CL via kernel

We restate our objectives below:

However, sampling from p (⋅∣ x ′ 1 , x ′ 2 ) is hard. Since X ′ 1 , X ′ 2 are continuous variables, directly sampling from the conditional distributions p (⋅∣ x ′ 1 , x ′ 2 ) may be difficult; training a generative model p θ ( x 1 , x 2 ∣ x ′ 1 , x ′ 2 ) from augmented data x ′ 1 , x ′ 2 to original data x 1 , x 2 can be expensive and nontrivial in a multimodal setup. In this work, we implement the conditioning in p ( x 1 , x 2 ∣ x ′ 1 , x ′ 2 ) through concatenation and the details are in Appendix D.1. Here we discuss an alternative solution to this problem introduced by Tsai et al. [78]. It leverages kernel methods for conditional sampling in contrastive learning by assigning weights to each sampled data given the kernel similarity between conditioned variables, avoiding directly sampling from the conditional distributions or training generative models. In our formulation, given multimodal input ( x 1 , x 2 ) with their augmentation ( x ′ 1 , x ′ 2 ) , we can simply use the technique from [78] to estimate I NCE ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) and I NCE-CLUB ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) , where the kernel measures the similarity between different pairs ( x ′ 1 , x ′ 2 ) of the conditional variable

X 1 , X 2 . Specifically,

where K X 1 ⊥ ⊥ X 2 ∣ X ′ 1 ,X ′ 2 = K X 1 X 2 ( K X ′ 1 X ′ 2 + λ I ) -1 K X ′ 1 X ′ 2 and [ K X 1 ⊥ ⊥ X 2 ∣ X ′ 1 ,X ′ 2 ] ii is the i th row and i th column of K X 1 ⊥ ⊥ X 2 ∣ X ′ 1 ,X ′ 2 . K X 1 X 2 is a kernel similarity matrix between X 1 and X 2 , and K X ′ 1 X ′ 2 is a separate kernel similarity matrix between X ′ 1 and X ′ 2 . f ∗ is the optimal solution of Eq.(70). By leveraging the similarity K X ′ 1 X ′ 2 between conditional variables X ′ 1 and X ′ 2 , K X 1 ⊥ ⊥ X 2 ∣ X ′ 1 ,X ′ 2 transforms the similarity scores between X 1 and X 2 under unconditional distributions into similarity scores under conditional distributions. Note that the expectations in Eqs.(70) and (71) are taken over the joint distribution p ( x 1 , x 2 , x ′ 1 , x ′ 2 ) , which comes naturally after augmenting both modalities X 1 and X 2 . This method could effectively alleviate the problem of sampling from conditional distributions in our formulation. We refer the reader to Tsai et al. [78] for more details.

C.3 Final estimators in FACTORCL

Theorem 6. (Contrastive estimators for shared and unique information). Under assumptions on single-view augmentations I ( X 1 ; Y ) = I ( X 1 , X ′ 1 ) (Definition 8) and optimal multi-view augmentation X ′ 2 such that I ( X 1 , X 2 ; X ′ 1 , X ′ 2 ) = I ( X 1 , X 2 ; Y ) (Definition 9), we can define contrastive objectives for task-relevant shared and unique information with:

Proof. The objectives follow from the fact that I NCE ( X 1 ; X 2 ) and I NCE ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) are lower bounds of I ( X 1 ; X 2 ) and I ( X 1 ; X 2 ∣ Y ) respectively, and I NCE-CLUB ( X 1 ; X 2 ) and I NCE-CLUB ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) are upper bounds of I ( X 1 ; X 2 ) and I ( X 1 ; X 2 ∣ Y ) respectively:

and symmetrically for U 2 .

Now we show that FACTORCL learns both shared and unique task-relevant information. First, we restate the definition of the factorized representations:

where I ( Z 1 ; X 2 ; Y ) = I ( Z 1 ; X 2 ) -I ( Z 1 ; X 2 ∣ Y ) is the shared information and I ( Z 2 ; X 1 ; Y ) = I ( Z 2 ; X 2 ) -I ( Z 2 ; X 1 ∣ Y ) is the unique information.

Theorem 7. (Optimality of FACTORCL ) If Z S 1 , Z S 2 , Z U 1 , Z U 2 perfectly maximize Eqs.(78-79) and the estimations in Eqs.(13-67) are tight, we obtain I ( X 1 , X 2 ; Y ) = I ( Z S 1 ; Z S 2 ; Y ) + I ( Z U 1 ; Y ∣ Z S 2 ) + I ( Z U 2 ; Y ∣ Z S 1 ) , suggesting that FACTORCL learns both shared and unique taskrelevant information.

Proof. Because I ( X 1 , X 2 ; Y ) = I ( X 1 ; X 2 ; Y ) + I ( X 1 ; Y ∣ X 2 ) + I ( X 2 ; Y ∣ X 1 ) , it is sufficient to show that I ( X 1 ; X 2 ; Y ) = I ( Z S 1 ; Z S 2 ; Y ) , I ( X 1 ; Y ∣ X 2 ) = I ( Z U 1 ; Y ∣ Z S 2 ) and I ( X 2 ; Y ∣ X 1 ) = I ( Z U 2 ; Y ∣ Z S 1 ) .

First we show I ( X 1 ; X 2 ; Y ) = I ( Z S 1 ; Z S 2 ; Y ) . Crucially, by definition of how Z S 1 and Z S 2 are optimized to maximize I ( X 1 ; X 2 ; Y ) , we have that:

We can then obtain

where Eq.(84) is because Z S 1 are deterministically obtained from S 1 and Eq.(85) is because Z S 1 maximizes the shared information. Finally, we go to Eq.(87) I ( Z S 2 ; Y ∣ Z S 1 ) = 0 as shown in Eqs.(26-33) using the fact that Z S 1 is learned to maximize I ( X 1 ; X 2 ; Y ) and I ( Z S 1 ; X 2 ; Y ) = I ( X 1 ; Z S 2 ; Y ) .

Next, we show I ( X 1 ; Y ∣ X 2 ) = I ( Z U ; Y ∣ Z S )

which is by the chain rule of conditional mutual information. Then we show I ( Z U 1 ; Y ; Z U 2 ∣ Z S 2 ) = 0 :

This is because Eq.(79) leads to I ( Z U 1 ; Y ∣ X 2 ) = I ( X 1 ; Y ∣ X 2 ) and I ( Z U 2 ; Y ∣ X 1 ) = I ( X 2 ; Y ∣ X 1 ) . If the estimations in Eqs.(13-67) are tight, by conditioning and by the previously stated I ( Z U 1 ; Y ∣ X 2 ) = I ( X 1 ; Y ∣ X 2 ) , Z U 1 tightly captures information from only X 1 and not in X 2 . The same applies to Z U 2 . We have I ( Z U 1 ; X 2 ) = I ( Z U 2 ; X 1 ) = I ( Z U 1 ; Z U 2 ) = I ( Z U 1 ; Z U 2 ∣ T ) = 0 with T being an arbitrary random variable because no shared information exists between Z U 1 and Z U 2 . Then we obtain:

We use the fact that conditioning on Z S 2 , Z U 2 and X 2 jointly reduces to conditioning on X 2 since Z S 2 and Z U 2 are deterministically obtained from X 2 . Lastly, since Eqs.(78-79) are satisfied, Z U 1 = arg max Z 1 = f θ ( X 1 ) I ( Z 1 ; Y ∣ X 2 ) therefore I ( Z U 1 ; Y ∣ X 2 ) = I ( X 1 ; Y ∣ X 2 ) . We have:

The proof for I ( X 2 ; Y ∣ X 1 ) = I ( Z U 2 ; Y ∣ Z S 1 ) is similar. We now have shown that I ( X 1 ; X 2 ; Y ) = I ( Z S 1 ; Z S 2 ; Y ) , I ( X 1 ; Y ∣ X 2 ) = I ( Z U 1 ; Y ∣ Z S 2 ) and I ( X 2 ; Y ∣ X 1 ) = I ( Z U 2 ; Y ∣ Z S 1 ) , adding up all LHS and RHS we have the theorem.

C.4 Extensions to masking and non-contrastive learning

We now show how similar ideas can be extended to other popular self-supervised learning objectives, such as non-contrastive learning [6, 94] and masked pre-training [20, 29]. Importantly, this paper provides a new principle for multimodel self-supervised learning: (1) learning task-relevant information and (2) removing task-irrelevant information from both shared and unique parts across modalities. Our paper focuses on realizing this principle via multi-view information theory and contrastive learning. Below we provide two potential alternatives to realize this principle on non-contrastive and masking methods, respectively:

Non-contrastive learning: Methods such as Barlow Twins [94] and VICReg [6] use invariance and covariance regularizations to maximally preserve shared information in the embeddings across two modalities. However, the embeddings learned may contain only contain task-relevant information from the shared part and not unique parts. To use the principle in this paper to capture more taskrelevant information from unique parts, one should perform VIC-regularization on X 1 features, on

X 2 features, and on X 1 , X 2 cross-modal features. When performing VICReg on unimodal features, one should condition on the other modality when performing augmentation. Specifically, similar to the idea of multimodal augmentation in this paper, the augmentation of the second modality should not interfere with the shared part (i.e., do not augment regions referred to by the first modality), making the invariance and covariance regularization of the second modality focus on the augmented modality-unique features. This makes the model learn unique modality features that are not captured by the joint embedding from standard independent augmentations.

Masking: Conceptually, masking [20, 29] can be interpreted as leveraging unmasked regions in the same modality to predict masked regions or leveraging the other modality to predict the masked region. However, the learned representation may not be all task-relevant. To use the principle in this paper to exclude task-irrelevant information and capture more task-relevant information from unique parts, we can perform conditional masking, where masking is conditioned on augmented views (similar to the multimodal augmentation in the paper, where the conditioned views are approximating the labels). As a result, only unique regions in the second modality can be masked out, making the model capture more unique information from the second modality by masked prediction. Here we have only provided high-level intuitions of extensions to these methods, and future work should explore these ideas in more detail.

D Experimental Details

D.1 Implementation details

Objective Formulation and Architecture

In Algorithm 2 in the main text, we see the sketch for doing contrastive learning with our proposed objectives. To implement all algorithms used in our ablation experiments, we start with two encoders e 1 (⋅) and e 2 (⋅) , which takes samples x 1 and x 2 from the modalities X 1 and X 2 , and outputs corresponding representations z 1 and z 2 . We also have a critic function f θ (⋅ , ⋅) parametrized by θ which takes z 1 and z 2 as inputs and returns a scalar. A popular way to perform contrastive learning aims to maximize I NCE ( X 1 ; X 2 ) , where

In our algorithms, we follow the derivations in Eqs.(8-9) to maximize each I NCE objective and minimize each I NCE-CLUB objective. Therefore, for each objective, we add two additional MLP heads on top of the two encoders and create a separate critic which takes in the outputs of the MLP heads instead of the encoders. In all the experiments, we adopt the concat critic design [58, 60, 68], where f θ ( x, y ) = h θ ([ x, y ]) with h θ being an MLP.

FACTORCL-SUP : In the supervised version of CL which uses label Y , the objective we aim to maximize is formulated as

Each I NCE and I NCE-CLUB term in this objective is calculated using its own critic as discussed above. The conditional terms involving the label Y are implicitly modeled by directly concatenating Y to the outputs of both heads before feeding into the critic. To obtain the learned representations Z S 1 , we concatenate the outputs of the heads on top of the encoder e 1 that correspond to the terms I NCE ( X 1 ; X 2 ) and I NCE-CLUB ( X 1 ; X 2 ∣ Y ) . To obtain Z U 1 , we concatenate e 1 's head outputs from the terms I NCE ( X 1 ; Y ) , I NCE-CLUB ( X 1 ; X 2 ) , and I NCE ( X 1 ; X 2 ∣ Y ) . Z S 2 and Z U 2 are obtained in a similar fashion, except we use the outputs from e 2 's heads instead of e 1 .

FACTORCL-SSL : In the self-supervised version of CL which uses augmentations X ′ 1 and X ′ 2 of the input modalities, the objective we aim to maximize is formulated as

Here the conditional terms are conditioned on the augmentations X ′ 1 and X ′ 2 , and we can similarly model it by concatenating the head outputs of X ′ 1 to X 1 and the head outputs of X ′ 2 to X 2 before feeding into the critic. We use Figure 5 to illustrate this. The way to obtain the learned representations is the same as described in FACTORCL-SUP.

Estimation of CMI : To estimate the conditional mutual information (CMI) I ( X 1 ; X 2 ∣ X ′ 1 , X ′ 2 ) , we can estimate the lower or upper bounds of true CMI [51, 55, 69]. However, direct sampling from the conditional distribution p ( x 1 , x 2 ∣ x ′ 1 , x ′ 2 ) can be expensive because we should consider a different conditional distribution p ( x 1 , x 2 ∣ x ′ 1 , x ′ 2 ) for each data pair x ′ 1 , x ′ 2 . Sordoni et al. [69] address this by concatenating the conditioning variable with the input in the critic: ϕ ( x 1 , x 2 , c ) , and showing that Conditional InfoNCE (Eq.(15) is a lower bound and estimator of CMI. This estimator can be made more exact by further importance sampling [69]. However, adding importance sampling [69] or using more accurate estimators [26] comes with a trade-off in complexity. Since we focus on capturing unique information to learn a scalable multimodal representation instead of accurately estimating the CMI, we leveraged a simpler version of the estimator from Sordoni et al. [69]: generating multiple augmented pairs from x 1 , x 2 , and concatenating the input x 1 , x 2 and each augmented pair x ′ 1 , x ′ 2 to define samples from the conditional distribution p ( x 1 , x 2 ∣ x ′ 1 , x ′ 2 ) . We argue that since augmentations do not significantly change the semantics of images, p ( x 1 , x 2 ∣ x ′ 1 , x ′ 2 ) could be approximated by p ( x ′′ 1 , x ′′ 2 ∣ x ′ 1 , x ′ 2 ) where x ′′ 1 , x ′′ 2 are other augmented pairs in addition to x ′ 1 , x ′ 2 . In this submission, we use one pair of augmented samples for consistency, but our code easily supports increasing the number of augmented pairs that can improve the accuracy of CMI estimation.

Regardless, our existing one-pair implementation can already show that our estimators are empirically comparable to CMI estimators with guarantees such as Mukherjee et al. [55] (Table 9), and our estimators empirically satisfy that the lower bound is smaller than the true CMI, and the true CMI smaller than the upper bound, i.e., Conditional InfoNCE ≤ true CMI ≤ Conditional InfoNCE-CLUB (also in Table 9). We refer the reader to Sordoni et al. [69] for tighter bounds for CMI.

OurCL-SUP : For this ablation, we remove the factorization and only learn Z 1 for X 1 and Z 2 for X 2 . The objective we use is the same as that of FACTORCL-SUP. The only difference is that we now take e 1 ( x 1 ) and e 2 ( x 2 ) as the learned representations for inputs x 1 and x 2 .

OurCL-SSL : This is a similar ablation for FACTORCL-SSL where we remove the factorization. The objective is the same as that of FACTORCL-SSL and we use e 1 ( x 1 ) and e 2 ( x 2 ) as the learned representations for inputs x 1 and x 2 .

Training Strategy : In regular contrastive learning using I NCE as the only objective, we can simply perform gradient descent to minimize I NCE, updating all parameters in the encoders, MLP heads, and critics. However, training any of the four methods above also involves the minimization of the I NCE-CLUB objectives, which require the optimal critic f ∗ from I NCE, as stated in Eq.(11). Therefore, within each iteration during our training, we need to first obtain the optimal critics for the I NCE-CLUB terms using the I NCE objective. We outline the training strategy using a sampling method in Algorithm 3. In this algorithm, L FACTORCL can be either L FACTORCL -SUP or L FACTORCL -SSL , and L NCE is the summation of I NCE objectives for the I NCE-CLUB terms. In particular, we have

We define ϕ to be the parameters of critics for the I NCE-CLUB terms, and θ corresponds to all the rest parameters in the network (parameters of encoders, heads, and critics for I NCE terms). In the outer loop, we update θ using the main learning objective L . In the inner loop, we update ϕ using the L NCE objective, which learns the optimal critics f ∗ needed to compute the I NCE-CLUB terms. Ideally in the inner loop we would update ϕ until convergence so we get a good approximation to the optimal critic. In practice we found sampling just one batch by setting k = 1 in Algorithm 3 works pretty well. Using only one iteration does not have a big impact on the convergence and still produces promising results. More importantly, it significantly reduces the time required for training, and allows our algorithms to have comparable running time to existing contrastive learning methods.

D.2 Datasets

Gaussian datasets for MI estimation : As shown in Figure 3 in the main text, we first demonstrate the quality of our proposed upper bounds I NCE-CLUB ( X 1 ; X 2 ) on a toy Gaussian dataset. We obtain the samples {( x i , y i )} from 4 multivariate Gaussian distribution with dimensions {20, 50, 100, 200}. In each dataset, we set the ground truth MI values to be {2, 4, 6, 8, 10}, and so we can compute the correlation ρ needed for achieving these MI values using the ground truth MI formula for Multivariate Gaussian: I ( X,Y ) = -d 2 log ( 1 -ρ 2 ) . At each true MI value we sample 4000 times using a batch size of 64.

Synthetic dataset with controlled generation : We generate data with controllable ratios of taskrelevant shared and unique information to analyze the behavior of each contrasive learning objective in Figure 1 in the main text. Starting with a set of latent vectors w 1 , w 2 , w s ∼ N( 0 d , Σ 2 d ) , d = 50 representing information unique to X 1 , X 2 and common to both respectively, the concatenated vector [ w 1 , w s ] is transformed into high-dimensional x 1 using a fixed full-rank transformation T 1 and likewise [ w 2 , w s ] to x 2 via T 2 . The label y is generated as a function (with nonlinearity and noise) of varying ratios of w s , w 1 , and w 2 to represent shared and unique task-relevant information. For experiments, we used 1-layer MLPs with 512 hidden size as encoders, and the embedding dimensions are 128 for both modalities. The heads on top of encoders are also 1-layer MLPs with the same hidden and output dimension as the input, and all critics are 1-layer MLPs with 512 hidden size.

Multimodal fusion datasets : We use a collection of 5 real-world datasets provided in MultiBench [45] and the IRFL dataset to test our method in the context of varying ratios of shared and unique information that is important for the task. In all the datasets below, the heads added on top of the encoders are 1-Layer MLPs with ReLU activations that map the encoder outputs to the same dimensions. All critics are also MLPs with 1 hidden layer of size 512 and ReLU activation.

MIMIC-III [38] (Medical Information Mart for Intensive Care III) is a large-scale dataset for healthcare which contains records of over 40,000 ICU patients in both forms of times-series data measured by hours and static data (age, gender, ethnicity) in the tabular numerical form. We use the preprocessed data provided in MultiBench [45], where the time-series data is measured every 1 hour in a 24-hour period and consists of vectors of size 12, and the tabular data consists of vectors of size 5. The task we use in the experiment is a binary classification on whether the patient fits any ICD-9 code in group 7 (460-519).

In the experiments, we use a 2-layer MLP with 10 hidden layer size for the tabular data modality, and map it to a vector of size 10. The time-series modality is encoded using a GRU with hidden size 30 and followed by a linear layer which projects the output to embeddings of size 15. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.

CMU-MOSEI [93] is the largest sentence-level multimodal sentiment and emotion benchmark with 23 , 000 monologue videos. It contains more than 65 hours of annotated video from more than 1,000 speakers and 250 topics. Each video is labeled with a sentiment intensity ranging from -3 to 3. In our experiments, we cast the intensity values to a binary classification on whether the sentiment is positive or negative. MultiBench [45] provides access to the extracted features of the vision, text, and audio modalities, and in our experiments, we use the vision and text features for doing contrastive learning.

In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.

CMU-MOSI [91] is a similar dataset for multimodal sentiment analysis created from 2 , 199 YouTube videos clips. The data focuses on videos that reflect the real-world distribution of speakers expressing their opinions in the form of monologues. The sentiment intensities are labeled continuously from -3 to 3. Again we cast the label into a binary classification on whether the sentiment is positive or negative, and we used the extracted vision and text features for contrastive learning.

In our experiments we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.

UR-FUNNY [27] is the first large-scale dataset for humor detection in human speech. The dataset consists of samples from more than 16 , 000 TED talk videos with speakers from diverse backgrounds sharing their ideas. The laughter markup is used to filter out 8,257 humorous punchlines from the transcripts. The context is extracted from the prior sentences to the punchline. Using a similar approach, 8,257 negative samples are chosen at random intervals where the last sentence is not immediately followed by a laughter marker. The task is to classify whether there is humor or not using the vision and text modalities.

Table 8: Additional experiments on CIFAR10 [43] and MNIST [19] datasets using our FACTORCL objectives on image classification.

In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.

MUSTARD [12] is a corpus of 690 videos for research in sarcasm detection from popular TV shows including Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholics Anonymous. It contains audiovisual utterances together with the textual context. We use the preprocessed features of the vision and text modalities for doing contrastive learning and performing sarcasm detection.

In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.

IRFL [88] is a dataset for understanding multimodal figurative languages. It contains 6 , 697 matching images and figurative captions (rather than literal captions) of three types of figurative languages: idiom, simile, and metaphor. The original data for the matching task is provided in the form of 1 caption, 3 distractor images, and 1 matching image. We convert it into a fusion task by only collecting the matching image and text pairs and assigning labels using the type of figurative language it belongs to.

For this dataset, we do not train from scratch. Instead, we performed continued pretraining using our proposed objectives on pretrained CLIP [61] models. We used the CLIP-VIT-B/32 model and its pretrained image and text encoders. We performed training for 10 epochs using the Adam optimizer with a 1e-6 learning rate.

D.3 Additional analysis and results

Fusion experiments : In Table 6 and 7 we present more detailed results on the Multibench [45] and IRFL [88] datasets computed from 5 independent runs. FACTORCL significantly outperforms the baselines that do not capture both shared and unique information in both supervised and selfsupervised settings, particularly on MUSTARD (where unique information expresses sarcasm, such as sardonic facial expressions or ironic tone of voice), and on MIMIC (with unique health indicators and sensor readings). There are also big improvements on the two sentiment analysis datasets MOSEI and MOSI, with 6 . 8% and 21 . 9% increases respectively when compared to SupCon [41].

In Table 7, we also see that FACTORCL substantially improves the state-of-the-art in classifying images and figurative captions which are not literally descriptive of the image on IRFL, outperforming zero-shot and fine-tuned CLIP [61] as well as continued pre-training baselines on top of CLIP. While the supervised version gives the best results overall, the self-supervised version with our proposed unique augmentations also performs better than independent augmentations, indicating that in the case without label information, we should always try to find and use unique augmentations when possible. In our experiments, we use word masking for text augmentations. For independent image

augmentations, we use cropping, flipping, and color jittering. The unique augmentation simply removes the cropping operation, as illustrated in Figure 4 in the main text.

Additional experiments on high shared information and low unique information : In Table 8 we include additional results using our method on the CIFAR10 [43] and MNIST [19] datasets. Our method outperforms the self-supervised contrastive learning on both datasets as expected, and roughly maintains the same performance as supervised contrastive learning. Therefore, in cases with abundant shared information (two modalities with high shared information or two different views generated from augmentations), our method recovers the performance of existing methods that do not capture unique information.

Experiments on CMI estimator verification : In Table 9 and Table 10, we include experiment results which verify that computing the conditional MI lower and upper bounds via concatenation indeed yields reliable estimates. In particular, we aim to verify that the the Conditional InfoNCE objective gives a lower bound of the CMI, and the Conditional InfoNCE-CLUB objective gives an upper bound of the CMI. We follow the experiment setups in [55], presenting the true CMI and results of our estimators on synthetic data with fixing dimension of representation Z and varying samples n , and fixing samples n and varying d z . The specific implementations used for conditional InfoNCE and conditional InfoNCE-CLUB can be found in Equation 13 and Equation 14, respectively. The results indicate that our Conditional InfoNCE gives estimations smaller than the true CMI, and Conditional InfoNCE-CLUB gives estimations greater than the true CMI. The performances are comparable to estimators in [55], suggesting that our method yields valid and competitive lower and upper bounds for CMI.

Empirical verification on InfoMin assumption : To verify the InfoMin assumption [72] ( I ( Z 1 ; Y ∣ X 2 ) = I ( Z 2 ; Y ∣ X 1 ) = 0 ), we use the same synthetic dataset as in Table 1 and measure I ( Z 1 ; Y ∣ X 2 ) . The results are shown in Table 11: we get I ( X 1 ; X 2 ) = 12 . 29 and I ( Z 1 ; Y ∣ X 2 ) = 0 . 4 . I ( Z 1 ; Y ∣ X 2 ) is much smaller and closer to zero than I ( Z 1 ; Y ∣ X 2 ) , indicating that the InfoMin assumption holds in practice.

Compute resources : All experiments in this paper are run on a single NVIDIA A100 GPU. It takes about 10 to 12 GPU hours to train the model on the CIFAR10 [43] for 300 epochs, and all the other experiments can be finished within 1 GPU hour using our specified hyperparameters.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FactorCL, a novel method for learning multimodal representations. The proposed method generalizes traditional MI maximization-based approaches by capturing both shared and unique information relevant to downstream tasks. Based on the information-theoretic perspective, the paper demonstrates that traditional CL approaches, based on shared task-relevant information maximization, perform sub-optimally in the presence of non-shared task-relevant information, which is an easily imaginable case in a multi-modal setting. From here, the paper proposes a novel CL objective and augmentation strategy to address the uniqueness gap. Experimental evaluation demonstrates that the proposed method outperforms several existing CL approaches in multi-modal setting.\n\nSTRENGTHS:\n1. The work is very timely. The paper addresses an important question of adopting CL methods for multi-modal scenarios.\n2. Paper is clearly written and easy to follow.\n3. The proposed method can be seen as an information-theoretic generalization of base contrastive learning approaches, which only address shared task-relevant information. This is a natural extension. \n4. Section 2 provides an intuitive information-theoretic analysis of multi-modal CL. Figure 2 gives a nice overview of the bounds and their role in the objective.\n5. Experimental results clearly demonstrate the benefits of FactorCL.\n\nWEAKNESSES:\n1. If the information is modality \"unique\", but task-relevant, shouldn't it end up in the shared task-relevant partition by means of simply optimizing for the shared information (as non-shared info is implicitly minimized this way) as standard CL methods do?\n2. If weakness 1 holds, then what is the reason why standard CL methods fail? Can it be solely attributed to sub-optimal augmentation, which limits the shared information?\n3. Table 2 reveals that most of the performance increase can be attributed to improved augmentation. Is it connected to Weaknesses 1 & 2?\n4. L 31-39. Isn't low shared information a direct consequence of high unique information (and vice versa)? If yes, then there is no need to define it as a separate imitation.\n\nMinor:\n - Figure 2 typo X1 (modality 2)\n - Eq (1) typo I(x1, x2, y)  -> I(x1, x2 | y) \n - Formatting of Table 2 can be improved, the top row X1;X2 is hard to read\n - L171-174 is confusing. Does it mean X1 and X2 are concatenated with the encoded features of (X1' & X2')?\n\nQUESTIONS:\nMy questions are in Weaknesses 1,2,3,4.\n\nLIMITATIONS:\nI suggest the paper include a separate section on the limitations of the methods, where authors elaborate on challenging scenarios, where they expect the proposed approach to not deliver substantial improvements.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This work addressed the problem of contrastive learning in a multimodal setting, particularly in capturing shared and modality-specific information regarding downstream tasks. Existing approaches assume that the information contained in different modalities is somewhat the same (redundant), but in the real world, it might not be the case. The authors proposed to factor the task-specific mutual information into shared and unique components. They provided a derivation for this factorization and their FactorCL algorithm optimized these two parts with approximated upper and lower bounds separately, under an assumption of optimal unimodal augmentation. The authors evaluated against chosen baselines on a collection of tasks in the MultiBnech benchmark and show improvements over these baselines.

STRENGTHS:
- The authors provided a lot of details about their approaches, from the analysis of CL algorithms, and their approximation to experiment results.
- Most of the simplifications look intuitively reasonable 
- On the chosen benchmark, the gains by their algorithm were decent

WEAKNESSES:
- The paper is packed with too many details which makes it hard to follow the main idea 
- On the experiment results, for example in Table 2, even though there were improvements over baselines, the results were too far below those reported in previous works, including the original MultiBench paper (https://arxiv.org/pdf/2107.07502.pdf - also Table 2). This casts doubt on the validity of the approach of this paper.

QUESTIONS:
- why don't you conduct experiments on various well-established vision language image-text datasets out there such as MS-COCO or Flickr30K?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
Based on the mutual-information theory, this paper proposes a new multi-modal contrastive learning method (FactorCL) to learn both shared and unique multi-modal task-relevant information, which captures task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bound. The method achieves significant influence in different real-world datasets.

STRENGTHS:
1. How to capture both the shared and unique task-relevant information in different modalities is interesting.
2. The theoretical analysis looks solid and cooperates well with empirical results.
3. The writing is well and easy to follow. Especially the figures represent the main ideas and are easy to understand.
4. The empirical results show significant improvements on different types of datasets.

WEAKNESSES:
1. It seems that the unique-augmentation step is strongly related to the task-relevant information of different modalities. For example, when the text is “a yellow flower”, the ColorJitter operation should be removed. And when the text is “a flower”, the ColorJitter operation is a useful data augmentation. Is it possible to design a strategy that can automatically select the appropriate data augmentations?
2. As shown in Table 2, FactorCL-SSL and FactorCL-SUP shows significant difference on some datasets. Especially, on MOSI, SimCLR and Supcon show similar performance while the performance of FactorCL-SSL and FactorCL-SUP have a large gap. Is it possible to provide more discussions about that?

QUESTIONS:
See weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FACTOR CL, a method for multimodal representation learning that captures both shared and unique task-relevant information, going beyond the common approach of focusing on shared information across different data modalities. FACTOR CL is based on three key contributions: factorizing task-relevant information into shared and unique representations, capturing and discarding task-relevant and irrelevant information through optimizing mutual information bounds, and utilizing multimodal data augmentations to approximate task relevance in the absence of labels. On large-scale real-world datasets, FACTOR CL achieves state-of-the-art results on six benchmarks. The premise of this paper is quite similar to the following work (Chaudhuri et al., Cross-Modal Fusion Distillation for\nFine-Grained Sketch-Based Image Retrieval: https://bmvc2022.mpi-inf.mpg.de/0499.pdf), however, the formulation, experimentations are different.\n\nSTRENGTHS:\n(1) This method is innovative in its attempt to model not only the shared information but also the unique information between different modalities, which is often overlooked by traditional Contrastive Learning (CL) methods. The framework is applicable to a broad range of settings, not just those where shared information is dominant. This allows it to handle diverse multimodal distributions, even those with substantial unique information.\n\n(2) The proposed method utilizes self-supervised data augmentations, which allows it to learn task-relevant information without access to labels. This enables the algorithm to learn in a more unsupervised manner, reducing the need for extensive labeled data.\n\nWEAKNESSES:\n(1) The method relies on certain assumptions, such as the optimal augmentation assumption, which may not always hold in real-world applications. If these assumptions do not hold, the effectiveness of the method may be significantly compromised. The method's performance might heavily rely on the quality of the data augmentation techniques used, which might not always be easy to decide or optimize.\n\n(2) Calculating mutual information can be challenging in practice, especially when dealing with high-dimensional data. Though approximations are used, they may not always accurately capture the true mutual information.\n\n(3) The method involves complex formalism and numerous mathematical assumptions, which could make it difficult to implement and adapt to different scenarios.\n\nQUESTIONS:\n(1) I wonder what are the limitations of current contrastive learning (CL) methods in modeling unique information in the context of multimodal data.\n\n(2) It would be nice to clarify how the proposed method distinguishes between shared and unique information across different modalities. What does the 'uniqueness gap' in this context mean? How does it affect the learning process?\n\n(3) How does the proposed framework handle situations where there is little to no shared information between modalities, or where most of the shared information is irrelevant to the task?\n\n(4) In the derivation of supervised contrastive learning objectives, how do the lower and upper bounds for mutual information terms impact the learning process?\n\n(5) What are the practical implications of applying semantic augmentations on each modality in the context of unsupervised contrastive learning?\n\n(6) How does the method ensure the learning of task-relevant information without access to labels in the case of unsupervised contrastive learning?\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations of the proposed method. Potential limitations could be found from my comments listed under the weaknesses and limitations sections.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of the proposed work is control over the information content of representations.  The most common form of contrastive learning leverages multi-view redundancy, where data points  are paired across different modalities or different augmentations in the same modality, and a contrastive loss is used to extract the redundant (or shared) information across the pairs.  The authors propose a methodology to extract the shared information and more---the information that is unique to each modality and still relevant to the task---with each component of the information in its own representation.  While contrastive learning based on multi-view redundancy involves maximizing a lower bound on the mutual information (e.g., InfoNCE), the proposed method requires multiple information terms, including the minimization of certain ones, for which the authors propose an efficient scheme of upper and lower bounds.  For scenarios where the task is not given, the task-relevant unique information is defined through a principled augmentation scheme.  The proposed method is benchmarked on a synthetic experiment, where the information content in each variable can be controlled, and then on several multi-modality datasets, where the goal is simply test set performance.\n\nSTRENGTHS:\nThe proposed work is well-motivated.  Control over the specific information content of learned representations is a valuable research pursuit, and the information theoretic analysis is reasonable.  The practicality of the methodology is an important strength: standard contrastive learning losses are all one needs, and the MI upper bounds are a clever re-utilization of critics trained concurrently for MI lower bounds.  The strategy of principled data augmentation as a replacement for task information is interesting.  Overall, the ideas pursued in this work have substantive originality and significance.\n\nWEAKNESSES:\nWhile the ideas pursued in this work are good, there are some issues that should be remedied before publication. \n- Most importantly, there doesn’t seem to be any support for the way in which the proposed work estimates conditional MI by concatenating the conditional variable(s) to the critic inputs.  This is a significant issue, as both objectives in the information decomposition rely on estimation of the conditional mutual information.  The details are not given (The Appendix refers to a nonexistent section: “In this work, we implement the conditioning in $p(x_1, x_2∣x_1^\\prime, x_2^\\prime)$ through concatenation and the details are in Appendix Sec.” (Appendix, L720).  Instead there is a seemingly irrelevant discussion of a kernel-based alternative to what is actually done.  \n- As currently written, Theorem 1 is incorrect, as shown by a simple counterexample.  The identity transform ($Z_1=X_1$, $Z_2=X_2$) “perfectly maximizes” eqn 2, and gives $I(Z_1,Z_2;Y) = I(X_1,X_2;Y)$.  The proof in the Appendix depends on the InfoMin proposition from Tian et al (2020), that posits good representations also minimize $I(Z_1;Y|Z_2)$.  This is not the same as saying that normal CL approaches necessarily minimize $I(Z_1;Y|Z_2)$.  The identity transform also optimizes Eqn 7, the objectives defining the unique representations, suggesting something else is needed in the objective. \n- The idealized augmentation strategy is a nice return to the self-supervised setting, but it feels too far removed from reality. L167: \"In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background.\"  No, the task relevant information is only a couple of bits, and $I(X;X^\\prime)=I(X;Y)$ is not achieved by only swapping out the background.  An augmentation that would achieve this extremely high bar would swap the object for another of its class, changing the image entirely.  Perhaps another way to see it: every pixel pertaining to the car in Fig 4 could be jittered, or the structure of the car could be changed quite significantly while still linking it to the original image X, and this represents a ton of information $I(X;X^\\prime)$ for any commonly used augmentation, far exceeding $I(X;Y) \\le H(Y)$.  While motivation by an ideal augmentation can be helpful, I think the section would benefit from a more sober take on the implications of realistic augmentations on the proposed information decomposition.\n- Important details seem to be missing.  \n  - What specifically is Cross+Self and Cross+Self+Fact?  It is referred to in vague terms, as a “category” (L219), “capturing a range of methods” (L218), and Cross+Self+Fact “is approximately done in prior work” (L221).  \n  - I like the setup of the synthetic experiments, but they are opaque as well.  Without more information, the accuracies of Fig. 1 mean little.  All methods achieve better than 90% for all settings, even SimCLR when all information is unique?  The text (main and appendix) is vague: “The label $y$ is generated as a function (with nonlinearity and noise) of varying ratios of $w_s$, $w_1$, and $w_2$ to represent shared and unique task-relevant information” (L239 and 874).  After searching the attached code, the label $Y$ is created with a threshold on a sigmoid of the average of vector components, which is no different than a simple hyperplane -- no nonlinearity and no noise.  With more clarity, the synthetic results could be a much stronger inclusion in the current work.\n\nQUESTIONS:\n- In theory, the interaction information (“task relevant shared information”) $I(X_1;X_2;Y)$ can be negative.  Are there any interesting consequences of such a scenario in the proposed method?\n- Is Definition 1 expressing what is intended?  I imagine the intent is to express $I(X_1;Y|X_2)>0$, but it does not seem correct.\n- What is RUS?  It is all over the attached code, without any reference in the text nor any comments in the code.  RUSModel, RUSAugment, RUS_CIFAR10, etc. \nMinor points:\n- There is a $1/K$ missing in the denominator of Eqn. 10\n- Typo in Fig 1, both input variables are labeled $X_1$\n- Typo in Line 113, the unique information is written as the shared info\n- Fig 2: it is confusing that the argmax LHS is replaced with information bounds on the RHS, and that $Y$ is replaced by $X_1^`$ and $X_2^`$ without explanation and without removing $Y$ from the Venn diagrams on the RHS\n- Some mutual information quantities seem to be in bits (Table 1) and others in nats (Fig 3), though it’s rarely stated.\n\nLIMITATIONS:\nYes, limitations were addressed in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}