Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
Authors: Paul Pu Liang
Zihao Deng
Martin Q. Ma
James Zou
Louis-Philippe Morency
Russ Salakhutdinov
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodal learning
contrastive learning
self-supervised learning
information theory
TL;DR: FactorCL is a new contrastive learning method that effectively learns both shared and unique task-relevant information across multiple modalities.

Abstract: In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data with only pairing information (e.g., image-caption or video-audio pairs). Underpinning these approaches is the assumption of multi-view redundancy - that shared information between modalities is necessary and sufficient for downstream tasks. However, in many real-world settings, task-relevant information is also contained in modality-unique regions: information that is only present in one modality but still relevant to the task. How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks? This paper proposes FactorCL, a new multimodal representation learning method to go beyond multi-view redundancy. FactorCL is built from three new contributions: (1) factorizing task-relevant information into shared and unique representations, (2) capturing task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bounds, and (3) multimodal data augmentations to approximate task relevance without labels. On large-scale real-world datasets, FactorCL captures both shared and unique information and achieves state-of-the-art results on six benchmarks.

FACTORIZED CONTRASTIVE LEARNING: Going Beyond Multi-view Redundancy
Paul Pu Liang",, Zihao Deng" Martin Q. Ma James Zou", Louis-Philippe Morency', Ruslan Salakhutdinoy"
CCrreegi Mellon University, Universityy of Pennsylvania, Stanforrd University pl pliangees. cmu. edu, zihaoden@cs. cmu, edu, qianl cmu. edu
Abstract
In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since can successfully learn representations from abundant unlabeled data with only pairing information (e.g.. image-caption vidco-audio pairs). Underpinning these approaches is the assumption of multi-view redundancy that shared information between modalities is necessary and sufficient for down- stream tasks. However, in many real- real-world settings, task-relevant information is also contained in modality- unique regions: information that is only present in modality but still relevant to the task. How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to down- stream tasks? This paper proposes FACTORCI CI, a new multimodal representation learning method go beyond multi-view redundancy. FACTORCL is built from three new contributions: (1) factorizing task-relevant information into shared and unique representations, (2) capturing task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bounds, and (3) multimodal data augmentations to approximate task relevance with- out labels. On large-scale real-world datasets, FACTORCI CL captures both shared and unique information and achieves state-of-the of-the-art results on six benchmarks,
Introduction
Learning representations from different modalities central paradigm in machine learning Today, popular learning method is to first pre-train general representations unlabeled multimodal data before fine-tuning on task-specific labels [10
48
40
47
48
[5]] These current multimodal pre-
training approaches have largely been inherited from prior work in multi-view learning [3
58
that exploit a critical assumption of multi-view redundancy: the property that shared information between modalities is almost exactly what is relevant for downstream tasks [70] 73 76). When this assumption holds, approaches based on contrastive pre-training to capture shared information [33
61 72), followed by fine-tuning keep task-relevant shared information [76), have seen successful applications in learning from images and captions
[61], video and audio Bl. speech and transcribed
text [58], and instructions and actions (21). However, our paper studies two fundamental limitations in the application of contrastive learning (CL) to broader real-wo world multimodal settings (see Figure for visual depiction and experimental results showing the performance drop of CL.):
Low shared information relevant to tasks: There exists wide range of multimodal tasks involving small amounts of shared information, such as between cartoon images and figurative captions (i.e., not literal but metaphoric or idiomatic descriptions of the images [52] (8]). In these situations, standard multimodal CL will only receive small percentage of information from the learned representations and struggle to learn the desired task-relevant information.
2. High unique information relevant tasks: Many real-world modalities provide unique information not present other modalities. Examples include healthcare with medical sensors or robotics with force sensors 45 49 Standard CL will discard task-relevant unique information, leading to poor downstream performance.
*First three authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1.0
Tesk-relevant shared information = 10 X;; Y)
0.98
A Acuraayy (%
0.95-
X
X2 (uodaliity 2)
(modality
0.91
1
Durs
SupCon
0.92
U = -(X
U2 12;1X;;i11
SimCLR
Cross+Self
Task-relevant unique information
(task)
Task-relevant unique information
0.90
52
(U1 + U2) U1 + U2)
Figure 1: Left: We define S ((x;; task-relevant shared information and U1 (())) U2- = I(X2; x) as lask-relevant unique information. Right: controllable datasets with varying ratios of S. U1. and U2, standard CL captures S but struggies when there is more U1 and U2. Our FACTORCL approach maintains best performance, whereas SimCLR [3] and SupCon [41] see performance drops as unique information increases, and Cross+Self [33 36 44
39 recovers in fully unique settings but suffers at other ratios.
In light of these limitations, how beyond multi-view redundancy? In this paper, starting from the first principles in information theory, we provide formal definitions of shared and unique information via conditional mutual information and propose an approach, FACTORIZED CONTRASTIVE LEARNING (FACTORCL for short), to learn these multimodal representations beyond multi-view redundancy using three key ideas. The first idea is to explicitly factorize shared and unique representations. The second idea is to capture relevrel information via maximizing lower bounds on MI and remove fask-irrelevant information via minimizing upper bounds on MI, resulting in representations with sufficient and necessary information content. Finally, notion of task relevance without explicit labels in the self- supervised setting is achieved by leveraging multimodal augmentations. Experimentally, we evaluate the effectiveness FACTORCI on a suite synthetic datasets and large-scale real-world multimodal benchmarks involving images and figurative language [8]] prediction of human sentiment [9]], emotions [93]. humor [27], and sarcasm [12]. as well as patient disease and mortality prediction from health indicators and sensor readings (38), achieving new state-of-the-ar ahr performance six datasets.
design suitable multimodal learning objectives that work
Overall, we summarize our key technical contributions here:
1. A new analysis of contrastive learning performance showing that standard multimodal CL fails to capture task-relevant unique information under low shared or high unique information cases.
2. A new contrastive learning algorithm called FACTORCL:
(a) FACTORCI. factorizes task-relevant information into shared and unique information, expand- ing contrastive learning to better handle low shared or high unique information.
(b) FACTORCI. optimizes shared and unique information separately, by removing task-irrelevant information via MI upper bounds and capturing task-relevant information via lower bounds, yielding optimal task-relevant representations.
(c) FACTORCL leverages multimodal augmentations to approximate task-relevant information, enabling self-supervised learning from our proposed FACTORCL.
2 Analysis of Multi-view Contrastive Learning
We begin by formalizing definitions of four types of information: shared, unique, task-relevant, and task-irrelevant information in multimodal data. To formalize the learning setting, we assume there exist two modalities expressed random variables X, and X₂ with outcomes 211 and 22, and task with the random variable Y and outcome y. We denote X-; as the other modality where appropriate.
Shared and unique information: We formalize shared and unique information by decomposing the total multimodal information 1(X1, X2; Y') into three conditional mutual information (MI) terms:
1(X1,X2: = (000==((X0)) X2x: 1(x1;1(X2)
+ ((x2;
(I)
S shared
U1 uniqueness in X,
U2 uniqueness X2
where X(x1, I ppx1, v) log p(21,22.1) dx dxzdy is the total MI between the p(a)=)))(1))
joint random variable X1, and the task Y. = 1(X1;X2;Y) ((x1; = f (((1,12)) log p(a, ax)) day ( daidxz dxz - T T(X1; (xx1;X2\V) is the task-relevant shared in-
1(X1;X2)
2

---

formation, X2X1; ((11, .2, v) log (aaly) p(z1 pip(zalu) de dozddy is the task-irrelevant shared information, and U1 = X(x); U22 = (X2; Y(X1) denote unique task-relevant information.
Limitations of CL: Current approaches for CL maximize mutual information / 1(X1; (and subsequently task-relevant shared information r(X1; X2; during supervised fine-tuning), without modeling unique information. These methods generally learn a pair of representations 73 76].
Z aig max 1 X2); Z2 arg max 1()1; (x2))
(2)
(x)))((X1)
For example, Z1 could encode images X1 and Zz encodes text X₂ via maximizing lower bound on I(X;; X2) using the NCE objective [38]. The NCE objective falls into a broader class of contrastive
learning methods [13 [3
28 41 61] that model the ratio between joint densities p(21, Z22 and
product of marginal densities p xr )((2 (22) using positive and negative samples [57] [59] 60, 79 [4] or
probabilistic classifiers [55] 777. It has been shown that contrastive learning works well under the
assumption of multi-view redundancy 4 [1 70 71 [6):
Definition 1. (Multi-view redundancy) Ee 0 such that I ((x1:X(X2) S e and ((2; S.
In other words, the task-relevant information in data is mostly shared across both views and the unique information is at most a small e. From a representation perspective, Tian et al. [72 further introduces the assumption that the optimal representation is minimal and sufficient, where all learned task-relevant information is shared information: 7(7;; Y|X2) = Z[2;; Y(X,) = O. While the multi- view redundancy is certainly true for particular types of multimodal distributions, it crucially ignores settings that display multi-view non-redundancy and unique information can be important, such as when health indicators, medical sensors, and robotic visual or force sensors each provide unique information not present in other modalities 45 49]
Definition 2. (Multi-view non non-redundancy) 3c O such that I(X1;Y(xx) > cor ((x;; >c
Under multi-view non-redundancy, we show that standard CL only receives weak training signal since it can only maximize a lower bound on shared information I (X1;; X2), and struggles to learn task-relevant unique information. We formalize this intuition with the following statement:
Theorem 1. (Suboptimality of standard CL) When there is multi-view non-redundancy as in Definition 2 given optimal representations (Zi, Zz) that satisfy Eq.(2 and (21;;||X;) = [(Z2;
122%, we have that
I(Z,, Zx; = X2x1, ((X)) 1(X2;V|Xx) = X(x1:X2) I(X1; < X2;
(3)
Correspondingly, the Bayes error rate Zz) == Ep(z) mmx P p( = 21, 21,22)
) of
contrastive representations { [Z,, Z2) for a downstream task Y is given by:
P. S 1 c -x 1 -1 [T -XX - HY)]
-((;;
(4)
H(Y]
(5)
We include proofs and a detailed discussion of the assumptions in Appendix B Based on 3 T T(Zi, Y) decreases with higher task-relevant unique information T(X:: and (X2; (I);; we call this the difference X2; 1(21, [[; Y) the uniqueness gap. The uniqueness gap measures the loss in task-relevant information between the input and encoded rep- resentation: as task-relevant unique information grows, the uniqueness gap increases, In addition, Zz: Zz; Y) also drops with lower X2) (i.e.. two modalities sharing little information to begin with), or with higher I 1(X1;X211 X2(Y) (i.e., when the shared information is mostly task irseeerrelev.nt.. Similarly, in (5. the Bayes error rate of using (Z1, Z2) for prediction is directly related the task-relevant information in (2, Z2]: error on the downstream task increases with higher unique information and lower shared information.
3 FACTORIZED CONTRASTIVE LEARNING
We now present a suite of new CL objectives that alleviate the challenges above and work at all ranges of shared and unique information. At high level, we aim to learn set of factorized representations 7s, 78, ZU; ZU, representing telk-rele information in X, shared with X2. in X₂ shared with
X1, unique to X1, and unique to X2 respectively. As common in practice [61 72), we define
3

---

4
Factorized Contrastive Learning
Zu
arg my (()))
xx
INCE pore-cuaaKK,122 X₂)
IicelXiiXX (x Xx.
X
Lower bound
Upper bound
Augmentatoo
Zzp Z5 = arg yaz IZz Zz: Y) 1x X2)
cxce- Xy (X,
Unique
Zu arg HIN P2; x1X,) = IycE(Xx;XX INCE-CuoOO X.)
INcclKi X21
Figure 2: FACTORCL: We propose self-supervised CL method learn factorized representations Zs, Zs, Z01 and Zuz capture task-relevant information shared in both X₁ and X2, unique to X1, and unique X2- By starting with information-thect thcooorticcc first principles of shared and unique information, we design contrastive estimators to both capture task-relevant and remove mask-irrelevan information, where a notion of task-relevance without explicit labels is afforded by a new definition of multimodal augmentations X X2. Lower bounds are in green and upper bounds in red.
neural networks fo with trainable parameters & extract representations from inputs X₁ and X2- Learning these parameters requires optimizing differentiable and scalable training objectives to capture task-relevant shared and unique information (see overview in Figure 2
Zs, arg max 1(21; 21=1o(X.)
Zs, arg I 1(22; 2x=1o(X,)
(6)
ZU, arg max 1(21;1|x,)
ZU2 arg max T(Z2; Y|x1) f2=fo(Xa))
(7)
21 fo())(
where 1(21; X2; Y) ((1;;X2) [(11; is the shared information and (2;) ((2;;X)) 1();) X,\Y) is the unique information. One important characteristic of our framework that when unique information is zero: Y|
and 1(x;; Y| 0, or all shared I(X;; X2). our framework recovers standard CL as in (2. However, as we have previously indicated and will show empirically, these assumptions can easily be violated, and our framework enlarges Eq. 2 to cases where unique information is present.
information is task-relevant:
The learned 7s can then be used input linear classifier and fine-tuned to predict the label for multimodal classification or retrieval tasks. However, the shared and unique MI terms above are often intractable in practice. In the next section, we will build up our method step by step, eventually showing that each term in Eqs. ( 7 can be approximated as follows:
S= 1=x;;;;;;;) 2 (NCE(Xx; - INCE-CLUB
(8)
U, - U(= 2 INCF (X,; X/) INCF.CI.UR X2) (NCE(XI; X
where INCE and INCE-CLUB are scalable contrastive estimators (Section 3.1 and X1 are suitable data augmentations (Section 3.2) on each modality. Overall, these equations can be interpreted as both positive and negative signals to learn representations for S and U. For shared information S, the estimator maximizes task-relevant shared information via INCE (xcE(X1;1) task-irrelevant shared information via a novel upper bound -INCE-CLUB (iIcE-CLUB(X;;XXX x, For unique information U, we capture task-relevant uniqueness via +INCE( (x:X() while non non-unique informa- tion removed via -(INCF. C.NNCE.C..UBBB (X1; X2) - INCE(X) X2 X))). the following sections, we derive this final objective step-by-step: (1) approximating the MI objectives in S and U with CL. estimators, (2) relaxing the dependence on labels Y with self-supervvsed data augmentations, finally (3) discussing overall training and implementation details of end-to-end self-supervised learning.
while removing
3.1 Supervised FACTOR CI. with shared and unique information
To capture shared and unique information via an objective function, we will need to maximize lower bounds for all terms with a positive sign in (q. and 9 (X,;; X2), (X,; Y), . X2)))
and minimize upper bounds for all terms with a negative sign (7 (X1; (2): ( (Xx;; (Xx:X2/1)) X2) Our first theorem derives general lower and upper bounds for MI terms variants of contrastive estimation:

---

Gaussian,
Gaussian,
Caussian, dim
True ICC
Mutual Information
IICE
Figure 3: Estimated INCE lower bound [58] and our proposed upper bound INCE-CLUB on sample distributions with changing mutual information: our upper bound is tighter, more accurate, and more stable than LCLUB upper bound [I6]. and also comes for 'free' via jointly estimating both lower and upper bounds simultaneously. We find that as dimension increases, the TeLUB estimator collapses zero and longer tracks true MI.
Theorem 2. (Contrastive estimators for X2)) Defining the NCE and NCE-CLUB estimators,
exp f(11,
IxcE(X1;X2) E - 222)(22)
log Σx exp f (21,
(10)
INCE-CLUB(X;) = B
[1 (11,
E pap((())
[f (7),
(11)
spmp((2)
where f (x1. is the optimal critic from INCE plugged into the ICLUB objective [τ6.. We call the proposed plug-in objective Eq. 77. INCE-CLUB, and obtain lower and upper bounds on I(X1;
INCE X2) (((1;:22) s INCE.CLUB (2):
(12)
follows from Oord et al. (58): optimizing the objective leads an optimal critic [60] f = log logp(21122) + c(²1), with deterministic function (.). Plugging optimal critic f into INCE cLUB (X1; X2) cancels out the e(z1) term and yields
Proof. The lower bound INCE (X1; x,) ) ((X1;X,))
INCE CLUB X1; X2) and T(X1: INCE- CLUB- We include detailed proof in Appendix C.
INCE-CLUB X2) gives desired upper bound of [(X;; X2) "for free" while avoiding separately optimizing lower bound and upper bounds. In Figure 3 we show these two bounds in practice across two Gaussian distributions X1 and X2 with varying amounts of MI [(X; X2). We use the second formulation of JCLUB 16]. which assumes p( (22) to be unknown. Our upper bound is empirically tighter (see Figure and comes for "free" via jointly maximizing the lower bound INCE. These lower and upper bounds can be seen as new contrastive objectives over positive and negative (I1, (2), pairs, enabling close integration with existing pre-training paradigms. Finally, we can similarly obtain bounds for the conditional MI (X; (11)))((((0))) 1(X1; s INCE-CLUB (X1; X2/Y): X2/;)))))
exp (
(xcI(XX;X22))) Ep(x)
E pla =(=p(1))))
log
(13)
Σ. exp (21,
INCE:cLUs(xx::X)1Y Ep(x)
E Ez (21,
(ey-i( [f [f (x1, a3-p(1))))
(14)
These two bounds result in conditional CL objectives 74 78
they differ critically from standard CL methods since they capture task-irrelevant shared information that remains between X, and X₂ after observing Y. This task-inrelevant shared information is removed by minimizing its upper bound. Note that ((11422, y) here denotes a different function from f(11 22) in Eq. 10). as the general forms different (taking in 21, versus æ1, T2, v). f(I1, 22, y) can be implemented in different ways, c.g., concatenation [69]
h((2) where g(), fi() are trainable encoders and [x1, y] denotes
3.2 Self-supervised FACTORCL. via multimodal augmentations
The derivations above bring about supervised CL objectives with access to Y 41 For unsupervised CL. [58] 72). we derive similar objectives without access to by leveraging semantic augmentations on each modality. Denote X' some augmentation of X (c.g., rotating, shifting, cropping). Under
5

---

the optimal augmentation assumption from Tian et al. [72] (restated below). replacing Y with X' in our formulations enables learning of task-relevant information without access to labels:
Definition 3. (Optimal unimodal augmentation) A72) Xi is an optimal unimodal augmentation for X1 when XX"; = I(X;Y), which implies that the only information shared between X and X' is task-relevant with no irrelevant noise.
This assumption is satisfied when all information shared between X and X' is task- which implies that the augmentation keeps task- information constant while changing irreevatt information. In the case of image classification, task-relevant information is the object in the picture, while task iireleettl information the background. By performing two separate unimodal augmentations giving X1 and X2, we can substitute contrastive estimators in Eqs. (13 and (4). by replacing [X;;; Y) terms with I(X;; X)) and replacing I [X1: XY)
terms with I 1(X X2 x:
exp [2) 22 21,2%
Xx X2) Ex(ag,ag)
E Em Shnat-p(anaaaaff.oo pl-plas
log
(15)
Σx exp (21, 24 2,,
(xcF, x²) Estoy x5)
B es mg) ( 22,
E parpl (a)) (iap(ra
[f"
(1I, +1 2b)1]
(16)
The objectives can be seen as conditional contrastive learning on augmentations (x)) Here again ([11,22, x2) is different from the critics in Eqs. [3] because of the different general forms. We implement f() here ([[21, =[12,,='1)) ")"))) x²2)) where 90), h() are trainable encoders specific for each modality and [ar, 22 denotes concatenation. This concatenation is justified by the CMI estimators in Sordoni et al. [69]. who show that concatenating the conditioning variable with the
input in the critic f( 22, "1.0 (()) yields a Conditional InfoNCE estimator (Eq. 15) that is a lower bound for CMI. However, the exact Conditional InfoNCE estimator learns a different conditional distribution p(21, x22 [1 .o6) for each augmented pair o' J.g. which can be prohibitively expensive. We could approximate this by creating multiple augmentations of single paired æ1, 22. Our code uses one augmented pair z1 242 for each I1, 222 but could be extended to multiple pairs, and we find this simple approach yields consistent CMI lower and upper bounds that are empirically comparable to existing CMI estimators [55] 69|. We include full comparisons and implementation details in Appendix D.I, and in Appendix C.2 we discuss an alternative interpretation based on viewing CL as kernel learning which permits using conditional kernel estimation for our objectives.
Although we find this method to work well in practice, more careful analysis reveals that separate unimodal augmentations X and X22 each satisfying T(X;; T( Y) do not together satisfy T(X;; X2 T(X;; X2|X1, X²) needed for the substitution in Eqs. (1)) and (16) to hold with equality. To satisfy this property exactly, we define optimal multimodal augmentations:
Definition 4. (Optimal multimodal augmentation) X1 and X are optimal multimodal augmentation for X1 and X2 when T(X1, X2; X X2) T(X,, X2; Y), which implies that the only information shared between X1 and X,, X2 is task-relevant with no irrelevant noise.
We satisfy X2; X²) I (X1,X2; Y) using two steps:
Unimodal aug: X s.t. T(X;; X1) = I(Xi; Unique aug: X² s.t. I(x2; I[X;;
(17)
(18)
We call the second step unique augmentation: after observing X1, we create augmented X from X₂ to keep task-relevant information not already in X,. To empirically satisfy optimal multimodal augmentations, we avoid augmentations in one modality that will remove or strongly destroy infor- mation shared with the other modality. For example, in image captioning, we should avoid image augmentations such cropping that destroy information from the caption (c.g., cropping object parts referred to by the caption), and instead, only augment images via flipping or color jittering which retains all caption information. 4iswrs shows example of unique augmentation that satisfies these conditions. In our experiments, we will show that our augmentations consistently perform better than standard augmentations (Tablee3), suggesting that approximately satisfying Eqs. and (T8) can be empirically sufficient, which is simple and straightforward to implement on real-world datasets.
3.3 Overall method and implementation

---

Algorithm Standard multimodal CL.
Algorithm FACTORCI.
Require: Multimodal dataset
Require: Multimodal dataset (
Initialize networks f()). while not converged do for sampled batch {æ), 222 do
Initialize networks f(-).
while not converged do for sampled batch { 22) do
Estimate INCE (- X2) from Eq.
zj Augment (x)
C -= - 1==INce(X1:200 (X1;
2 + Unique-Augment
Update f(-) minimize L
Plug x1 and x/2 into Eq. 15 and
end for end while return f())
Estimate S, U1. U2 from Eq. and
+ U1 U2)
Update f(-) to minimize L end for end while return f(-)
Task-relevant information
Input
The final algorithm sketch is in Algorithm 2 which we compare against standard CL in Algorithm | can be shown that FACTORCL learns all the task- relevant information from both modalities:
X1 (Text)
The car is as fast a cheetah.
Unique X "cheetah"
Unique X highway pixels
X (Image)
Theorem 3. (Optimality of FACTORCL) If Zs, Zso, Ze Zuz perfectly maximize Eqs. 7 and the estimations in Eqs. and 99) are light, we obtain T 12;1 T( 7 + ((Z,,; I(77y,; ((Zss) suggesting that FACTORCL learns both shared and unique task-relevant information.
Shared between X, and Xz: fast car
y
(image and caption are matched)
Standard Aug for Xg
Unique task information X2 destroyed: highway pixels
We include the full proof in Appendix In prac- tice, while we do not expect perfect estimation MI quantities and maximization with respect to MI ob- jectives, we include implementation details regarding architectures and contrastive objectives that improve empirical performance Appendix [..]
Crop + Fllp
Unique task information Xg preserved:
Unique Aug for X
highway pixels
No crop, flip only
Complexity: Compared to heuristic combinations of cross cross-mode modal and singlc-modality CL [33 36 44 64 81 85 89), our approach does not significantly in-
Figure 4: Standard unique augmentations for the figurative language [8]] dataset. After aug- menting text modality X, independently (same for both augmentation types), illustrate their differ- ences for image augmentation: unique augmenta- tion on images should avoid removing information referred to by X₁ (the text). The text mentions that the car is fast so unique augmentation for images should not remove the highway pixels of the image which suggest the is fast.
crease complexity: (1) upper bounds on MI can be es- timated "for free" by directly plugging in the optimal critic from INCE, (2) removal of task-irrelevant infor- mation via I [Xx1; X .x)) shares encoders with INCF, and (3) separate unimodal augmentations per- form empirically well. We describe some extensions (C.4 C.44
of other self-supervised methods Appendix
Experiments
We run comprehensive experiments on suite of synthetic and large-scale real-world datasets with varying requirements of shared and unique task-relevant information, comparing our FACTORCL method to key baselines:
1. SimCLR 13: the straightforward method of cross-modal (X,,X) contrastive learning.
Cross+Self [3 36 64 85 89): captures a range of methods combining cross-modal (X X2)
CL with additional unimodal (X,,x;) CL objectives. This category also includes other ways of preserving unique information, such as through (variational) autoencoder reconstructions [8I]
3. Cross+Self++Fct [86] 89]; : A factorized extension of Cross+Self, which is approximately done in prior work that adds separate (typically pre- me-trained) unimodal encoders for each modality.
4. SupCon [1], which learns I (Xx::x using CL conditioned on Y from labeled data.
We also carefully ablate each component of method and investigate factors, including training data size and choice of augmentations. The intermediate ablations that emerge include:
1. FACTORCL-SUP. The supervised CL version which uses labels Y in Eqs. 3 and

---

Table 1: We probe whether contrastive representations learned by classic CL methods and FACTORCL contain shared wa or unique wI, wy information. FACTORCL captures the most unique information.
Model Representations
SimCLR
Cross+self
SupCon
FACTORCL
Z1
Zz
Z1
Zz
Zi
Z2
Zu,
Zu:
Zs.
Zg,
4.45
0.16
4.39
0.14
5.17
0.19
7.83
0.03
6.25
0.04
T T(Z; w2)
0.17
3.92
0.13
4.26
0.23
5.17
0.06
7.17
0.05
5.79
T( T(Z:ws)
12.61
12.06
11.30
11.47
7.48
7.17
9.47
9.89
10.13
9.40
2. FACTORCL-SSL: The fully self-supervised version of our approach replacing Y with multimodal augmentations X1 and X to approximate the task.
3. OurCL-SUP: FACTORCL-SUP but removing the factorization so only two features Z, is optimized for both T(X1; X2; Y) and 1(X1; X2),, Z2 optimized for both T(X1 X2; Y) and 7(x;; YIXi).
4. OurCL-SSL: FACTORCI-SSL. but also removing the factorization in the self-supervised setting.
The formulation of each ablation and implementation can be found in Appendix D.I
4.1 Controlled experiments synthetic datasets
Synthetic data generation: We begin by generating data with controllable ratios of task-relevant shared and unique information. Starting with a set of latent vectors W1, W2, W, ~ ~N(0 M d 50 representing information unique to X1, X2 and common to both respectively, the concatenated vector [w1, We] is transformed into high-dimensional #1 using a fixed transformation T1 and likewise [W2, We] to 22 via T2. The label y is generated as a function (with nonlinearity and noise) of varying ratios of Wss W1, and W2 to represent shared and unique task-relevant information.
Results: In Figure 1 we show our main result on synthetic data comparing FACTORCI. with existing CL baselines. FACTORCL consistently maintains the best performance, whereas SimCLR [3] and
SupCon [11] see performance drops as unique information increases. Cross+Self 33 36 89
recovers in fully unique settings (x-axis= 1.0) but suffers at other ratios.
Representation probing information: We run probing experiment to compute how well different contrastive representations capture shared and unique information. In Table I for the Z, 'ss learned by each method, we approximately compute I (Z;; wi ), Z2; w2), and I [2,; w with respect to ground truth generative variables Ws. W1, and W2. As expected, existing methods such as SimCLR capture smaller amounts of unique information (roughly 4 bits in 7(Z;; w1) ) and 7(Z;; W2))), focusing instead on learning T T(Z;; w:)) (12 bits). Cross+self captures slightly larger T(Z;; w2) 4.26, and SupCon with labeled data captures up to 5 bits of unique information. Our FACTORCL approach captures 7 bits of unique information and maintains 10 bits of shared information, with total information captured higher than the other approaches. Furthermore, (Zs, Zs,} capture more information about Ws. Zu, about wi, and ZU, about w2, indicating that factorization in our approach successful.
4.2 Self-supervised multimodal learning with low redundancy and high uniqueness
Multimodal fusion datasets: We use a large collection of real-world datasets provided in Multi- Bench 45) where we expect varying ratios of shared and unique information important for the task, to compare FACTORCL with other CL baselines:
1. MIMIC [3
mortality and disease prediction from 36, 212 medical records (tabular patient data and medical time-series sensors from ICU).
2. MOSEI [93] multimodal sentiment and emotion benchmark with 23, 000 monologue videos.
MOSI 911: multimodal sentiment analysis from 2, 199 YouTube videos.
4. UR-FUNNY [27]: a dataset of humor detection from more than 16, 000 TED talk videos.
5. MUSTARD [2]: corpus of 690 videos for research in sarcasm detection from TV shows.
6. IRFL [88): 6, 697 matching images and figurative captions (rather than literal captions).
Together, these datasets cover seven different modalities from the healtheare, affective computing, and multimedia research areas and total more than 84, 000 data points. For MIMIC with tabular and medical sensor inputs, we train self self-supervised CL models on top of raw modality inputs. For IRFL with image and caption inputs, we start with a pretrained CLIP model [61] and perform continued pre-training to update CLIP weights with our FACTORCL objectives, before linear classifier testing. For the remaining four video datasets, we train sule-supe CL models starting from standard pre-extracted text, video, and audio features 45) Please refer to Appendix D.2 for experimental details. We release code and models at https::////// /tt:://gittuu www.:hhw..cc.c.pppecee hwtpee/pppiinngge 7 hta.aanngggga/aacorro.oooa FactorCL
8

---

Table 2: Results on MultiBench
45 datasets with varying shared and unique information: FACTORCL achieves strong results vs self-supervieed (top rows) and supervised (bottom rows) baselines that do not have unique representations, factorization, upper-bounds to remove irrelevant information, and multimodal augmentations.
Model
(21; (X;; X1) (X1: X2 (XT)
Fact MIMIC MOSEI MOSI
UR-FUNNY MUSTARD
SimCLR 13
66.67%
71.03% 46.21%
50.09%
53.48%
Cross+eelf 81
x
65.20%
71.04% 46.92%
56.52%
53.91%
89
x
x
65.49%
71.07% 52.37%
59.91%
53.91%
OurCL-SSL
/
/
65.22%
71.16% 48.98%
58.79%
53.98%
FACTORCL-SSL
/
67.34%
74.88%
52.91%
60.50%
55.80%
SupCon +]]
x
67.37%
72.71%
47.23%
50.98%
52.75%
OurCL-SUP
7
X
X
68.16%
71.15% 65.32%
58.32%
65.05%
FACTORCL-SUP
76.79%
77.34% 70.69%
63.52%
69.86%
Multimodal fusion results: From Table 2 FACTORCL significantly outperforms the baselines that do not capture both shared and unique information in both supervised and self-supervised settings, particularly on MUSTARD (where unique information expresses sarcasm, such as sardonic facial expressions ironic tone of voice), and MIMIC (with unique health indicators and sensor readings). In T we also show that FACTORCL substantially improves the state-of-the-art in classifying images and figurative captions which are not literally descriptive of the image on IRFL, outperforming zero-shot and fine-tuned CLIP 61 as well as continued pre-training baselines on top of CLIP.
Modeling ablations: In Table 2 we also carefully ablate each component in our method and indicate either existing baselines newly-run ablation models.
Factorized representations: In comparing FACTORCL-SSI. with OurCL-SSL, and also FAC- TORCL-SUP with OurCL-SUP, we find that factorization is critical: without it, performance drops on average 6.1%, with performance drop as high as 8.6% for MIMIC
2. Information removal via upper bound: By comparing FACTORCL with SimCLR, Cross+ and Cross+Self+Factt, and SupCon that only seek to capture task-relevant information via con- trastive lower bounds on MI, we find that separately modeling the task-relevant information (to be captured) and task-irrelevant information (to be removed) is helpful. Without removing task-irrelevant information via the upper-bound objective, performance drops average 13.6%, with performance drops as high as 23.5% for the MOSI dataset. We also found that training was more difficult without this objective, which is expected due overwhelming superfluous information from the dataset [93]. 3. Multimodal augmentations: Finally, investigate the differences between separate uni- modal augmentations (FACTORCL-IndAug in T) versus a joint multimodal augmentation (FACTOR CL-SSL on the IRFL dataset. We choose this dataset since its images and captions the easiest to visualize (see Figure(ffoc augmentations from both strategies). In the self-supervised setting, we find that multimodal augmentations achieve 95% performance, higher than the 92% for separate unimodal augmentations, and both outperform baselines SimCLR and Cross+
Ablations on S,U, and Uz: In Table 4 we also test FAC- TORCL when training linear classifiers on top of only shared and unique Zu., Zu, separately. We call these models FACTORCL-S, FACTORCL-U,, and FACTORCL-U;, Immediately, we observe that performance drops as compared to the full FACTORCL model, indicating that both shared and unique information are critical in real-world multimodal tasks. As expected, the best-performing submodel is the one that captures the region with the largest amount of task-relevant information: MOSEI and MOSI are known to include a lot of redundancy and unique information since language is very important for detecting sentiment [93] so FACTORCL-S and FACTORCL-U, perform best. For sarcasm detection on MUS- TARD, video information is most important with FACTORCL- Ui performing best (59.4%), and ablation models are also the furthest away from full multimodal performance (69.9%). This is aligned with intuition where sarcasm is expressed through tone of voice and visual gestures (high U1), as well as from contradictions between language and video (higher multimodal performance).
Table 3: Continued pre- pre-training on CLIP with our FACTORCL objectives on clas- sifying images and figurative language.
Task
IRFL
Zero-shot CLIP [6]
89.15%
SimCLR 13
91.57%
Cross+Self 81
95.18%
FACTORCL-IndAug
92.77%
FACTORCL-SS
95.18%
Fine-tuned CLIP [6]
96.39%
SupCon 41
89.16 10%%
FACTORCL-SUP 98.80%

---

Table 4: We ablate using only shared representations (7,, 7.,)) unique representation Zu, and Zu. separately for prediction. Both shared and unique information are critical in real-world multimodal tasks.
Model
MIMIC MOSEI
MOSI
UR-FUNNY MUSTARD
FACTORCL-S
63.77%
77.17%
70.12%
63.42%
57.25%
FACTORCL-Ui
55.90%
77.06%
70.11%
62.00%
59.42%
FACTORCL-U2
69.08%
71.01%
52.33%
54.35%
53.62%
FACTORCL-SUP
%
77.34%
70.69%
63.52%
69.86%
Additional results: Appendix D.3
we also verify FACTORCL. in settings with abundant shared
information, where we expect to recover the same performance as standard CL [13 58 72].
Related Work
Contrastive learning a successful self-supervised learning paradigm for computer vision
13
[4|25] 28 (58. natural language (24] [54] [56.. speech 58 63], and multimodal tasks [ 37
61 Its
foundational underpinnings are inspired by work in multiview information theory 23 70 72 76
studying the shared information between two views and whether they are necessary or sufficient in predicting the label. Recently, Wang et al. [8] and Kahana and Hoshen [39] discuss the limitations of assuming multiview redundancy and propose autoencoder reconstruction or unimodal contrastive learning to retain unique information, which resembles the Cross+self baselines in our experiments. We refer the reader to Shwartz-Ziv and LeCun [67] for comprehensive review on multiview and
contrastive learning. Our work also relates to conditional contrastive learning [17 51 78 87], where
positive negative pairs supposed to sample from conditional distributions.
Multimodal contrastive learning aims align related data from different modalities, typically provided as positive pairs. This could be done via optimizing contrastive objective for inter- modality pairs [
37 or both intra- and inter-modality data pairs B3 36
R9 Our
work also relates to factorized representation learning, which primarily studies how to capture modality-specinf information primarily in each modality and multimodal information redundant in both modalities [32 75
Prior work has used disentangled latent variable models [ 30 32 75
mixture-of-experts
66 or product-of-exxerts
83 layer to explain factors in multimodal data.
has been used to study several phenomena in multimodal learning, 92 and multi-view learning (3 76). Due its theoretical importance, several lower and upper bounds have been proposed for practical estimation (58/60] [84). We build on the CLUB upper bound [66] to create a more accurate and stable bound. Our characterizations of shared and unique information are also related to partial information decomposition 82]. co-
Information theory 18
65
including co-learning 62
information 7 80), interaction information (53) and cross- doosin disentanglement [35 research.
6 Conclusion
This paper studied how standard CL methods suffer when task-relevant information lies in regions unique to each modality, which is extremely common in real-world applications such sensor placement, medical testing, and multimodal interaction. In response, we proposed FACTORCL. new method expanding CL. techniques through the use of factorized representations, removing task-irrelevan information via upper bounds on MI. and multimodal data augmentations suitable for approximating the unobserved task. Based on FACTORCL's 's strong performance, there are several exciting directions in extending these ideas for masked and non-contrastive pre-training; we further discuss broader impacts and limitations of this line of work in Appendix
Acknowledgements
This material is based upon work partially supported by Meta, National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH125740, ROIMHI 32225, M019699951 and R21MM130767. PPL is supported in part by Siebel Scholarship and a Waibel Presidential Fellowship. RS is supported in part by ONR grant N000142312368 and DARPA FA87502321015. One of the aims of this project is to understand the comfort zone of people for better privacy and integrity. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred. Finally, we would also like to acknowledge feedback from anonymous reviewers who significantly improved the paper and NVIDIA's GPU support.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 11242066-422111 2021.
[2] Jean- Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self superviee multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
the IEEE international [3] Relja Arandjelovic and Andrew Zisserman. Look, listen and leam. In Proceedings the conference computer vision, pages 609-617, 2017.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances neural information processing systems, 32, 2019.
[5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449-12466 2020.
[6] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivarrince--ovvainnn regularization for self-supervised learning. In International Conference on Learning Representations, 2021.
[7] Anthony J Bell. The co-information lattice. In Proceedings of the fifth international workshop on independent component analysis and blind signal separation: ICA, volume 2003, 2003.
[8] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learningg A review and new perspectives. TPAMI, 35(8), August 2013.
[9] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai, Man to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
[10] Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki, and Desmond Elliott. Multimodal pretraining unmasked: A meta-amalysis and unified framework of dision-and-languaal berts. Transactions of the Association for Computational Linguistics, 9:978-994, 2021.
[11] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9992 2020.
[12] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously_ perfect paper). arXiv preprint arXiv: 19069990688888 018100185.000 2019.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607, PMLR, 2020.
[14] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[15] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training supf-supe vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640 9649, 2021.
[16] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning. pages 1779-1788. PMLR, 2020.
[17] Jianfeng Chi, William Shand, Yaodong Yu, Wei Chang, Han Zhao, and Yuan Tian. Conditional supervised contrastive learning for fair text classification. arXiv preprint arXiv:2205.1142 2022.
[18] Thomas M Cover and Joy A Thomas. Information theory and statistics. Elements of information theory, L (1):279-33)) 335, 1991.
[19] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 19(6))141112. 2012.

---

[20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[21] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35: 35603-35620, 2022.
[22] Meir Feder and Neri Merhav. Relations between entropy and enorr probability. IEEE Transactions on Information theory, 40(1):259-266 1994.
[23] Marco Federici, Aujan Dutta. Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust represen- tations via multi-view information bottleneck. arXiv preprint 20000:2002.00017, 2020.
[24] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint (04.02204 2021.
[25] Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent-a new approach to self-supervised learning. Advances neural information processing systems, 33:21271 21284, 2020.
[26] Qing Guo, Junya Chen, Dong Wang. Yuewei Yang, Xinwei Deng, Jing Huang, Larry Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. Advances in Neural Information Processing Systems, 35:28319-2838 2022.
[27] Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Phill-piil Morency, and Mohammed Ehsan Hoque Ur-funny: multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 1046-2056- 2019.
[28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 9729-9738, 2020.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[30] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Leamnnng basic visual concepts with constrained variational framework. 2016.
[31] R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018
[32] Wei-Ning Hsu and James Glass. Disentangling by partitioning: representation learning framework for multimodal sensory data. arXiv preprint arXiv:1805. 1224. 2018
[33] Po- Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig. Florian Metze, and Alexander Hauptmann. Multilingual multimodal pre-training for zero-shot cross-lingual transfer vision-language models. arXiv preprint 203: 08849, 2021.
[34] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34: 10944-10956, 2021.
[35] HyeongJoo Hwang, Geon-Hyeong Kim. Seunghoon Hong, and Kee-Eung Kim. Variational interaction information maximization for cross-domain disentanglement. Advances in Neural Information Processing Systems, 33:22479-222 224911, 2020.
[36] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. Mural: multimodal, multitask retrieval across languages. arXiv preprint 200.v:2109..0112 2021.
[37] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Hun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling visual and vision languaee representation learning with noisy text supervision. In International Conference Machine Learning, pages 4904 4916. PMLR, 2021.
12

---

[38] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi. Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[39] Jonathan Kahana and Yedid Hoshen. A contrastive objective for learning disentangled representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 579-595. Springer, 2022.
[40] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLLT, pages 4171-4186, 2019.
[41] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan, Supervised contrastive learning. Advances in neural information processing systems, 33: 11611 2020.
[42] Byoungjp Kim, Sungik Choi, Dasol Hwang, Moontac Lec, and Honglak Lec. Transferring pre-trained multimodal representations with cross-modal similarity matching. Advances in Neural Information Processing Systems, 35:30826-3088 2022.
[43] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research). URL httt::t/ttwww.... www.onttttoooooooooooooooooooo.. www.orotttouuoooo...oooo http:/kklz/claazz htt]
[44] Sangho Lee. Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parameter efficient multimodal transformers for video representation learning, arXiv preprint arXiv:2012.04124, 2020.
[45] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Multibench: Multiscale benchmarks for multimodal representation learning. NeurIPS Datasets and Benchmarks Track, 2021.
[46] Paul Pu I Liang, Chiyu Wu, Louis-Philippe Moreney, and Ruslan Salakhutdinev. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565-6576 PMLR, 2021.
[47] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Shengtong Mo, Dani Yogatama, et al. Highmmt: Towards modality and task generalization for high-modality representation learning. arXiv preprint arXiv:2203.0133 2022.
[48] Paul Pu Liang, Amir Zadch, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430. 2022.
[49] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nic, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis Philippe Morency, Quantifying & modeling feature interactions: An information decomposition framework. arXiv preprint
230.. 2023.
[50] Jiasen La, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: pretraining task-agnostic visiolinguistic representations for ssioo-and-languase tasks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 13-23, 2019.
[51] Martin Q Ma, Yao-Hung Hubert Tsai, Paul Pu Liang, Han Zhao, Kun Zhang, Ruslan Salakhutdinov, and Louis-PPilippe Morency. Conditional contrastive learning for improving fairness in self supervupeee learning. arXiv preprint arXiv:2106.0288 2021.
[52] Emily E Marsh and Marilyn Domas White. A taxonomy of relationships between images and text. Journal of documentation. 2003.
[53] William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
[54] Yu Meng, Chenyan Xiong, Payal Bajaj, Paul Bennett, Jiawei Han, Xia Song, et al. Coco-Im: Correcting and contrasting text sequences for language model pretraining. Advances in Neural Information Processing Systems, 34:23102-2311 14, 2021.
[55] Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Cemi: Classifier based conditional mutual information estimation. In Uncertainty in artificial intelligence, pages 1083-1093. PMLR, 2020.
[56] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak. Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint 2201 2022.
13

---

[57] XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions Information Theory, 56(11): 5847-5861, 2010.
[58] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0377 037448 2018.
[59] Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learninnn Advances in Neural Information Processing Systems, 32, 2019.
[60] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8863, PMLR. 2021.
[62] Anil Rahate, Rahee Walambe, Sheela Ramanna, and Ketan Kotecha. Multimodal co-learning: challenges, applications with datasets, recent advances and future directions. Information Fusion, 81:203- 2000002-2,,222 2022.
[63] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre- training for speech recognition. arXiv preprint 19455:190, 2019.
[64] Bin Shan, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Erice-vll 2.0: Multi-view contrastive learning for image-text pre-training, 2022.
[65] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379-223 423, 1948.
[66] Yuge Shi, Brooks Paige, Philip Torr, et al. Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in Neural Information Processing Systems, 32, 2019.
[67] Ravid Shwartz-Ziv and Yann LeCun. To compress not to compress-sell-superriied learning and information theory: review, arXiv preprint arX 2023.
[68] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. CORR, abs/1910.06222,, 2019. www.httpp//arsiw..rr////e
191009990..000. 06222
[69] Alessandro Sordoni, Nouha Dziri, Hannes Schulz, Geoff Gordon, Philip Bachman, and Remi Tachet Des Combes. Decomposed mutual information estimation for contrastive representation learning. In International Conference on Machine Learning, pages 9859-9869. PMI.R, 2021.
[70] Karthik Sridharan and Sham M Kakade, An information theoretic framework for multi-view view learning. In Conference on Learning Theory, 2008
[71] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. ECCVV 2020.
[72] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan. Cordelia Schmid, and Phillip Isola. What makes for good views contrastive learning? Advances in Neural Information Processing Systems, 33:6827 6839, 2020.
[73] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi- redundancy, and lincar models. In Algorithmic Learning Theory, pages 1179-1206. PMLR, 2021.
[74] Yao-Hung Hubert Tsai, Tianqin Li, Weixin Liu, Peiyuan Liao, Ruslan Salakhutdinov, and Louis Phill-p Morency. Learning weakly- supely-sueee contrastive representations. In International Conference on Learning Representations.
[75] Yao- Mho-Hun Hubert Tsai, Paul Pu Liang. Amir Zadch, Louis Shii--Phllippe Morency, and Ruslan Salakhutdinov. Learning factorized multimodal representations. ICLR, 2019.
[76] Yao-Hung Hubert Tsai, Yue Wu. Ruslan Salakhutdinov. and Lhiis-Philippe Morency. Self-supervised lcarning from a multi-view perspective. In International Conference on Learning Representations, 2020.
[77] Yao- muo-Hun Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Russ R Salakhutdinov. Neural methods for point-wise wise dependency estimation. Advances in Neural Information Processing Systems, 33:62-72. 2020.
14

---

[78] Yao-Hung Hubert Tsai, Tianqin Li, Martin Q Ma, Han Zhao, Kun Zhang, Louis-Philipee Morency, and Ruslan Salakhutdinov. Conditional contrastive learning with kemel. arXiv preprint arXiv:2202.05458. 2022.
[79] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutuaa infor- mation maximization for representation learning. International Conference on Learning Representations. 2019.
[80] Jorge Vergara and Pablo A Estévez. A review feature selection methods based on mutual information. Neural computing and applications, 24:175-188 2014.
[81] Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 16041-16050, 2022.
[82] Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXivv1004.2512 2010.
[83] Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. Advances in Neural Information Processing Systems, 31, 2018.
[84] Mike Wu, Chengxu Zhuang. Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual information in contrastive learning for visual representations. arXiv preprint arXiv:2005.13149, 2020.
[85] Jianwei Yang, Chunyuan Li, Pengchuan Zhang. Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in mnage-text-label space, 2022.
[86] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng. Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning, 2022.
[87] Zesheng Ye and Lina Yao. Contrastive conditional neural processes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9687-9696, 2022.
[88] Ron Yosef, Yonatan Bitton, and Dafna Shahaf. Irfl: Image recognition of figurative language. arXiv preprint arXiv:2303. 15445, 2023.
[89] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 6995-7004, 2021.
[90] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6032, 2019.
[91] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint vrXiv:1606.06255 2016.
[92] Amir Zadeh, Paul Liang, and Louis-Philippe Morency. Foundations of multimodal co-learning. Information Fusion, 64: 1893 2020.
[93] AmirAli Bagher Zadeh, Paul Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246. 2018
[94] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self sele-super learning via redundancy reduction. In International Conference on Machine Learning, pages 12310 123200. PMLR, 2021.
15

---

Appendix Broader Impact
A
Multimodal data and self-supervised models are ubiquitous in a range of real-world applications. This paper is our attempt broadening the applicability of self-supervised contrastive methods to wider range of multimodal tasks beyond those that exhibit multi-view redundancy. We believe that special care must be taken to ensure that these models are safely deployed for real-world benefit:
Time and space complexity: Modern multimodal models are large and take up a significant amount of carbon footprint during training and testing. As compared to heuristic combinations of modal and single-modality CL. [33] 36 44 64 81
85 89,, we believe that FACTORCI. does not significantly increase complexity: (1) upper bounds on MI can be estimated "for free" by directly plugging in the optimal critic from JNCE. and (2) removal of task-irrelevant information via 7 T(X;; X2 Xx) shares encoders with INCE, and (3) separate unimodal augmentations perform well enough in practice. We also release our code and models so that they can be evaluated quickly on new tasks, which can amortize complexity costs.
Privacy and security: There may be privacy risks associated with making predictions from multi- modal data of recorded human behaviors and medical data (i.e., the datasets used in our experiments for analysis of sentiment, emotions, personality, sarcasm, and humor, as well as disease prediction from medical data). We have followed best practices in maintaining the privacy and safety of these datasets: (1) the creators of these video datasets have taken the appropriate steps to only access public data that participants or content creators have consented for public release (creative commons license and following fair use guidelines of YouTube) [12] 27 93), (2) MIMIC has been rigorously de-identified in accordance with Health Insurance Portability and Accountability Act (HIPAA) such that all possible personal information has been removed from the dataset (88. (3) all video data was also anonymized and stripped of all personal (e.g.. personally identifiable information) and protected attributes (e.g., race, gender), (4) all models trained on affect recognition datasets use only pre-extracted non-in -nveerirtee features that rely on general visual or audio features such as the presence of a smile or magnitude of voice which cannot be used to identify the speaker [9] |3 and (5) we studied the videos collected in these affective computing datasets and found no offensive words used or personal attacks recorded in the video. Finally, we only use these datasets for research purposes and emphasize that any multimodal models trained to perform prediction should only be used for scientific study and should not in any way be used for real-world harm.
We acknowledge risks of social bias due to imbalanced datasets, resulting in potential biases surrounding gender, race, and ethnicity, among others 9 46). We note that our FACTORCL approach has a close link with conditional CL (5I). which can also be adapted to condition on sensitive attributes and therefore reduce bias. Studying these research questions is an important direction for future work.
Social blases:
We discuss more limitations and potential future work in this direction. Firstly, optimizing our objectives using better MI lower and upper bounds such in Guo et al. (26) and Sordoni et al. 169], could improve the performance for inputs of higher dimension and complex modality. Next, the current data augmentation method requires one pick augmentations to approxi- mately satisfy Definition 4 future work could extend InfoMin 772 to automatically generate data augmentations satisfy Definition 4 leverage future progress in multimodal generative models for data augmentation. Lastly, future work could quantify whether shared or unique information is more important for different tasks and reweight the terms the FACTORCL objective suit the tasks.
Future work:
B Analysis of Multi-view Contrastive Learning
Multi-view shared information describes the extent and dimensions in which information can be shared across different views. The presence of shared information is often in contrast to unique information that exists solely in a single modality, and can be formalized via information theory:
Definition 5. (Shared information) Given X, and X2, X(x;;X2) ff ((#1,1))) log p(21,22)
p(r))p(())
measures the degree of information- theoreeii shared information between X, and X2.
Definition 6. (Task-relevans shared information) Given X1, X2. and a target Y, 1(X1;x;; p(2).22) T(X1; X2) - T(X1; X2(Y) = p(21,22) R2) log p(1))p(22) -f p(21, log p(2).221)) measures
p(alw)p(azM))
16

---

the amount of task-relevant shared information between X1 and X2 for predicting Y. X2))) represents the irsk-irrelevant shared information.
Learning shared information via contrastive learning: Current approaches for multi view con- trastive learning model shared information I [(x1;X)) (and subsequently task-relevant shared infor- mation 1(X1; X2; Y) during downstream task fine-tuning), without modeling unique information.
Z1 arg max X2);; Z2 arg max 1(X1;
(19)
fo==))()))
2z=ff(Xa))
Optimizing for ((X1;XX) is performed via a surrogate loss during self-supervised pre-training (where we do not have access to the label Y) by maximizing the InfoNCE objective:
exp (
INFONCE ssup E, (x1.2-p((e1.0))) log
(20)
Σn exp f5)1
<g-p(m2)
Oord et al. [58] show that 1 Xx) log k ONCE(X2 X2) where (xc is negative of INFONCE and is the loss to minimize (rather than maximize) in training. NCE falls into a broader class of contrastive learning methods [13] 15| 28 41 61] that model the ratio between joint densities p(x) 22) and product of marginal densities p (2) using positive and negative
samples [57 59 60
79
84
or probabilistic classifiers [55 77). all of which can also be used to
capture shared information.
Tian et al. 771] argues that the optimal view of contrastive learning is also minimal: the minimal representations only extract relevant information of the contrastive task (maximizing the shared part) and throw away other information. Therefore, from this minimal assumption, we have [Zi Y (X2) and / (Z2; x 0 as minimal Zn and Zz only captures task-relevant information from the shared part. By conditioning X₃ or X2, the shared part is removed, and Z1 and Y (or Zz and Y) do not share information.
Lastly, we restate the multi-view non-redundancy from Definition 2 Definition 7. (Multi-view non-redundacyy) 3c 0 such that I [(xx; cor cor
IX2; c.
We would like to compare and clarify the differences between the multiview redundancy assumption in (q.(T) and the multi-view nonredundancy in Def. 7 The multiview redundancy assumption in [ states that the task-relevant information from the unique part is minimal (s <). The multiview non-redundancy states the opposite: the task-relevant information from the unique part is nonzero and nonminimal, as it is not bounded by e. Next we briefly clarify the relationship between these two assumptions and the InfoMin assumption: I [(21; 2 = / [(2z; Y|X, = O. InfoMin is about representation Z while the redundancy assumptions are only about data X. InfoMin states that the optimal (sufficient and minimal) representation learns task-relevant information only from the shared part, as we discussed in the paragraph above. We empirically checked the two assumptions: Tables []] and 4 in the main text show that the multiview non-red con-reuuudaccy assumption holds empirically. and Table shows that the InfoMin assumption holds empirically.
We now show the limitations of CL methods, first restating the Theorem here:
Theorem 4. (Suboptimalitty of standard CL) When there is multi-view non-redundancy as in Definition
given optimal representations {Z1, Z2) that satisfy 19 and I ((11; = 1(2;; X()))
O 1721, we have that
I(Z, Z2;Y) = I(X1, X2: I(X.: -at.:-xxx)-(x111111111 T(X2;YIX)) II::: I(X:;
(21)
Proof. Since Z1 and Z3 maximize (() we have that 1 (2;;;2;;)) = I(xi; Z2 [(x;;; Y) and 1(2;;
so
We now show the relationship between I(X,, Xy; Y), which is the total information X1, con- tributes towards predicting Y in classic supervised learning, with I(Z,, Zz: Y), which is the infor- mation that our learned self-supervised representations can contribute towards Y during supervised fine-tuning. We first derive the relationship between I( (Z,; Y) and Y(X;;Y):
I(Z, = I(1 I (Z,;
(22)
= =x; T(<;
(23)
+I(Z2;1122)
(24)
(25)
17

---

Given X1, we further derive relationship betwenn ((2; and ((x;;
1(22; - 11
(26)
(27)
(28)
(29)
(30)
(10,:))0((22xxx3))00333333>>11-1 (2)
(31)
(32)
(33)
In Eqs. 28 and (30p, we the fact that conditioning on Z1 and Xi jointly reduces conditioning on X1 since Zí is deterministically obtained from X,, and in Eq. we use the definition of learning
Zg to maximize 1(x1;X2) so = Y). Finally, adding both terms up
1(2, 22; +(2;)))
(34)
-[xx;>))[(x;;113)
(35)
=1(X1;X2;))
(36)
(37)
(38)
gives the desired result.
Bayes error rate. The Bayes error rate Pe(Z1,Z,) = E22, max P P(Ý = z1, of
contrastive representations {Z1, Z2) is given by:
- - - -
Pe $ 1 exp - exp [XX; H(Y)]
(39)
(40)
Proof. We use the inequality between Pe and HY
22 76 81
(1 - PP:) s HY)))), or equivalently, Pa s exp(-H((
(41)
we regard Z the joint Z₁ and Zz, then we have
P. s 1- ex1-- 22)]
(42)
We further expand H(Y]Z,, Zz) by definition of mutual information, I(X;Y) H(X) Theorem 4 and the
H(Y = (I(Y))((2,44)))
(43)
=H(Y) ((X,, + ((X;; ([x;
(44)
=H(Y) - -(X1;Xx2) (T0e)++[([;;))) (2
(45)
=H(Y) 1(X1;2;;))
(46)
Plugging in Eq. 42.44).
42).
have
P. s 1 exp[ -xp((H(((2(
(47)
= 1 cxp
- -(x
(48)
- - exp(-H(())) - -(Xx:
(49)
and
P. 1=S1-exp|-H(\|Z1.0
(50)
= e1-eep[-(HHte) (Ix;
(51)
= 1i-eepp--Ai -Xxxx)(((+(((xx))
(52)
resulting in the Bayes enror rate as desired.
18

---

C FACTORIZED CONTRASTIVE LEARNING
C.1 Contrastive estimators
Theorem S. (Contrastive estimators X()1 Defining the NCE estimator and NCE-CLUB
estimator as follows,
exp f(11
(x;E(X1;X2) -=n,s)-p(e1,.e)) 22-0(22)
log Σx exp f (211
(53)
(xCE-LLUBXx;;X)) =1 [f 2;), Barp((s) [f (x1,
(54)
spmp(z2)
where I (2) is the optimal critic from INCE plugged into the TCLUB objective [T6.. We call the proposed plug-in objective Eq. INCE-CLUB, and obtain lower and upper bounds on 1(X1; X2):
INCE(X1;:)) $ {1(1;:x2) & ((1E-CLUB((1::X)))
(55)
Proof. The lower bound INCE (X1; X2) / [(x;; X2) follows from Oord al. 588: optimizing the objective leads to an optimal critic f = log p (2) c(x2) (60] or without loss of generality
I" log (122 c(x1), where c(.) is arbitrary deterministic function. Plugging the optimal
critic into the INCE INCE(XI;XX X2) gives the result: (xcE(X1;Xx)) + log N & X2) 58 60).
Next, the original ICLUB X2) [6 is defined as:
(cLuB(X1; X2) == Bp(m),2)) [logp(>y]>1) -E-EEEEE (p(z))((z>) [logp(12|<1]]
(56)
As mutual information is symmetric w.r.t and 22: 1(X1; = 1(X2; X1) without loss of generality, we have:
TCLUB X2) IcLun (X2; X1) (p))) [log((m)(22) Ep(e))((zz) [log (57)
The formulation above assumes p(21 1 (x22 is known, which satisfied when we use the optimal critic I" log (22) c(x1) from INCE X1; X2). Plugging the optimal critic f" into (XLun(X;;;))) X2).
we obtain desired upper bound /NCE CLUB (X1; X2) of X(X1;X2):
INCD.CLLUD (X1 X2) (x))) (I (22) c(x1)] (p(c))(( (log((<1(22) + ((x1) (58)
=Ep (2))))0)) (ogp(=)|/>)) E Bx( [c(=:)] - (x(r))(())) [log((r][22] (p(x))((2)) [c( (c(x1)) )
(59)
[p) logg(<r)zz) E Ex( [logp(z1|<)] ((x);X2).
(60)
(61)
55) is from the linearity of expectation, Eq. 60
from the fact that c(x1) not a function of x2
and therefore Ep(zy [c(x1)] = Ep(ar )p(2g) [c(x1]] = (p(r,) [c(x1)], and Eq.66l) is directly from
the original ICLUB( (X1; X2) paper [16].
C.2 Unimodal and multimodal augmentations
We first restate the definitions of optimal single-view and multi-view augmentation:
Definition 8. (Optimal unimodal augmentation) X1 is an optimal unimodal augmentation for Xi when I'X;X')= ((X;Y), which implies that the only information shared between X and X is tusk-relevant with no irrelevant noise.
Definition 9. (Optimal mudtimodal augmentation) X and X are optimal multimodal augmentation for X1 and X2 when [(X,, X2; X1 X²) = I 1(X1,X2; Y). which implies that the only information shared between X1, X1,Xa X2 and X1, X12 is task-relevant with no irrelevant noise.
When are these assumptions satisfied? T(X;'') tween X and X' is task-relevant, which implies that the augmentation keeps task-relevant information constant while changing task-irrelevant information. In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background. To satisfy 1(X1, X2; X X2) I(X1, X2; Y), by the chain rule of MI, we augment in two steps:
Y): holds when all information shared be-
Unimodal aug: X s.t. I(X;;Xf) I(X1;Y), Unique aug: X2 s.t. 1(X;; -
(62)
(63)
19

---

Table 5: More examples of optimal single-view and multi-view augmentations.
Standard Aug
Unique Aug
Dataset
X
X2
X2
Cartoon
Caption
Image
Word Masking
Crop Flip Resize
Flip Resize
MIMIC
Signals
Tahles
Time Warping
CutMix [90] on All Features
CutMix on Nonelinical Features
MOSEI
Transcripts
Video+Audio
Word Masking
Noise Injection on Any Frames Noise Injection Silent Frames
UR-FUNNY Transcripts
Video+Audio Word Masking
Noise Injection on Any Frames Noise Injection on Silent Frames
MUSTARD
Transcripts
Vidco+Audio
Word Masking
Noisc Injection on Any Frames Noise Injection Silent Frames
the second step is the unique augmentation: after observing X1, we create augmented X2 from X2 to keep the task-relevant information but meanwhile do not affect any information from X In Table
5 include some more examples of how unique augmentations could be designed across different datasets.
Final objectives: If Definitions 8 and 9 are both satisfied, substitute contrastive estimators in the following equations:
exp f(11, 22, v) log
(X1;(\X))2)))) Ep(n)
E eo 2 p(22l)))
(64)
Σ. exp 27.
(Nce:cLus((:::e21) Ep(n))
E at (21, v)
Bry ( [f (x1,
(65)
by replacing I(X;Y) terms with I(X;; and replacing I(X1:Xy|[X) X2 X;):
terms
with
log exp f22
azylh (ox,cuuu(XX;;,,)) X²) Extay Po
Xx X²) Entg
(66)
Σn exp f (011 2/,
(2
Tr. ae a4) [f' (2, ag-p(xalaf.aa))
(67)
C.2.1 Implementing conditional CL via kernel
We restate objectives below:
exp fx1, 222
INCE Xx;; Ep(a)
E E apaa anaa peaalaa
log
(68)
Σn exp [(a t0,,az, 2i
Eptry
E P ag) [f (1), (x1, 22, a1,
(69)
However, sampling from ((.)) ) is hard. Since X X²2 are continuous variables, directly sam- pling from the conditional distributions p(-1=1,+44) may be difficult; training a generative model from augmented data 21, to original data X2, can be expensive and nontrivial Pe multimodal setup. In this work, we implement the conditioning in p(x1 (ur,,,//< xg|=11 (41,2) xx1) through con- catenation and the details are in Appendix D.I] Here we discuss alternative solution to this problem introduced by Tsai et al. (78). leverages kernel methods for conditional sampling in contrastive learning by assigning weights to each sampled data given the kernel similarity between conditioned variables, avoiding directly sampling from the conditional distributions or training generative models. In our formulation, given multimodal input (x1,02) with their augmentation (1/,2/) 2/).. we can sim- ply use the technique from (78 to estimate INCE( X1; X2 X;) and INCE.CLUR CLUB X1; X2|X X²), where the kernel measures the similarity between different pairs (2f, x²) of the conditional variable
20

---

X2, Specifically,
exps(r1,
X2/ x²) B Bpta log exp 32) +** [K Xx,,
(70)
INCE.CLUB INCE.CLum(Xx; (X X2)) Ep(or.a2.ff.ng (2), log K Xx11X11X3x;;1a
(71)
where 1x,,,,,, Xx,x, + X1) and [Kx, lu is the ith row and
ith column of 1x11X Kx,x, is a kerne similarity matrix between X1 and X2, and Kx; x
is separate kernel similarity matrix between X2 and X3. f is the optimal solution of Eq 70) By leveraging the similarity Kx;x; x3 between conditional variables X and X2, X -x; transforms the similarity scores between X1 and X2 under unconditional distributions into similarity scores under conditional distributions. Note that the expectations in Eqs.(7)) and (7T) are taken over the joint distribution p(11, 22, x²,x,), which comes naturally after augmenting both modalities X1 and X2. This method could effectively alleviate the problem of sampling from conditional distributions our formulation. We refer the reader Tsai et al. 78 for more details.
C.3 Final estimators in FACTORCL.
Theorem 6. (Contrastive estimators for shared and unique information). Under assumptions on single-view augmentations I(X;; I(X1, X1) (Definition 8 and optimal multi-view augmen-
tation X2 such that [X;,,x2;X; ",,X x²) = (1; (Definition
9..
we can define contrastive
objectives for task-relevant shared and unique information with:
S = == INCE (NCE(X1; INCE-CLUB (X1;X2]XX;
(72)
U: 1(X; 2 INCE INCE(X;;: X!) JNCE.CLUB (1;; X2) INCE X1: X2|X1,
(73)
Proof. The objectives follow from the fact that [NC X2) and INCE X2|x X²))
lower bounds of I(X1; X2) and I(X;; respectively, and INCE- SICE-CLUB(C (X1; X2) and INCE-CLUB(X, X3) are upper bounds of ((x and ([Xx;X21>)) respectively:
S (K1; Y) = ((1;: ((X;;
(74)
2 JNCE (NCE(X;; X2)
Xx
(75)
U, I(X;; vIX-s) I(X;Y) (
(76)
INCE (x); (UNCE-CLLB(X:: - -Nx X²))
(77)
and symmetrically for U2.
Now show that FACTORCL learns both shared and unique task-relevant information. First, we restate the definition of the factorized representations:
Zs, arg max I(Z;; X2;
Zs. arg max 1(Zz;X1;Y) 7 (X2)
(78)
Z1 (x{)((
Zu, arg I(Z1; Y\X2),
Zu, arg max I(Z;; Y|X)). Zo=fo(Xa)
(79)
Zi=1o(X1)
where ([Z; Y) X2) X(11) is the shared information and T(Z2; X1;Y)
T(Z;; X₂) - Y is the unique information.
Theorem 7. (Optimality of FACTORCL If Zs, and the estimations in Eqs. 13 67) are tight, we obtain
Zu, perfectly maximize Eqs.778 79
I(Zs, Zszi Y)
IZZ,; (|Z,;) + + Z| suggesting that FACTORCL learns both shared and unique task- relevant information.
Proof. Because show that 1(X1;X2; = (X) ((Zu; and Y|X,) I [[Zu, l
T(X1; X(Xx; Y(X,; is sufficient to
21

---

First we show x(x;;:x;;)) : Crucially, by definition of how Zs, and Zs, are
optimized maximize (xx1;X2;;) Y), have that:
T(X1;X2;)) :(2s;;:X;\1) x
(80)
We can then obtain
(2;1;X2;10 = I(Xi; Zs;
(81)
(82)
= ((Zs, [(Zs,; Yiss, X1)+ T(Zs;; 7s;; X1;
(83)
= ((Zss;YlZss |Zss))((Zss;) + ;
=I(Zs;; - +
(84)
(85)
-[(ss; Y[Zs,) [ss,
(86)
T(Zs;; ZssyY)
(87)
where Eq.(84) is because Zs, are deterministically obtained from S and Eq. is because Zs.
maximizes the shared information. Finally, go to (8. 1 1(Zs,i Z| 0 shown in Eqs. 33 using the fact that Zs, is learned to maximize ((X1; Y) and I (Zs, X2;
I(Xi: Y).
Next, we show
I Y(Zs,))(Zs,) [(Zu, Y/ Zu.) I(Zu, Zu, [s,,) (z,
(88)
which is by the chain rule of conditional mutual information. Then we show T(Zu;; YY:
I(Zv; ; Y; Zu, Zs.) I(Zy; (r, Zo, 2s,) = 00
(89)
This is because Eq. (7 leads to [(, V|X2) (X1; and ;( x| = ((2;; Y
If the estimations in Eqs. 11 13 [Zo, :
67 are tight, by conditioning and by the previously stated Zu; tightly captures information from only X1 and not in X2.
The same applies to Zu.- We have [(Zv, x22) I(Zu,i [Zy, Zv.) =ZZ, Zv,lT)=
with T' being an arbitrary random variable because no shared information exists between Zu, and Zuz Then we obtain:
I
Zu.) YY =[(Zu; :(
+ I(o,; Zu.)
(90)
(91)
We use the fact that conditioning on Zsa, Zue and X₂ jointly reduces to conditioning on X2 since 7s5 and Zuz are deterministically obtained from X2. Lastly, since Eqs. (7.79) are satisfied, Zu, arg max (x1) 1( Y|X2) therefore T(Zu; x(X2) = T(X1; 22X2). We have:
I(Ze, :Y = (Zz,; :Y|2) = [(2;; ((x2).
(92)
The proof for ((X;; = (Zo, is similar. We now have shown that I(X;: X2; Y)= )
T(Zg; Zs,;Y) I(X1; I(X;; 1(2,;; Y|,s,) and T(X2;Y|X,) = ;(Zu,;Y Y| ) adding up all LHS and RHS have the theorem.
C.4 Extensions to masking and non-contrastive learning
We now show how similar ideas can extended to other popular self-supervised learning objectives, such as non-contrastive learning 6 94] and masked pre-training (20 29). Importantly, this paper pro- vides a new principle for multimodel self-supervieed learning: (1) learning task-relevant information and (2) removing task-irrelevant information from both shared and unique parts across modalities. Our paper focuses on realizing this principle via multi-view information theory and contrastive learning. Below provide two potential alternatives to realize this principle on non-contrastive and masking methods, respectively:
Non-contrastive learning: Methods such as Barlow Twins 94 and VICReg 6 use invariance and covariance regularizations to maximally preserve shared information the embeddings across two modalities. However, the embeddings learned may contain only contain task-relevant information from the shared part and not unique parts. To use the principle in this paper to capture more task- relevant information from unique parts, one should perform VIC-regularization on X₁ features,
22

---

X2 features, and on X2 cross- moosl features. When performing VICReg on unimodal features, one should condition on the other modality when performing augmentation. Specifically, similar to the idea of multimodal augmentation in this paper, the augmentation of the second modality should not interfere with the shared part (i.e., do not augment regions referred by the first modality), making the invariance and covariance regularization of the second modality focus on the augmented modality-unique features. This makes the model learn unique modality features that are not captured by the joint embedding from standard independent augmentations.
Conceptually, masking 20 29 can be interpreted as leveraging unmasked regions in the same modality to predict masked regions or leveraging the other modality to predict the masked region. However, the learned representation may not be all task-relevant. To use the principle in this paper to exclude iisk-inrelevant information and capture more task-relevant information from unique parts, we can perform conditional masking, where masking is conditioned on augmented views (similar to the multimodal augmentation in the paper, where the conditioned views are approximating the labels). As a result, only unique regions in the second modality can be masked out, making the model capture more unique information from the second modality by masked prediction. Here have only provided high-level intuitions of extensions to these methods, and future work should explore these ideas in more detail.
Masking:
D Experimental Details
D.1 Implementation details
Objective Formulation and Architecture
In Algorithm in the main text, see the sketch for doing contrastive learning with our proposed objectives. To implement all algorithms used in our ablation experiments, we start with two encoders (.) and c2)-), which takes samples 21 and 22 from the modalities X2 and X2, and outputs corresponding representations 21 and 22. We also have a critic function fo(--) parametrized by which takes 21 and 22 as inputs and returns scalar. A popular way to perform contrastive learning aims to maximize INCE( (1; X2), where
log exp (xp
INCE (xc;(X1;)2) = E 20(22)0()1
(93)
fo(e(())) (x))
=5-p(1)))
In our algorithms, we follow the derivations in Eqs. to maximize each INCE objective and minimize each INCE-CLUB objective. Therefore, for each objective, we add two additional MLP heads on top of the two encoders and create separate critic which takes in the outputs of the MLP heads instead of the encoders. In all the experiments, we adopt the concat critic design [58 60 68 where fo = ho [[) with he being an MLP.
FACTORCL-SUP: In the supervised version of CL which uses label Y, the objective we aim to maximize is formulated as
LEACTORCL-SUP- INCE (x); INCE-CLUB (X;;
(94)
INCI [NNC((X;; (X1; Y)+ + TNCE(X2;
(95)
L/NCR.CLUn(X, X2) + INCE XX;;
(96)
Each INCE and INCE-CLUB term in this objective calculated using its own critic as discussed above. The conditional terms involving the label Y are implicitly modeled by directly concatenating Y to the outputs of both heads before feeding into the critie. To obtain the learned representations 7s, we concatenate the outputs of the heads on top of the encoder that correspond to the terms INCE (X;; and INCE-CLUB (X1; X2lY". To obtain Zy, we concatenate e1 's head outputs from the terms INCE YY;YY), INCE-CLUB (Xi: X2), and INCE (x1:(X))))))) Zs, and Zu, are obtained in similar fashion, except we use the outputs from e2 's heads instead E1.
CACTORCL-SSL: In the self supervised version of CL which uses augmentations X1 and X²2 of the input modalities, the objective we aim to maximize is formulated as
LEXCTORCL-SSL INCE (X); INCE-CLUB (x;: X;) INCE X1: +INCE(XI; Xf) INCE(X2;;X)
(97)
(98)
- -INCE-CLUB (xti)uuu(x;; (1; X2) + INCE (x,: X!).
(99)
23

---

Concat
Z
Zi
Z2
Z
f
X
Xi
X₂
X2
Figure S: An illustration of conditioning by concatenation the implementation of FACTORCL. Conditioning done by concatenating Z1, the encoded representation X1, and Zi, the encoded representation of X similar operation is performed for X2 and X2. The concatenated vectors are then fed to MI cstimators, such as INCE and IXCE-CLUR (the figure illustrates INCE).
Here the conditional terms are conditioned on the augmentations X1 and X2. and we can similarly model by concatenating the head outputs of X to X, and the head outputs of X²2 to X2 before feeding into the critic. We use Figure Bo illustrate this. The way to obtain the learned representations the same as described in FACTORCL-SUR.
Estimation of CMI: To estimate the conditional mutual information (CMI) X we estimate the lower upper bounds of true CMI [1 55 69]. However, direct sampling from the conditional distribution p(21 222z1, (01,,0 x2)) can be expensive because we should consider a different conditional distribution p(21, 222a4 2:22) for each data pair 2/:42 Sordoni et al. [69] address this by concatenating the conditioning variable with the input in the critic: 6(I1, ((,,,,2,,0 c2, and showing that
Conditional InfoNCE (Eq. (5) is lower bound and estimator of CMI. This estimator can be made more exact by further importance sampling [69] However, adding importance sampling [69] or using more accurate estimators (66] comes with trade-off in complexity. Since we focus capturing unique information to learn a scalable multimodal representation instead of accurately estimating the CMI, we leveraged a simpler version of the estimator from Sordoni et al. [69]: generating multiple augmented pairs from I1 22, and concatenating the input T1, 22 22,22 and each augmented pair 21 x²2 to define samples from the conditional distribution p(11, Z2 xyyaa/ xz2). We argue that since augmentations do not significantly change the semantics of images, p(z; 2yyaa,, xz2) could be approximated by P(=", x²" 1 x2/) where 24 2" are other augmented pairs in addition to x'j 2zg. In this submission, we use one pair of augmented samples for consistency, but our code easily supports increasing the number of augmented pairs that can improve the accuracy of CMI estimation.
Regardless, our existing one-pair implementation can already show that our estimators are empirically comparable to CMI estimators with guarantees such Mukherjee al. (55) (Table ()), and our estimators empirically satisfy that the lower bound is smaller than the true CMI, and the true CMI smaller than the upper bound, i.e., Conditional InfoNCE true CMI Conditional InfoNCE-CLUB (also in Table 9). We refer the reader to Sordoni et al. 69 for tighter bounds for CMI.
For this ablation, remove the factorization and only learn Z1 for X₃ and Zz for X2- The objective we use is the same as that of FACTORCL-SUP. The only difference is that we now take ei (x1) and (2(22) as the learned representations for inputs x1 and z2-
CL-SL-UUP:
similar ablation for FACTORCL--SS where we remove the factorization. The objective is the same that of FACTORCL-SSI and we use e (21) and (2(22) as the learned representations for inputs Ij and Z2.
OurCI-SSL: This
Training Strategy: In regular contrastive learning using INCE as the only objective, we can simply perform gradient descent minimize INCE, updating all parameters in the encoders, MLP heads, and critics. However, training any of the four methods above also involves the minimization of the INCE-CLUB objectives, which require the optimal critic f* from INCE, as stated in Eq [I]. Therefore, within each iteration during our training, we need to first obtain the optimal critics for the INCE. CLUB terms using the INCE objective. We outline the training strategy using a sampling method in Algorithm 3 In this algorithm, LFACTORCI can be either LFACTORCI- PUP or LFACTORCI-SSIL and CNCE is the summation of INCE objectives for the INCE-CLUB terms. In particular, we have
CNCE = |NNCR(X, X1; X2/Y) + INCE (X1;
if L LFACTOCIL-SUP
(100)
Xxlx INCE (x1;
if L LFACTORLL-SSL.
24

---

Algorithm 3 CL training with sampling
Require: Multimodal dataset (X1, X2).
0, Ø + Initialize network parameters.
while not converged do
[11 222) + Sampled batch from (X1, X2)
- Update parameters by maximizing LFACTORCL
for 1 to do
{2f, 2/2 Sampled batch from {X1, X2)
Φ <- Update parameters by maximizing LNCE
end for end while return 0, Φ
We define & to be the parameters of critics for the INCE-CLUB terms, and 0 corresponds to all the rest parameters in the network (parameters of encoders, heads, and critics for INCE terms). In the outer loop, we update 9 using the main learning objective L. In the inner loop, we update 6 using the LxcE objective, which learns the optimal critics f needed compute the INCE-CLUB terms. Ideally in the inner loop we would update á until convergence so we get a good approximation to the optimal critic. In practice we found sampling just one batch by setting = in Algorithm 3 works pretty well. Using only one iteration does not have big impact on the convergence and still produces promising results. More importantly, significantly reduces the time required for training, and allows our algorithms to have comparable running time to existing contrastive learning methods.
D.2 Datasets
Gaussian datasets for MI estimation: As shown in Figure 3 in the main text, we first demonstrate a toy Gaussian dataset. We obtain the samples {xx, y:) from 4 multivariate Gaussian distribution with dimensions (20, 50, 100, 200). compute the correlation needed for achieving these MI values using the ground truth MI formula for Multivariate
the quality of our proposed upper bounds INCF-CI.UB X2)
In each dataset, we set the ground truth MI values to be (2, 4, 6, 8, 10), and
Gaussian: I(X,Y) 2 log(1 1 - p²). At each true MI value we sample 4000 times using a batch size of 64.
Synthetic dataset with controlled generation: We generate data with controllable ratios of task- relevant shared and unique information to analyze the behavior of each contrasive learning objective in Figure 1 in the main text. Starting with a set of latent vectors w1, W2, Ws, ~ NO,,,, E2), d 50 representing information unique to X1, X2 and common to both respectively, the concatenated vector [w1, w.] is transformed into high-dimensional I1 using a fixed full-rank transformation Ti and likewise [W2, w.] to 22 via T2. The label is generated as function (with nonlinearity and noise) of varying ratios of Ws, W1. and W2 to represent shared and unique task-relevant information. For experiments, used 1-layer MLPs with 512 hidden size encoders, and the embedding dimensions are 128 for both modalities. The heads on top of encoders are also 1-layer MLPs with the same hidden and output dimension as the input, and all crities are 1-layer MLPs with 512 hidden size.
Multimodal fusion datasets: We use a collection of 5 real-world datasets provided in Multi- Bench 45 and the IRFL. dataset to test our method in the context of varying ratios of shared and unique information that is important for the task. In all the datasets below, the heads added on top of the encoders are I-Layer MLPs with ReLU activations that map the encoder outputs to the same dimensions. All critics are also MLPs with hidden layer of size 512 and ReLU activation.
1. MIMIC-III 38 (Medical Information Mart for Intensive Care III) is large-scale dataset for healthcare which contains records of over 40,000 ICU patients in both forms of times-series data measured by hours and static data (age, gender, ethnicity) in the tabular numerical form. We use the preprocessed data provided MultiBench [45), where the time-series data is measured every hour in a 24rhour period and consists of vectors of size 12, and the tabular data consists of vectors of size 5. The task we use in the experiment a binary classification on whether the patient fits any ICD-9 code in group 7 (460-519).
25

---

Table Results on MultiBench 45] datasets with varying shared and unique information: FACTORCI. achieves strong results vs self-supervised (top rows) and supervised (bottom rows) baselines that do not have unique representations, factorization, upper-hounds to remove irrelevant information, and multimodal augmentations.
Model
MIMIC
MOSEI
MOSI
UR-FUNNY
MUSTARD
SimCLR
13
66.7 0.0%
71.9 0.3%
47.8 + 1.8%
50.1 + 1.9 %
53.5 = 2.9%
Cross+Self [8]
65.2 + 0.0%
71.1 + 0.2%
48.6 + 1.2%
56.5 = 0.7%
53.9 = 4.5%
SeltFaat [89
65.5 0.0%
71.9 0.2%
49.0 = 1.1%
59.9 = 0.9%
53,9 = 4.0%
OurCL-SSL
65.2 + 0.0%
71.2 # 0.2%
49,0 = 0.8%
58,8 = 1.3%
54.0 = 2.5%
FACTORCL-SSL
67.3 0.0%
74.5 0.1 %
51.2 1.6%
60.5 0.8%
55.8 0.9%
SupCon [4]
67.4 0.0%
71.0 0.1%
47.2 # 1.2%
50.1 = 2.0%
52.7 2.2%
OUrCL-SUP
68.2 a: 0.0%
71.1 0.2%
65.3 1 0.8%
58.3 = 1.1%
65.1 = 1.8%
SACTORCL-SUP
76,8 0.0%
77.8 0.3%
69.1 0.6%
63.5 0.8%
69.9 1.9%
Table 7: Continued pre-training CLIP with our FACTORCL objectives classifying images and figurative language. Our approach shows strong results compared standard fine-tuning and continued pre-training.
Task
IRFI.
Zero-shot CLIP
61
89.2 4 0.0%
SimCLR [3
91.6 1 0.0%
Cross+Self 81
89
91.1 # 1.2%
FACTORCL-IndAN
91.6 + 1.3%
FACTORCL-SSL
93.8 + 1.4%
Fine-tuned CL.IP 61
96.4 0.0%
SupCon 41
87.7 = 4.7%
FACTORCL-
98.3 + 1.2%
In the experiments, we 2-layer MLP with 10 hidden layer size for the tabular data modality, and map it to a vector of size 10. The time-series modality is encoded using a GRU with hidden size 30 and followed by linear layer which projects the output embeddings of size 15. We train the model for 100 epochs using the Adam optimizer with a le-4 learning rate.
the largest sentence-level multimodal sentiment and emotion benchmark with 23, 000 monologue videos. It contains more than 65 hours of annotated video from more than 1,000 speakers and 250 topics. Each video is labeled with sentiment intensity ranging from -3 to 3. In our experiments, we cast the intensity values to a binary classification on whether the sentiment is positive or negative. MultiBench (45) provides access to the extracted features of the vision, text, and audio modalities, and in our experiments, we use the vision and text features for doing contrastive learning.
CMU-MOSEI 9
In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a le-4 learning rate.
CMU-MOSI [9]] is a similar dataset for multimodal sentiment analysis created from 2, 199 YouTube videos clips. The data focuses on videos that reflect the real-world distribution of speakers expressing their opinions in the form of monologues. The sentiment intensities are labeled continuously from -3 to 3. Again we cast the label into a binary classification on whether the sentiment is positive or negative, and we used the extracted vision and text features for contrastive learning.
In our experiments we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a le-4 learning rate.
4. UR-FUNNY 27 is the first large-scale dataset for humor detection in human speech. The dataset consists of samples from more than 16, 000 TED talk videos with speakers from diverse backgrounds sharing their ideas. The laughter markup is used to filter out 8,257 humorous punchlines from the transcripts. The context is extracted from the prior sentences to the punchline. Using a similar approach, 8,257 negative samples are chosen at random intervals where the last sentence is not immediately followed by a laughter marker. The task is to classify whether there humor or not using the vision and text modalities.
26

---

Table 8: Additional experiments on CIFARIO
4
and MNIST
19
datasets using our FACTORCI, objectives
image classification.
Task
CIFAR 10 MNIST
SimCLR
13
87.0%
98.84%
SupCon 41
92.7%
99.38%
FACTORCL-SUP
91.3%
99.21%
Table 9: We verify our conditional lower and upper bound estimators on synthetic dataset with fixed dimension of representation d, and varying number of samples
Number of samples (x (x10"), ) d, 20
5
10
20
50
CCMI (MI- Miff-iff+ + Classifier)
2.03
2.06
2.15
2.20
Conditional InfoNCE
2.19
2.20
2.20
2.20
Conditional InfoNCE-CLUB
3.45
3.53
2.98
2.86
True CMI
2.32
2.32
2.32
2.32
In our experiments, we encode both the vision and text modalities using Transformer encoders with heads and layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with le-4 learning rate.
MUSTARD [2 corpus of 690 videos for research in sarcasm detection from popular TV shows including Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholies Anony- mous. It contains audiovisual utterances together with the textual context. We use the preprocessed features of the vision and text modalities for doing contrastive learning and performing sarcasm detection.
In our experiments, encode both the vision and text modalities using Transformer encoders with heads and layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a le-4 learning rate.
a dataset for understanding multimodal figurative languages. contains 6, 697 matching images and figurative captions (rather than literal captions) of three types of figurative languages: idiom, simile, and metaphor. The original data for the matching task is provided the form of I caption, 3 distractor images, and matching image. We convert it into a fusion task by only collecting the matching image and text pairs and assigning labels using the type of figurative language it belongs to.
IRFL S8
For this dataset, we do not train from scratch. Instead, we performed continued pretraining using proposed objectives pretrained CLIP [6]] models. We used the CLIP- CLIP-VIT-E model and its pretrained image and text encoders. We performed training for 10 epochs using the Adam optimizer with le-6 learning rate.
D.3 Additional analysis and results
Fusion experiments: In Table 6 and we present more detailed results on the Multibench 455 and IRFL [88) datasets computed from 5 independent runs. FACTORCL significantly outperforms the baselines that do not capture both shared and unique information both supervised and self- supervised settings, particularly
MUSTARD (where unique information expresses sarcasm, such as sardonic facial expressions or ironic tone of voice), and on MIMIC (with unique health indicators and sensor readings). There are also big improvements the two sentiment analysis datasets MOSEI and MOSI, with 6.8% and 21.9% increases respectively when compared to SupCon [TI.
In Table 7 we also see that FACTORCL substantially improves the state-of tfiee-of-theaaae art in classifying images and figurative captions which are not literally descriptive of the image on IRFL, outperforming zero-shot and fine-tuned CLIP [6]] as well as continued pre- pre-training baselines on top of CLIP. While the supervised version gives the best results overall, the self-supervised version with our proposed unique augmentations also performs better than independent augmentations, indicating that in the case without label information, we should always try to find and use unique augmentations when possible. In experiments,
word masking for text augmentations. For independent image
27

---

Table 10: We verify conditional lower and upper bound estimators synthetic dataset with varying dimensions representation d, and fixed number samples n.
Dimension dz, nt 2 x 10
10
20
50
100
CCMI (MI-Diff + Classifier)
2.30
2.18
2.15
1.98
1.67
Conditional InfoNCE
2.18
2.20
2.20
2.26
2.30
Conditional InfoNCE-CLUB
3.70
2.95
2.98
2.79
2.86
True CMI
2.32
2.32
2.32
2.32
2.32
Table 11: We probe whether the InfoMin assumption from Tian et al., [(1])) and I(Z2; Y|xi) reasonable for Theorem Compared the shared information ((Xx;XX2), I ((Z;Y||,, much smaller and closer to zero, indicating that the InfoMin assumption is reasonable, and Theorem holds in practice.
T(X1:X2 Y(X),
([x,,Y;X)
X1X2;X1)
T(Z2;1|X,
12.69
12.29
0.40
11.31
10.92
0.38
augmentations, we use cropping, flipping, and color jittering. The unique augmentation simply removes the cropping operation, as illustrated in Figure in the main text.
Additional experiments on high shared information and low unique information: In Table we include additional results using our method on the CIFARIO 43 and MNIST [19] datasets. Our method outperforms the self-supervised contrastive learning on both datasets as expected, and roughly maintains the same performance as supervised contrastive learning. Therefore, in cases with abundant shared information (two modalities with high shared information or two different views generated from augmentations), our method recovers the performance of existing methods that do not capture unique information.
Experiments on CMI estimator verification: In Table 9 and Table 10 we include experiment results which verify that computing the conditional MI lower and upper bounds via concatenation indeed yields reliable estimates. In particular, we aim to verify that the the Conditional InfoNCE objective gives lower bound of the CMI, and the Conditional InfoNCE-CCU CLCB objective gives an upper bound of the CMI. We follow the experiment setups in (55), presenting the true CMI and results of our estimators on synthetic data with fixing dimension of representation Z and varying samples n, and fixing samples n and varying d=. The specific implementations used for conditional InfoNCE. and conditional InfoNCE-CI CECBBLBBB can found Equation 13 and Equation 14 respectively. The results indicate that our Conditional InfoNCE gives estimations smaller than the true CMI, and Conditional BHANCE-CLUB gives estimations greater than the true CMI. The performances are comparable to estimators in 55 suggesting that our method yields valid and competitive lower and upper bounds for CMI.
Empirical verification on InfoMin assumption: To verify the InfoMin assumption [72
YY [(Z2;;\[X,) 0), we use the same synthetic dataset as Table I and measure X(2); The results are shown in Table II we get I (xx:: 12.29 and I [(2: Y (X2) 0.4. T(Z1;||X2 much smaller and closer to zero than I(Z;; 222 indicating that the InfoMin assumption holds in practice.
Compute resources: All experiments in this paper
single NVIDIA A100 GPU. It takes about 10 to 12 GPU hours to train the model on the CIFARIO 43 for 300 epochs, and all the other experiments can be finished within GPU hour using our specified hyperparameters.
28


Reviews:
[{"review_id": "AOo2oIHXkm", "reviewer": "Reviewer_fNpT", "summary": "The goal of the proposed work is control over the information content of representations.  The most common form of contrastive learning leverages multi-view redundancy, where data points  are paired across different modalities or different augmentations in the same modality, and a contrastive loss is used to extract the redundant (or shared) information across the pairs.  The authors propose a methodology to extract the shared information and more---the information that is unique to each modality and still relevant to the task---with each component of the information in its own representation.  While contrastive learning based on multi-view redundancy involves maximizing a lower bound on the mutual information (e.g., InfoNCE), the proposed method requires multiple information terms, including the minimization of certain ones, for which the authors propose an efficient scheme of upper and lower bounds.  For scenarios where the task is not given, the task-relevant unique information is defined through a principled augmentation scheme.  The proposed method is benchmarked on a synthetic experiment, where the information content in each variable can be controlled, and then on several multi-modality datasets, where the goal is simply test set performance.", "questions": "- In theory, the interaction information (“task relevant shared information”) $I(X_1;X_2;Y)$ can be negative.  Are there any interesting consequences of such a scenario in the proposed method?\n- Is Definition 1 expressing what is intended?  I imagine the intent is to express $I(X_1;Y|X_2)>0$, but it does not seem correct.\n- What is RUS?  It is all over the attached code, without any reference in the text nor any comments in the code.  RUSModel, RUSAugment, RUS_CIFAR10, etc. \nMinor points:\n- There is a $1/K$ missing in the denominator of Eqn. 10\n- Typo in Fig 1, both input variables are labeled $X_1$\n- Typo in Line 113, the unique information is written as the shared info\n- Fig 2: it is confusing that the argmax LHS is replaced with information bounds on the RHS, and that $Y$ is replaced by $X_1^`$ and $X_2^`$ without explanation and without removing $Y$ from the Venn diagrams on the RHS\n- Some mutual information quantities seem to be in bits (Table 1) and others in nats (Fig 3), though it’s rarely stated.", "limitations": "Yes, limitations were addressed in Appendix A.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed work is well-motivated.  Control over the specific information content of learned representations is a valuable research pursuit, and the information theoretic analysis is reasonable.  The practicality of the methodology is an important strength: standard contrastive learning losses are all one needs, and the MI upper bounds are a clever re-utilization of critics trained concurrently for MI lower bounds.  The strategy of principled data augmentation as a replacement for task information is interesting.  Overall, the ideas pursued in this work have substantive originality and significance.", "weaknesses": "While the ideas pursued in this work are good, there are some issues that should be remedied before publication. \n- Most importantly, there doesn’t seem to be any support for the way in which the proposed work estimates conditional MI by concatenating the conditional variable(s) to the critic inputs.  This is a significant issue, as both objectives in the information decomposition rely on estimation of the conditional mutual information.  The details are not given (The Appendix refers to a nonexistent section: “In this work, we implement the conditioning in $p(x_1, x_2∣x_1^\\prime, x_2^\\prime)$ through concatenation and the details are in Appendix Sec.” (Appendix, L720).  Instead there is a seemingly irrelevant discussion of a kernel-based alternative to what is actually done.  \n- As currently written, Theorem 1 is incorrect, as shown by a simple counterexample.  The identity transform ($Z_1=X_1$, $Z_2=X_2$) “perfectly maximizes” eqn 2, and gives $I(Z_1,Z_2;Y) = I(X_1,X_2;Y)$.  The proof in the Appendix depends on the InfoMin proposition from Tian et al (2020), that posits good representations also minimize $I(Z_1;Y|Z_2)$.  This is not the same as saying that normal CL approaches necessarily minimize $I(Z_1;Y|Z_2)$.  The identity transform also optimizes Eqn 7, the objectives defining the unique representations, suggesting something else is needed in the objective. \n- The idealized augmentation strategy is a nice return to the self-supervised setting, but it feels too far removed from reality. L167: \"In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background.\"  No, the task relevant information is only a couple of bits, and $I(X;X^\\prime)=I(X;Y)$ is not achieved by only swapping out the background.  An augmentation that would achieve this extremely high bar would swap the object for another of its class, changing the image entirely.  Perhaps another way to see it: every pixel pertaining to the car in Fig 4 could be jittered, or the structure of the car could be changed quite significantly while still linking it to the original image X, and this represents a ton of information $I(X;X^\\prime)$ for any commonly used augmentation, far exceeding $I(X;Y) \\le H(Y)$.  While motivation by an ideal augmentation can be helpful, I think the section would benefit from a more sober take on the implications of realistic augmentations on the proposed information decomposition.\n- Important details seem to be missing.  \n  - What specifically is Cross+Self and Cross+Self+Fact?  It is referred to in vague terms, as a “category” (L219), “capturing a range of methods” (L218), and Cross+Self+Fact “is approximately done in prior work” (L221).  \n  - I like the setup of the synthetic experiments, but they are opaque as well.  Without more information, the accuracies of Fig. 1 mean little.  All methods achieve better than 90% for all settings, even SimCLR when all information is unique?  The text (main and appendix) is vague: “The label $y$ is generated as a function (with nonlinearity and noise) of varying ratios of $w_s$, $w_1$, and $w_2$ to represent shared and unique task-relevant information” (L239 and 874).  After searching the attached code, the label $Y$ is created with a threshold on a sigmoid of the average of vector components, which is no different than a simple hyperplane -- no nonlinearity and no noise.  With more clarity, the synthetic results could be a much stronger inclusion in the current work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[Conditional MI] We apologize for the incomplete reference in Appendix and have fixed it. This conditioning scheme is briefly stated in Lines 171-172 and elaborated in Lines 826-830 for supervised and Lines 834-837 for SSL. Conditioning is done by concatenating the encoded $X_1$ and encoded $X_1’$, concatenating encoded $X_2$ and encoded $X_2’$, and feeding the two vectors into $I_{NCE}$ and $I_{NCE-CLUB}$ - see Figure 1 in attached rebuttal pdf. Conditioning by concatenating is commonly used, for example, in:\n\nMirza and Osindero, Conditional Generative Adversarial Nets. 2014.\n\nReed et al., Generative Adversarial Text to Image Synthesis. ICML 2016.\n\nRombach et al., High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.\n\n[Theorem 1] **We would like to clarify three assumptions referred to in this paper, which have subtle differences (using $I(X_1; Y | X_2) $ without loss of generality):**\n1. Multi-view redundancy (about data): $I(X_1; Y | X_2) \\le \\epsilon$, used by standard CL but **not** by Theorem 1, states that the task-relevant information from unique part is minimal ($\\le \\epsilon$);\n2. Non-negative unique information (about data): $I(X_1; Y | X_2) > 0$, used by Theorem 1 (Line 94), states that task-relevant information from unique part is nonzero;\n3. InfoMin Tian et al. (about representation): $I(Z_1; Y | X_2) = 0$, used by Theorem 1 (Lines 645-649), states that the optimal representation $Z_1$ learns task-relevant information only from the shared part.\n\nWe will clarify these in the main text. **Theorem 1 is correct under Assumptions 2 and 3.** Under Assumptions 2 and 3, $Z_1=X_1, Z_2=X_2$ is not possible for Theorem 1, as $Z_1$ and $Z_2$ only capture the shared part from Assumption 3, but $X_1$ and $X_2$ contain task-relevant information from the unique part from Assumption 2, which cannot be captured by $Z_1$. \n\n**We also check Assumptions 2 and 3 empirically**. Table 1 and 4 in the main text show that Assumption 2 holds empirically. To verify Assumption 3, we use the synthetic dataset as Table 1 and measure $I(Z_1; Y | X_2)$ in standard CL (SimCLR). We get $I(X_1; X_2)=12.29$ and $I(Z_1; Y | X_2)=0.4$ (see Table 1 in the rebuttal pdf). $I(Z_1; Y | X_2)$ is much smaller and closer to zero than $I(X_1; X_2)$, indicating that both Assumptions are reasonable and Theorem 1 holds in practice. \n\nNevertheless, if the InfoMin assumption does not hold, we get $I(Z_1, Z_2; Y) \\leq I(X_1, X_2; Y)$, with the equality satisfied only if $Z_1=X_1$ and $Z_2=X_2$. We will add the equality case to our results. Since identity transformation is nearly impossible in real-world datasets, Theorem 1 still holds for empirical scenarios. \n\nIdentity transformation also optimizes the uniqueness in Eq. 7: this does not violate any of our assumptions and results but does not yield ideal representations for each term (Lines 124-129). Empirically, through the term $-I_{NCE-CLUB}(X_1; X_2)$ in Eq. 9, the loss will remove the shared part in $Z_{U_1}$ and $Z_{U_2}$, making $Z_{U_1} = X_1$ and  $Z_{U_2} = X_2$ practically very unlikely.\n\n[Augmentation] Below, we discuss the implications of three types of augmentations:\n1. Augmentations that exactly satisfy $I(X_1; X_1') = I(X_1; Y)$ and $I(X_2; X_2'|X_1) = I(X_2; Y|X_1)$ (Eqs.17-18); exactly satisfying these conditions is hard, but empirically we do not exactly require this assumption - instead we relax it to:\n2. Augmentations that approximately satisfy $I(X_1; X_1') \\approx I(X_1; Y)$ and $I(X_2; X_2'|X_1) \\approx I(X_2; Y|X_1)$: our intuition is to avoid augmentations that will remove or strongly destroy information shared by the other modality (e.g., the caption), and augment via cropping or color jittering in the image. We will clarify this in the paper as a guideline for practitioners. These **consistently outperform independent augmentations (Table 3), suggesting that approximately satisfying this condition is sufficient**.\n3. Augmentations that do not consider this condition at all (i.e., independent modality augmentations as done in prior work): this has worse performance.\n\n[Cross+self] Cross + Self refers to all self-supervised methods that learn one representation trained jointly for two objectives: cross-modal contrastive loss (e.g., image-text contrastive) plus unimodal contrastive (e.g., vision only contrastive from two image views) [e.g., 27,29,56].\n\nCross+Self+Fact includes self-supervised methods with separate representations, one trained for cross-modal contrastive objectives and another trained for unimodal contrastive objectives, such as [76,79].\n\n[synthetic exps] The data is generated using multivariate Gaussians with fixed means and variances, which add noise and randomness. The label is a linear function of the latent variables - we found this setup to display the most evident trends. We also tried non-linear data with noise but observed that all methods fluctuate more since we only do linear probing on the final representation. We refer the reader to our experiments on real-world datasets (Table 2) for comparisons with non-linear labels and noise.\n\n[Negative interactive information] Interaction information is:\n1. Positive when there is more task-relevant shared info than irrelevant, so FactorCL has to do more work in **capturing** task-relevant information (via lower bound).\n2. Negative when there is more task-irrelevant shared info than relevant, so FactorCL has to do more work in **removing** task-irrelevant information (via upper bound).\n\nThese insights also give a better idea of the weights assigned to learning from these two objectives.\n\n[RUS] RUS was the initial acronym we used, standing for redundant, unique, and synergistic interactions that we aimed to learn via factorized learning.\n\n[Typos and bits vs. nats] We have corrected the term and added notes to distinguish when we use nats or bits.\n\n[Figure] We have fixed the typo in Figures 1 and 2 (used $X’$ instead of $Y$) and added them to the rebuttal PDF."}, {"review_id": "vzA01lF7FW", "reviewer": "Reviewer_859W", "summary": "This paper presents FACTOR CL, a method for multimodal representation learning that captures both shared and unique task-relevant information, going beyond the common approach of focusing on shared information across different data modalities. FACTOR CL is based on three key contributions: factorizing task-relevant information into shared and unique representations, capturing and discarding task-relevant and irrelevant information through optimizing mutual information bounds, and utilizing multimodal data augmentations to approximate task relevance in the absence of labels. On large-scale real-world datasets, FACTOR CL achieves state-of-the-art results on six benchmarks. The premise of this paper is quite similar to the following work (Chaudhuri et al., Cross-Modal Fusion Distillation for\nFine-Grained Sketch-Based Image Retrieval: https://bmvc2022.mpi-inf.mpg.de/0499.pdf), however, the formulation, experimentations are different.\n", "questions": "(1) I wonder what are the limitations of current contrastive learning (CL) methods in modeling unique information in the context of multimodal data.\n\n(2) It would be nice to clarify how the proposed method distinguishes between shared and unique information across different modalities. What does the 'uniqueness gap' in this context mean? How does it affect the learning process?\n\n(3) How does the proposed framework handle situations where there is little to no shared information between modalities, or where most of the shared information is irrelevant to the task?\n\n(4) In the derivation of supervised contrastive learning objectives, how do the lower and upper bounds for mutual information terms impact the learning process?\n\n(5) What are the practical implications of applying semantic augmentations on each modality in the context of unsupervised contrastive learning?\n\n(6) How does the method ensure the learning of task-relevant information without access to labels in the case of unsupervised contrastive learning?\n", "limitations": "The authors have not adequately addressed the limitations of the proposed method. Potential limitations could be found from my comments listed under the weaknesses and limitations sections.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This method is innovative in its attempt to model not only the shared information but also the unique information between different modalities, which is often overlooked by traditional Contrastive Learning (CL) methods. The framework is applicable to a broad range of settings, not just those where shared information is dominant. This allows it to handle diverse multimodal distributions, even those with substantial unique information.\n\n(2) The proposed method utilizes self-supervised data augmentations, which allows it to learn task-relevant information without access to labels. This enables the algorithm to learn in a more unsupervised manner, reducing the need for extensive labeled data.", "weaknesses": "(1) The method relies on certain assumptions, such as the optimal augmentation assumption, which may not always hold in real-world applications. If these assumptions do not hold, the effectiveness of the method may be significantly compromised. The method's performance might heavily rely on the quality of the data augmentation techniques used, which might not always be easy to decide or optimize.\n\n(2) Calculating mutual information can be challenging in practice, especially when dealing with high-dimensional data. Though approximations are used, they may not always accurately capture the true mutual information.\n\n(3) The method involves complex formalism and numerous mathematical assumptions, which could make it difficult to implement and adapt to different scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[Assumption] Augmentations that exactly satisfy $I(X_1; X_1') = I(X_1; Y)$ and $I(X_2; X_2'|X_1) = I(X_2; Y|X_1)$ (Eqs.17-18) are hard, so instead we relax it to $I(X_1; X_1') \\approx I(X_1; Y)$ and $I(X_2; X_2'|X_1) \\approx I(X_2; Y|X_1)$, by using these intuitions to guide augmentation design (details in Appendix D.3 and examples in Table 5): our intuition is to avoid augmentations that will remove or strongly destroy information shared by the other modality (e.g., the caption), and augment via cropping or color jittering in the image. We will clarify this in the paper as a guideline for practitioners.\n**Our augmentations consistently perform better than independent augmentations (Table 3), suggesting that approximately satisfying this condition is sufficient**. Most importantly, the unique augmentations are simple and scalable to real-world datasets.\n\n[Estimator accuracy] Our goal is not to calculate MI precisely but to derive lower and upper MI bounds to design objectives that scale on real-world datasets, making training Factor-CL on large datasets possible. Our work provides opportunities for future work to integrate tighter MI estimators (e.g., Guo et al.) to capture task relevance.\n\nGuo et al. Tight mutual information estimation with contrastive fenchel-legendre optimization. NeurIPS 2022.\n\n[Complex math] The details cover the exact mathematical derivation from shared and unique task-relevant information definitions to our final self-supervised objectives. For clarity, we will add an overview of the derivation steps and intermediate objectives. **We also emphasize that the actual implementation is relatively straightforward to adapt to a standalone contrastive framework (Algorithms 2 and 3). Code is included in the supplementary, and we will release it publicly so that all implementation details are fully transparent and reproducible.**\n\n[Limitations of CL] Theorem 1 shows that current CL methods only keep shared information and discard unique information from modalities. This makes representations of standard CL sub-optimal because task-relevant information from uniqueness is not captured. This is also empirically supported by Figure 1, where the downstream performance degrades consistently as unique information increases, suggesting the sub-optimality of standard CL. Table 2 also shows that standard CL’s failure to capture unique information leads to inferior performances on real-world datasets. These are fundamental limitations of CL methods that require new learning paradigms to solve.\n\n[Shared and unique info, uniqueness gap] We define shared information as $I(X_1; X_2; Y)$, unique information as $I(X_1; Y | X_2)$ and $I(X_2; Y | X_1)$, and uniqueness gap as $I(X_1, X_2; Y) - I(Z_1, Z_2; Y)$. The uniqueness gap measures the difference of task-relevant information between input ($X_1, X_2$) and encoded representation ($Z_1, Z_2$). Standard CL will have this gap because current CL methods only keep shared information and discard unique information. In practice, datasets such as IFRL in Table 3 may have high task-relevant unique information, which may widen this gap in standard CL, as discussed in Lines 36-39. In the proposed Factor-CL, the optimal representations will close this gap by capturing both shared and unique info, as shown in Theorem 3.\n\n[No shared or shared is task-irrelevant] Our method handles both of these cases:\n1. If there is little shared information, our method will still capture the shared information by the lower bound in Eq. 8 and unique information by the lower bounds in Eq 9.\n2. If most shared information is irrelevant - our method will discard the irrelevant shared information by the upper bound in Eq. 9 and keep the relevant shared information by the lower bound in Eq. 8.\n\n[How bounds impact learning] Our derived lower and upper bounds approximate the actual shared and unique information regions intractable to compute exactly. Our bounds are closer to true MI than existing ones (NCE and CLUB), as shown in Figure 3. Optimizing these tight lower and upper bounds enables us to learn representations to capture different information regions efficiently.\n\nFactorization is also important - each representation optimizes different lower or upper bound objectives, so we can capture the correct information to satisfy multiple terms simultaneously. This makes training easier and performance better (Table 2 and Lines 287-289).\n\n[Practical implications] Besides the implications for multimodal augmentation (Lines 189-196), we discuss augmentation intuitions for each individual modality. For vision, if downstream tasks are object-oriented, the implication is to augment task-irrelevant parts only (e.g., non-object semantic parts if the task is object-centric) or to avoid image augmentations that hugely remove information (e.g., cropping and greyscale). For text, we should try to avoid masking large chunks of text since this can hugely remove information. We want to emphasize that these choices are scalable and much less expensive than labeling data.\n\n[How to ensure learning task-relevant information without labels] We discuss this in section 3.2, specifically definitions 2 and 3. Essentially we design suitable augmentations of each modality, extending the work in [62], and replace Y with X’ (augmented data view) to enable the learning of task-relevant information without access to labels. The details can be found in Lines 157-174.\n\n[Limitations] We will add more discussions regarding limitations:\n1. Optimizing better MI lower and upper bounds for higher-dimensional and complex modalities.\n2. Extending the work of InfoMin [Tian et al.] to automatically generate data augmentations to satisfy the optimal augmentation assumption, and leveraging progress in generative models for data augmentation.\n3. Quantifying shared and unique information to optimize FactorCL better."}, {"review_id": "lk5YEVA3j2", "reviewer": "Reviewer_bfCW", "summary": "Based on the mutual-information theory, this paper proposes a new multi-modal contrastive learning method (FactorCL) to learn both shared and unique multi-modal task-relevant information, which captures task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bound. The method achieves significant influence in different real-world datasets.", "questions": "See weaknesses.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. How to capture both the shared and unique task-relevant information in different modalities is interesting.\n2. The theoretical analysis looks solid and cooperates well with empirical results.\n3. The writing is well and easy to follow. Especially the figures represent the main ideas and are easy to understand.\n4. The empirical results show significant improvements on different types of datasets.", "weaknesses": "1. It seems that the unique-augmentation step is strongly related to the task-relevant information of different modalities. For example, when the text is “a yellow flower”, the ColorJitter operation should be removed. And when the text is “a flower”, the ColorJitter operation is a useful data augmentation. Is it possible to design a strategy that can automatically select the appropriate data augmentations?\n2. As shown in Table 2, FactorCL-SSL and FactorCL-SUP shows significant difference on some datasets. Especially, on MOSI, SimCLR and Supcon show similar performance while the performance of FactorCL-SSL and FactorCL-SUP have a large gap. Is it possible to provide more discussions about that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[Automatic augmentations] We find that **augmentations that approximately satisfy the optimal multimodal augmentation defined in Eqs.17-18 are sufficient for good performance, which is simpler and straightforward to implement on real-world datasets** than strictly satisfying Eqs.17-18. Our intuition is to avoid augmentations that remove or strongly destroy information shared by the other modality (e.g., the caption) and augment via cropping or color jittering in the image. We will clarify this in the paper as a guideline for practitioners. From Table 3, we outperform independent augmentations and other baselines.\n\nIt is possible to extend the work of InfoMin [Tian et al.] to generate data augmentations to automatically satisfy the optimal augmentation assumption. Still, these methods require generative models for multiple modalities. We think this is an exciting direction for future work and expect progress in multimodal generative models to yield further advances in these problems.\n\n[Discussion on the gap between FactorCL-SSL and FactorCL-SUP] We found that training a generic multimodal model with SSL was difficult due to the small sample size and high-dimensional and temporal challenges of the MOSI dataset. During rebuttal, we adapted the best supervised multimodal architecture for MOSI. We performed CL pre-training on this architecture, which yielded an improved FactorCL-SSL performance of 80%, closer to FactorCL-SUP performance. We include these updated results and comparisons in Table 2 of the attached rebuttal pdf.\n"}, {"review_id": "sMoE2Pf0pq", "reviewer": "Reviewer_4hLy", "summary": "This work addressed the problem of contrastive learning in a multimodal setting, particularly in capturing shared and modality-specific information regarding downstream tasks. Existing approaches assume that the information contained in different modalities is somewhat the same (redundant), but in the real world, it might not be the case. The authors proposed to factor the task-specific mutual information into shared and unique components. They provided a derivation for this factorization and their FactorCL algorithm optimized these two parts with approximated upper and lower bounds separately, under an assumption of optimal unimodal augmentation. The authors evaluated against chosen baselines on a collection of tasks in the MultiBnech benchmark and show improvements over these baselines.", "questions": "- why don't you conduct experiments on various well-established vision language image-text datasets out there such as MS-COCO or Flickr30K? ", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors provided a lot of details about their approaches, from the analysis of CL algorithms, and their approximation to experiment results.\n- Most of the simplifications look intuitively reasonable \n- On the chosen benchmark, the gains by their algorithm were decent\n", "weaknesses": "- The paper is packed with too many details which makes it hard to follow the main idea \n- On the experiment results, for example in Table 2, even though there were improvements over baselines, the results were too far below those reported in previous works, including the original MultiBench paper (https://arxiv.org/pdf/2107.07502.pdf - also Table 2). This casts doubt on the validity of the approach of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[Too many details] The details cover the exact mathematical derivation from the definitions of shared and unique task-relevant information to our final self-supervised objectives. We will add an overview of the derivation steps and intermediate objectives and move some details to Appendix for clarity. **We also emphasize that the actual implementation is relatively straightforward to adapt to a standalone contrastive framework (Algorithms 2 and 3). Code is included in the supplementary, and we will release it publicly so that all implementation details are fully transparent and reproducible.**\n\n[Performance gap] There are 2 key differences between the experiments:\n1. Our approach is currently implemented for two modalities as we primarily study two views in our framework, while the best reported MultiBench results are for supervised learning with three modalities.\n2. MultiBench and related work design the best modality and task-specific multimodal architectures to really push for the best supervised learning performance (e.g., complex multimodal transformer methods), while we use generic encoders (not necessarily the most complex architectures) and focus on contrastive representation learning objectives, and use linear probing to evaluate performance. This is in line with observations where SSL with generic encoders (e.g., SimCLR) can have lower accuracies than supervised models whose architecture is optimized for supervised performance (e.g., CoAtNet [Dai et al.]) and due to linear probing evaluation (e.g., Chen et al.).\n\n\nDuring the rebuttal period, we scaled up some experiments to address the differences due to points 1 and 2 and summarize the results in Table 2 in the rebuttal pdf:\n1. We use the architecture from the best supervised models, and we first pretrain the model using FactorCL-SSL and then perform supervised fine-tuning to evaluate. The total number of epochs for FactorCL-SSL pretraining equals the total number of epochs for training supervised baselines, with FactorCL-SSL having a few epochs (less than ten) for fine-tuning. We also extended FactorCL to three modalities: we first perform vision + language FactorCL-SSL pretraining and concatenate audio features for supervised fine-tuning. The number of fine-tuning epochs is also less than ten.\n2. **Overall, we see better results from Factor-CL (ours) than the supervised baselines we reproduced across different datasets as well as architectures**, suggesting that our method achieves stronger results by capturing unique task-relevant information. There is still a gap between the FactorCL-SSL results and the reported supervised results in Multibench; we attribute this small gap to the fact that the supervised models in Multibench use all three modalities to train from scratch, while we only use the audio modality to fine-tune with a few epochs, so it does not learn all multimodal interactions.\n\nDai et al. Coatnet: Marrying convolution and attention for all data sizes. NeurIPS 2021.\n\nChen et al. Big self-supervised models are strong semi-supervised learners. NeurIPS 2020.\n\n[Datasets] These established vision language retrieval benchmarks test only for shared information between images and captions and do not need unique information. \n\n**During the rebuttal period, we further added experiments on the NYCaps dataset** [Hessel et al.], a new dataset testing the matching of cartoon images and humorous captions, which requires unique humorous information in images and text. We show strong performance when continuing contrastive training on top of a pre-trained CLIP using SimCLR, SupCon, or FactorCL and evaluating using zero-shot retrieval, **SimCLR has an accuracy of 49.2%, SupCon has an accuracy of 49.7%, and our method outperforms both by achieving 50.5%.**\n\nOur paper also includes experiments on IRFL [78], which aims to match images and figurative captions (rather than literal captions), requiring more unique information. From Table 3, we outperform the state-of-the-art in classifying images and figurative captions, outperforming zero-shot, fine-tuned, and continued pre-trained CLIP models.\n\nHessel et al., Do Androids Laugh at Electric Sheep? Humor \"Understanding\" Benchmarks from The New Yorker Caption Contest. ACL 2023.\n"}, {"review_id": "Csg8MdwrY3", "reviewer": "Reviewer_N8Dx", "summary": "The paper introduces FactorCL, a novel method for learning multimodal representations. The proposed method generalizes traditional MI maximization-based approaches by capturing both shared and unique information relevant to downstream tasks. Based on the information-theoretic perspective, the paper demonstrates that traditional CL approaches, based on shared task-relevant information maximization, perform sub-optimally in the presence of non-shared task-relevant information, which is an easily imaginable case in a multi-modal setting. From here, the paper proposes a novel CL objective and augmentation strategy to address the uniqueness gap. Experimental evaluation demonstrates that the proposed method outperforms several existing CL approaches in multi-modal setting.", "questions": "My questions are in Weaknesses 1,2,3,4.", "limitations": "I suggest the paper include a separate section on the limitations of the methods, where authors elaborate on challenging scenarios, where they expect the proposed approach to not deliver substantial improvements.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work is very timely. The paper addresses an important question of adopting CL methods for multi-modal scenarios.\n2. Paper is clearly written and easy to follow.\n3. The proposed method can be seen as an information-theoretic generalization of base contrastive learning approaches, which only address shared task-relevant information. This is a natural extension. \n4. Section 2 provides an intuitive information-theoretic analysis of multi-modal CL. Figure 2 gives a nice overview of the bounds and their role in the objective.\n5. Experimental results clearly demonstrate the benefits of FactorCL. ", "weaknesses": "1. If the information is modality \"unique\", but task-relevant, shouldn't it end up in the shared task-relevant partition by means of simply optimizing for the shared information (as non-shared info is implicitly minimized this way) as standard CL methods do?\n2. If weakness 1 holds, then what is the reason why standard CL methods fail? Can it be solely attributed to sub-optimal augmentation, which limits the shared information?\n3. Table 2 reveals that most of the performance increase can be attributed to improved augmentation. Is it connected to Weaknesses 1 & 2?\n4. L 31-39. Isn't low shared information a direct consequence of high unique information (and vice versa)? If yes, then there is no need to define it as a separate imitation.\n\nMinor:\n - Figure 2 typo X1 (modality 2)\n - Eq (1) typo I(x1, x2, y)  -> I(x1, x2 | y) \n - Formatting of Table 2 can be improved, the top row X1;X2 is hard to read\n - L171-174 is confusing. Does it mean X1 and X2 are concatenated with the encoded features of (X1' & X2')? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments! We respond to some concerns below:\n\n[Unique partition] We define shared information as $I(X_1; X_2; Y)$ and unique information as $I(X_1; Y | X_2)$ and $I(X_2; Y | X_1)$ based on information theory, and from this definition the two areas of shared and unique information are strictly disjoint. **Note that multimodal and multiview are different: in multimodal, the $X_1$ and $X_2$ are fixed, so the shared and unique regions are fixed. In multiview, a single input $X$ is augmented into $X$’, so there are ways to control how much shared and unique information there is between $X$ and $X’$ [Tian et al., InfoMin].**\n\nIn practice, Theorem 1 shows the result of CL on multimodal data - it will discard unique information according to our definition. Figure 1 shows this in practice, with a gradual decrease of standard CL in performance as the ratio of unique information increases.\n\n[Is Weakness 1 true, and why does standard CL fail] Weakness 1 does not hold, as we explained in the answer above. Why standard CL fails is not solely due to sub-optimal augmentation but is a fundamental limitation of standard CL methods - **contrasting between $(X_1, X_2)$ pairs can only capture shared mutual information between $X_1$ and $X_2$, but not unique information regardless of how augmentation is performed. Unique information has to be captured via modality-specific signals, alongside the right modality-specific augmentations and factorized representations, which we show in FactorCL.**\n\n[Performance improvement] The Table 2 performance increase is not only from better augmentation. Both FactorCL-SUP and Factor-SSL have increased performance, but improved augmentation is only used in the FactorCL-SSL. Performance increases from Table 2 come from our method being able to unique task-relevant information in both supervised and SSL, directly addressing limitation 2 in the introduction.\n \n[Low shared and high unique] High unique information may not imply low shared information. There could be cases where both shared information and unique information are high. The MOSEI dataset [83] is an example where modality-unique information (text expressing a joyful event) and modality-shared information (happiness from a delighted voice and a smiling face) are both rich and useful for downstream sentiment/emotion tasks.\n\n[Minor comments] We thank the reviewers for the suggestions. We fixed the Figure 2 typo. The Eq. 1 does not have a typo, and $I(X_1; X_2; Y)$ is the correct form of the shared information. We have improved the formatting of Table 2. And yes, Lines 171-174 mean $X_1$ and $X_2$ are concatenated with the encoded features of ($X_1'$ & $X_2'$)\n\n[Limitations] We will add more discussions regarding limitations to Appendix:\n1. Optimizing better MI lower and upper bounds could improve performance for higher-dimensional and complex modalities.\n2. Extending the work of InfoMin [Tian et al.] to automatically generate data augmentations to satisfy the optimal augmentation assumption, and leveraging future progress in multimodal generative models for data augmentation.\n3. Quantifying whether shared or unique information is more important for the task to optimize FactorCL better."}]

Author rebuttal:
Dear reviewers, we are extremely grateful for your valuable feedback and insightful comments. We are glad that you agree that our results are innovative (859W), original (fNpT), significant (fNpT), and applicable to a broad range of settings in contrastive learning (859W, bfCW, N8Dx, 4hLy). Your concrete suggestions are a valuable step in this direction, and we have revised our submission accordingly to take these into account. In this short note, we summarize the main changes we made to our submission:
1. Figure 1 to clarify the conditioning process of conditional Info-NCE question raised by Reviewer fNpT. 
2. Table 1 to verify the InfoMin assumption (Tian et al.) used in Theorem 1 empirically, showing that Theorem 1 holds in practice - standard CL methods struggle to learn unique information.
3. Table 2 shows FactorCL (ours) further improves performance by using the same modalities and architectures as the supervised baselines from Multibench [37], and FactorCL outperforms the reproduced supervised methods.
4. In Table 3, we also added experiments on NYCaps, a new dataset that requires unique information in cartoon images and humorous captions, and we show FactorCL outperforms SimCLR and SupCon.
5. Even though the optimal augmentations are hard to satisfy, in practice, we use these intuitions to guide augmentation design, yielding state-of-the-art results and scales to real-world datasets. Our intuition is to avoid augmentations that remove or strongly destroy information shared by the other modality (e.g., the caption) and instead augment via cropping or color jittering in the image. We will clarify this in the paper as a guideline for practitioners.
6. Finally, we would like to emphasize that all code and data are in the supplementary material, and we plan to release all code and data on GitHub after the review period so details are transparent and reproducible. We have added all requested details regarding the method and experiments to the appendix.

Meta-review: This paper presents a self-supervised multimodal representation learning method to capture both shared and unique information in multiple modalities. The proposed method factorizes task-relevant information into shared and unique parts, learns specific information content through optimizing upper and lower mutual information bounds, and performs multimodal data augmentations to approximate task relevance. 

Reviewers appreciated the motivation, the idea, and the novelty of the proposed method, but also raised concerns on the support of the estimation of conditional MI, Theorem 1, the reality of the augmentation and assumptions, the practical implementation, the experimental results, and technical details. The rebuttal addressed most concerns. The AC agreed with the strengths of the proposed method. Reviewers did raise some valuable concerns that should be addressed. The authors are encouraged to make the necessary changes in the camera-ready version.
Decision: Accept (poster)