SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) = \mathbb {E} _ {p \left(x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ \mathbb {E} _ {\substack {x _ {1}, x _ {2} ^ {\prime} \sim p \left(x _ {1}, x _ {2} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \\ x _ {2} ^ {\prime} \sim p \left(x _ {2} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)}} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+} , x _ {1} ^ {\prime} , x _ {2} ^ {\prime}\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-} , x _ {1} ^ {\prime} , x _ {2} ^ {\prime}\right)} \right] \right] \tag{15}
$$
$$
\begin{array}{l} I _ {\mathrm {N C E - C L U B}} \big (X _ {1}; X _ {2} | X _ {1} ^ {\prime}, X _ {2} ^ {\prime} \big) = \mathbb {E} _ {p (x _ {1} ^ {\prime}, x _ {2} ^ {\prime})} \Big [ \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p (x _ {1}, x _ {2} | x _ {1} ^ {\prime}, x _ {2} ^ {\prime})} \big [ f ^ {*} \big (x _ {1}, x _ {2} ^ {+}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime} \big) \big ] \\ \left. - \mathbb {E} _ {\substack {x _ {1} \sim p \left(x _ {1} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \\ x _ {2} ^ {-} \sim p \left(x _ {2} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)}} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {-}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \right] \right] \tag{16} \\ \end{array}
$$
The objectives can be seen as conditional contrastive learning on augmentations  $(X_1', X_2')$ . Here again  $f(x_1, x_2, x_1', x_2')$  is different from the critics in Eqs.(13) because of the different general forms. We implement  $f()$  here as  $g([x_1, x_1'])^T h([x_2, x_2'])$  where  $g(), h()$  are trainable encoders specific for each modality and  $[x_1, x_1']$  denotes concatenation. This concatenation is justified by the CMI estimators in Sordoni et al. [69], who show that concatenating the conditioning variable with the input in the critic  $f(x_1, x_2, x_1', x_2')$  yields a Conditional InfoNCE estimator (Eq.(15)) that is a lower bound for CMI. However, the exact Conditional InfoNCE estimator learns a different conditional distribution  $p(x_1, x_2 | x_1', x_2')$  for each augmented pair  $x_1', x_2'$ , which can be prohibitively expensive. We could approximate this by creating multiple augmentations of a single paired  $x_1, x_2$ . Our code uses one augmented pair  $x_1', x_2'$  for each  $x_1, x_2$  but could be extended to multiple pairs, and we find this simple approach yields consistent CMI lower and upper bounds that are empirically comparable to existing CMI estimators [55, 69]. We include full comparisons and implementation details in Appendix D.1, and in Appendix C.2 we discuss an alternative interpretation based on viewing CL as kernel learning which permits using conditional kernel estimation for our objectives.
Although we find this method to work well in practice, a more careful analysis reveals that 2 separate unimodal augmentations  $X_{1}^{\prime}$  and  $X_{2}^{\prime}$  each satisfying  $I(X_{i};X_{i}^{\prime}) = I(X_{i};Y)$  do not together satisfy  $I(X_{1};X_{2}|Y) = I(X_{1};X_{2}|X_{1}^{\prime},X_{2}^{\prime})$  needed for the substitution in Eqs.(15) and (16) to hold with equality. To satisfy this property exactly, we define optimal multimodal augmentations:
Definition 4. (Optimal multimodal augmentation)  $X_1'$  and  $X_2'$  are optimal multimodal augmentation for  $X_1$  and  $X_2$  when  $I(X_1, X_2; X_1', X_2') = I(X_1, X_2; Y)$ , which implies that the only information shared between  $X_1, X_2$  and  $X_1', X_2'$  is task-relevant with no irrelevant noise.
We satisfy  $I(X_{1},X_{2};X_{1}^{\prime},X_{2}^{\prime}) = I(X_{1},X_{2};Y)$  using two steps:
$$
\text {U n i m o d a l} X _ {1} ^ {\prime} \text {s . t .} I \left(X _ {1}; X _ {1} ^ {\prime}\right) = I \left(X _ {1}; Y\right), \tag {17}
$$
$$
\text {U n i q u e} X _ {2} ^ {\prime} \text {s . t .} I \left(X _ {2}; X _ {2} ^ {\prime} \mid X _ {1}\right) = I \left(X _ {2}; Y \mid X _ {1}\right). \tag {18}
$$
We call the second step unique augmentation: after observing  $X_{1}$ , we create augmented  $X_{2}^{\prime}$  from  $X_{2}$  to keep task-relevant information not already in  $X_{1}$ . To empirically satisfy optimal multimodal augmentations, we avoid augmentations in one modality that will remove or strongly destroy information shared with the other modality. For example, in image captioning, we should avoid image augmentations such as cropping that destroy information from the caption (e.g., cropping object parts referred to by the caption), and instead, only augment images via flipping or color jittering which retains all caption information. Figure 4 shows an example of unique augmentation that satisfies these conditions. In our experiments, we will show that our augmentations consistently perform better than standard augmentations (Table 3), suggesting that approximately satisfying Eqs.(17) and (18) can be empirically sufficient, which is simple and straightforward to implement on real-world datasets.
3.3 Overall method and implementation
Algorithm 1 Standard multimodal CL.	Algorithm 2 FACTORCL.
Require: Multimodal dataset {X1, X2}.	Require: Multimodal dataset {X1, X2}.
Initialize networks f(·).
while not converged do
for sampled batch {x1, x2} do
Estimate INCE(X1; X2) from Eq. 10
L = -INCE(X1; X2)
Update f(·) to minimize L
end for
end while
return f(·)	Initialize networks f(·).
while not converged do
for sampled batch {x1, x2} do
x1' ← Augment(x1)
x2' ← Unique-Augment(x2|x1)
Plug x1' and x2' into Eq. 15 and 16
Estimate S, U1, U2 from Eq. 8 and 9
L = -(S + U1 + U2)
Update f(·) to minimize L
end for
end while
return f(·)
Input	Task-relevant information	The final algorithm sketch is in Algorithm 2, we
we compare against standard CL in Algorithm
can be shown that FACTORCL learns all the
relevant information from both modalities:
Theorem 3. (Optimality of FACTOR
If ZS1, ZS2, ZU1, ZU2 perfectly max
Eqs.(6-7) and the estimations in E
and (9) are tight, we obtain I(X1, X2; Y)
I(ZS1; ZS2; Y) + I(ZU1; Y|ZS2) + I(ZU2; Y)
suggesting that FACTORCL learns both share
unique task-relevant information.
X1
(Text)	The car is as
fast as a cheetah.	Unique in X1:
"cheetah"
X2
(Image)	Unique in X2:
highway pixels
Y	1
(image and caption
are matched)	Shared between
X1 and X2:
fast car
Standard
Aug for X2	Crop + Flip	Unique task information
in X2 destroyed:
highway pixels
Unique
Aug for X2	Unique task information
in X2 preserved:
highway pixels
No crop, flip only	We include the full proof in Appendix C.3. In
tice, while we do not expect perfect estimation
quantities and maximization with respect to M
jectives, we include implementation details rega
architectures and contrastive objectives that im
empirical performance in Appendix D.1.
Figure 4: Standard vs. unique augmentations for the figurative language [88] dataset. After augmenting text modality  $X_{1}$  independently (same for both augmentation types), we illustrate their differences for image augmentation: unique augmentation on images should avoid removing information referred to by  $X_{1}$  (the text). The text mentions that the car is fast so unique augmentation for images should not remove the highway pixels of the image which can suggest the car is fast.
We run comprehensive experiments on a suite of synthetic and large-scale real-world datasets with varying requirements of shared and unique task-relevant information, comparing our FACTORCL method to key baselines:
1. SimCLR [13]: the straightforward method of cross-modal  $(X_{1},X_{2})$  contrastive learning.
2. Cross+Self [33, 36, 44, 64, 85, 89]: captures a range of methods combining cross-modal  $(X_{1}, X_{2})$  CL with additional unimodal  $(X_{i}, X_{i}^{\prime})$  CL objectives. This category also includes other ways of preserving unique information, such as through (variational) autoencoder reconstructions [81].
3. Cross+Self+Fact [86, 89]: A factorized extension of Cross+Self, which is approximately done in prior work that adds separate (typically pre-trained) unimodal encoders for each modality.
4. SupCon [41], which learns  $I(X_{1};X_{2}|Y)$  using CL conditioned on  $Y$  from labeled data.
We also carefully ablate each component of our method and investigate factors, including training data size and choice of augmentations. The intermediate ablations that emerge include:
1. FACTORCL-SUP: The supervised CL version which uses labels  $Y$  in Eqs.(13) and (14).
Table 1: We probe whether contrastive representations learned by classic CL methods and FACTORCL contain shared  $w_{s}$  or unique  $w_{1},w_{2}$  information. FACTORCL captures the most unique information.
Model Representations	SimCLR	Cross+self	SupCon	FACTORCL
Z1	Z2	Z1	Z2	Z1	Z2	ZU1	ZU2	ZS1	ZS2
I(Z;w1)	4.45	0.16	4.39	0.14	5.17	0.19	7.83	0.03	6.25	0.04
I(Z;w2)	0.17	3.92	0.13	4.26	0.23	5.17	0.06	7.17	0.05	5.79
I(Z;ws)	12.61	12.06	11.30	11.47	7.48	7.17	9.47	9.89	10.13	9.40
2. FACTORCL-SSL: The fully self-supervised version of our approach replacing  $Y$  with multimodal augmentations  $X_{1}^{\prime}$  and  $X_{2}^{\prime}$  to approximate the task.
3. OurCL-SUP: FACTORCL-SUP but removing the factorization so only two features  $Z_{1}$  is optimized for both  $I(X_{1};X_{2};Y)$  and  $I(X_{1};Y|X_{2})$ ,  $Z_{2}$  optimized for both  $I(X_{1};X_{2};Y)$  and  $I(X_{2};Y|X_{1})$ .
4. OurCL-SSL: FACTORCL-SSL but also removing the factorization in the self-supervised setting.
The formulation of each ablation and implementation can be found in Appendix D.1.
4.1 Controlled experiments on synthetic datasets
Synthetic data generation: We begin by generating data with controllable ratios of task-relevant shared and unique information. Starting with a set of latent vectors  $w_{1}, w_{2}, w_{s} \sim \mathcal{N}(0_{d}, \Sigma_{d}^{2}), d = 50$  representing information unique to  $X_{1}, X_{2}$  and common to both respectively, the concatenated vector  $[w_{1}, w_{s}]$  is transformed into high-dimensional  $x_{1}$  using a fixed transformation  $T_{1}$  and likewise  $[w_{2}, w_{s}]$  to  $x_{2}$  via  $T_{2}$ . The label  $y$  is generated as a function (with nonlinearity and noise) of varying ratios of  $w_{s}, w_{1}$ , and  $w_{2}$  to represent shared and unique task-relevant information.
Results: In Figure 1, we show our main result on synthetic data comparing FACTORCL with existing CL baselines. FACTORCL consistently maintains the best performance, whereas SimCLR [13] and SupCon [41] see performance drops as unique information increases. Cross+Self [33, 36, 44, 89] recovers in fully unique settings (x-axis= 1.0) but suffers at other ratios.
Representation probing information: We run a probing experiment to compute how well different contrastive representations capture shared and unique information. In Table 1, for the  $Z_{i}$ 's learned by each method, we approximately compute  $I(Z_{i};w_{1}),I(Z_{i};w_{2})$ , and  $I(Z_{i};w_{s})$  with respect to ground truth generative variables  $w_{s},w_{1}$ , and  $w_{2}$ . As expected, existing methods such as SimCLR capture smaller amounts of unique information (roughly 4 bits in  $I(Z_{i};w_{1})$  and  $I(Z_{i};w_{2})$ ), focusing instead on learning  $I(Z_{i};w_{s})$  (12 bits). Cross+self captures slightly larger  $I(Z_{i};w_{2}) = 4.26$ , and SupCon with labeled data captures up to 5 bits of unique information. Our FACTORCL approach captures 7 bits of unique information and maintains 10 bits of shared information, with total information captured higher than the other approaches. Furthermore,  $\{Z_{S_1},Z_{S_2}\}$  capture more information about  $w_{s},Z_{U_1}$  about  $w_{1}$ , and  $Z_{U_2}$  about  $w_{2}$ , indicating that factorization in our approach is successful.
4.2 Self-supervised multimodal learning with low redundancy and high uniqueness
Multimodal fusion datasets: We use a large collection of real-world datasets provided in Multi-Bench [45], where we expect varying ratios of shared and unique information important for the task, to compare FACTORCL with other CL baselines:
1. MIMIC [38]: mortality and disease prediction from 36, 212 medical records (tabular patient data and medical time-series sensors from ICU).
2. MOSEI [93]: multimodal sentiment and emotion benchmark with 23,000 monologue videos.
3. MOSI [91]: multimodal sentiment analysis from 2, 199 YouTube videos.
4. UR-FUNNY [27]: a dataset of humor detection from more than 16,000 TED talk videos.
5. MUSTARD [12]: a corpus of 690 videos for research in sarcasm detection from TV shows.
6. IRFL [88]: 6,697 matching images and figurative captions (rather than literal captions).
Together, these datasets cover seven different modalities from the healthcare, affective computing, and multimedia research areas and total more than 84,000 data points. For MIMIC with tabular and medical sensor inputs, we train self-supervised CL models on top of raw modality inputs. For IRFL with image and caption inputs, we start with a pretrained CLIP model [61] and perform continued pre-training to update CLIP weights with our FACTORCL objectives, before linear classifier testing. For the remaining four video datasets, we train self-supervised CL models starting from standard pre-extracted text, video, and audio features [45]. Please refer to Appendix D.2 for experimental details. We release our code and models at https://github.com/pliang279/FactorCL.
Table 2: Results on MultiBench [45] datasets with varying shared and unique information: FACTORCL achieves strong results vs self-supervised (top 5 rows) and supervised (bottom 3 rows) baselines that do not have unique representations, factorization, upper-bounds to remove irrelevant information, and multimodal augmentations.
Model	(X1; X2)	(Xi; Xi')	(X1; X2|Y)	(X2'')	Fact	MIMIC	MOSEI	MOSI	UR-FUNNY	MUSTARD
SimCLR [13]	✓	X	X	X	X	66.67%	71.03%	46.21%	50.09%	53.48%
Cross+Self [81]	✓	✓	X	X	X	65.20%	71.04%	46.92%	56.52%	53.91%
Cross+Self+Fact [89]	✓	✓	X	X	✓	65.49%	71.07%	52.37%	59.91%	53.91%
OurCL-SSL	✓	✓	✓	✓	X	65.22%	71.16%	48.98%	58.79%	53.98%
FACTORCL-SSL	✓	✓	✓	✓	✓	67.34%	74.88%	52.91%	60.50%	55.80%
SupCon [41]	X	X	✓	X	X	67.37%	72.71%	47.23%	50.98%	52.75%
OurCL-SUP	✓	✓	✓	X	X	68.16%	71.15%	65.32%	58.32%	65.05%
FACTORCL-SUP	✓	✓	✓	X	✓	76.79%	77.34%	70.69%	63.52%	69.86%
Multimodal fusion results: From Table 2, FACTORCL significantly outperforms the baselines that do not capture both shared and unique information in both supervised and self-supervised settings, particularly on MUSTARD (where unique information expresses sarcasm, such as sardonic facial expressions or ironic tone of voice), and on MIMIC (with unique health indicators and sensor readings). In Table 3, we also show that FACTORCL substantially improves the state-of-the-art in classifying images and figurative captions which are not literally descriptive of the image on IRFL, outperforming zero-shot and fine-tuned CLIP [61] as well as continued pre-training baselines on top of CLIP.
Modeling ablations: In Table 2, we also carefully ablate each component in our method and indicate either existing baselines or newly-run ablation models.
1. Factorized representations: In comparing FACTORCL-SSL with OurCL-SSL, and also FACTORCL-SUP with OurCL-SUP, we find that factorization is critical: without it, performance drops on average  $6.1\%$ , with performance drop as high as  $8.6\%$  for MIMIC.
2. Information removal via upper bound: By comparing FACTORCL with SimCLR, Cross+Self, and Cross+Self+Fact, and SupCon that only seek to capture task-relevant information via contrastive lower bounds on MI, we find that separately modeling the task-relevant information (to be captured) and task-irrelevant information (to be removed) is helpful. Without removing task-irrelevant information via the upper-bound objective, performance drops on average  $13.6\%$ , with performance drops as high as  $23.5\%$  for the MOSI dataset. We also found that training was more difficult without this objective, which is expected due to overwhelming superfluous information from the dataset [93].
3. Multimodal augmentations: Finally, we investigate the differences between separate unimodal augmentations (FACTORCL-IndAug in Table 3) versus a joint multimodal augmentation (FACTORCL-SSL) on the IRFL dataset. We choose this dataset since its images and captions are the easiest to visualize (see Figure 4 for augmentations from both strategies). In the self-supervised setting, we find that multimodal augmentations achieve  $95\%$  performance, higher than the  $92\%$  for separate unimodal augmentations, and both outperform baselines SimCLR and Cross+Self.
Ablations on  $S, U_{1}$  and  $U_{2}$ : In Table 4, we also test FACTORCL when training linear classifiers on top of only shared  $\{Z_{S_1}, Z_{S_2}\}$  and unique  $Z_{U_1}, Z_{U_2}$  separately. We call these models FACTORCL-S, FACTORCL-U1, and FACTORCL-U2. Immediately, we observe that performance drops as compared to the full FACTORCL model, indicating that both shared and unique information are critical in real-world multimodal tasks. As expected, the best-performing submodel is the one that captures the region with the largest amount of task-relevant information: MOSEI and MOSI are known to include a lot of redundancy and unique information since language is very important for detecting sentiment [93], so FACTORCL-S and FACTORCL-U2 perform best. For sarcasm detection on MUS-TARD, video information is most important with FACTORCL-
Table 3: Continued pre-training on CLIP with our FACTORCL objectives on classifying images and figurative language.
Task	IRFL
Zero-shot CLIP [61]	89.15%
SimCLR [13]	91.57%
Cross+Self [81, 89]	95.18%
FACTORCL-IndAug	92.77%
FACTORCL-SSL	95.18%
Fine-tuned CLIP [61]	96.39%
SupCon [41]	89.16%
FACTORCL-SUP	98.80%
$U_{1}$  performing best  $(59.4\%)$ , and ablation models are also the furthest away from full multimodal performance  $(69.9\%)$ . This is aligned with intuition where sarcasm is expressed through tone of voice and visual gestures (high  $U_{1}$ ), as well as from contradictions between language and video (higher multimodal performance).
Table 4: We ablate using only shared representations  $\{Z_{S_1}, Z_{S_2}\}$ , unique representation  $Z_{U_1}$ , and  $Z_{U_2}$  separately for prediction. Both shared and unique information are critical in real-world multimodal tasks.
Model	MIMIC	MOSEI	MOSI	UR-FUNNY	MUSTARD
FACTORCL-S	63.77%	77.17%	70.12%	63.42%	57.25%
FACTORCL-U1	55.90%	77.06%	70.11%	62.00%	59.42%
FACTORCL-U2	69.08%	71.01%	52.33%	54.35%	53.62%
FACTORCL-SUP	76.79%	77.34%	70.69%	63.52%	69.86%
Additional results: In Appendix D.3, we also verify FACTORCL in settings with abundant shared information, where we expect to recover the same performance as standard CL [13, 58, 72].
Contrastive learning is a successful self-supervised learning paradigm for computer vision [11, 13, 14, 25, 28, 58], natural language [24, 54, 56], speech [5, 58, 63], and multimodal tasks [1, 37, 61]. Its foundational underpinnings are inspired by work in multiview information theory [23, 41, 70, 72, 76] studying the shared information between two views and whether they are necessary or sufficient in predicting the label. Recently, Wang et al. [81] and Kahana and Hoshen [39] discuss the limitations of assuming multiview redundancy and propose autoencoder reconstruction or unimodal contrastive learning to retain unique information, which resembles the Cross+self baselines in our experiments. We refer the reader to Shwartz-Ziv and LeCun [67] for a comprehensive review on multiview and contrastive learning. Our work also relates to conditional contrastive learning [17, 51, 78, 87], where positive or negative pairs are supposed to sample from conditional distributions.
Multimodal contrastive learning aims to align related data from different modalities, typically provided as positive pairs. This could be done via optimizing a contrastive objective for intermodality pairs [1, 2, 37, 61], or both intra- and inter-modality data pairs [33, 36, 42, 44, 89]. Our work also relates to factorized representation learning, which primarily studies how to capture modality-specific information primarily in each modality and multimodal information redundant in both modalities [32, 75]. Prior work has used disentangled latent variable models [8, 30, 32, 75], mixture-of-experts [66], or product-of-experts [83] layer to explain factors in multimodal data.
Information theory [18, 65] has been used to study several phenomena in multimodal learning, including co-learning [62, 92] and multi-view learning [34, 76]. Due to its theoretical importance, several lower and upper bounds have been proposed for practical estimation [58-60, 84]. We build on the CLUB upper bound [16] to create a more accurate and stable bound. Our characterizations of shared and unique information are also related to partial information decomposition [82], co-information [7, 80], interaction information [53], and cross-domain disentanglement [35] research.
This paper studied how standard CL methods suffer when task-relevant information lies in regions unique to each modality, which is extremely common in real-world applications such as sensor placement, medical testing, and multimodal interaction. In response, we proposed FACTORCL, a new method expanding CL techniques through the use of factorized representations, removing task-irrelevant information via upper bounds on MI, and multimodal data augmentations suitable for approximating the unobserved task. Based on FACTORCL's strong performance, there are several exciting directions in extending these ideas for masked and non-contrastive pre-training; we further discuss broader impacts and limitations of this line of work in Appendix A.
This material is based upon work partially supported by Meta, National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH125740, R01MH132225, R01MH096951 and R21MH130767. PPL is supported in part by a Siebel Scholarship and a Waibel Presidential Fellowship. RS is supported in part by ONR grant N000142312368 and DARPA FA87502321015. One of the aims of this project is to understand the comfort zone of people for better privacy and integrity. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred. Finally, we would also like to acknowledge feedback from anonymous reviewers who significantly improved the paper and NVIDIA's GPU support.
Multimodal data and self-supervised models are ubiquitous in a range of real-world applications. This paper is our attempt at broadening the applicability of self-supervised contrastive methods to a wider range of multimodal tasks beyond those that exhibit multi-view redundancy. We believe that special care must be taken to ensure that these models are safely deployed for real-world benefit:
Time and space complexity: Modern multimodal models are large and take up a significant amount of carbon footprint during training and testing. As compared to heuristic combinations of cross-modal and single-modality CL [33, 36, 44, 64, 81, 85, 89], we believe that FACTORCL does not significantly increase complexity: (1) upper bounds on MI can be estimated "for free" by directly plugging in the optimal critic from  $I_{\mathrm{NCE}}$ , and (2) removal of task-irrelevant information via  $I(X_1; X_2|X_1', X_2')$  shares encoders with  $I_{\mathrm{NCE}}$ , and (3) separate unimodal augmentations perform well enough in practice. We also release our code and models so that they can be evaluated quickly on new tasks, which can amortize complexity costs.