SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Synthetic dataset with controlled generation: We generate data with controllable ratios of task-relevant shared and unique information to analyze the behavior of each contrastive learning objective in Figure 1 in the main text. Starting with a set of latent vectors  $w_{1}, w_{2}, w_{s} \sim \mathcal{N}(0_{d}, \Sigma_{d}^{2})$ ,  $d = 50$  representing information unique to  $X_{1}, X_{2}$  and common to both respectively, the concatenated vector  $[w_{1}, w_{s}]$  is transformed into high-dimensional  $x_{1}$  using a fixed full-rank transformation  $T_{1}$  and likewise  $[w_{2}, w_{s}]$  to  $x_{2}$  via  $T_{2}$ . The label  $y$  is generated as a function (with nonlinearity and noise) of varying ratios of  $w_{s}$ ,  $w_{1}$ , and  $w_{2}$  to represent shared and unique task-relevant information. For experiments, we used 1-layer MLPs with 512 hidden size as encoders, and the embedding dimensions are 128 for both modalities. The heads on top of encoders are also 1-layer MLPs with the same hidden and output dimension as the input, and all critics are 1-layer MLPs with 512 hidden size.
Multimodal fusion datasets: We use a collection of 5 real-world datasets provided in Multi-Bench [45] and the IRFL dataset to test our method in the context of varying ratios of shared and unique information that is important for the task. In all the datasets below, the heads added on top of the encoders are 1-Layer MLPs with ReLU activations that map the encoder outputs to the same dimensions. All critics are also MLPs with 1 hidden layer of size 512 and ReLU activation.
1. MIMIC-III [38] (Medical Information Mart for Intensive Care III) is a large-scale dataset for healthcare which contains records of over 40,000 ICU patients in both forms of times-series data measured by hours and static data (age, gender, ethnicity) in the tabular numerical form. We use the preprocessed data provided in MultiBench [45], where the time-series data is measured every 1 hour in a 24-hour period and consists of vectors of size 12, and the tabular data consists of vectors of size 5. The task we use in the experiment is a binary classification on whether the patient fits any ICD-9 code in group 7 (460-519).
Table 6: Results on MultiBench [45] datasets with varying shared and unique information: FACTORCL achieves strong results vs self-supervised (top 5 rows) and supervised (bottom 3 rows) baselines that do not have unique representations, factorization, upper-bounds to remove irrelevant information, and multimodal augmentations.
Model	MIMIC	MOSEI	MOSI	UR-FUNNY	MUSTARD
SimCLR [13]	66.7 ± 0.0%	71.9 ± 0.3%	47.8 ± 1.8%	50.1 ± 1.9%	53.5 ± 2.9%
Cross+Self [81]	65.2 ± 0.0%	71.1 ± 0.2%	48.6 ± 1.2%	56.5 ± 0.7%	53.9 ± 4.5%
Cross+Self+Fact [89]	65.5 ± 0.0%	71.9 ± 0.2%	49.0 ± 1.1%	59.9 ± 0.9%	53.9 ± 4.0%
OurCL-SSL	65.2 ± 0.0%	71.2 ± 0.2%	49.0 ± 0.8%	58.8 ± 1.3%	54.0 ± 2.5%
FACTORCL-SSL	67.3 ± 0.0%	74.5 ± 0.1%	51.2 ± 1.6%	60.5 ± 0.8%	55.8 ± 0.9%
SupCon [41]	67.4 ± 0.0%	71.0 ± 0.1%	47.2 ± 1.2%	50.1 ± 2.0%	52.7 ± 2.2%
OurCL-SUP	68.2 ± 0.0%	71.1 ± 0.2%	65.3 ± 0.8%	58.3 ± 1.1%	65.1 ± 1.8%
FACTORCL-SUP	76.8 ± 0.0%	77.8 ± 0.3%	69.1 ± 0.6%	63.5 ± 0.8%	69.9 ± 1.9%
Table 7: Continued pre-training on CLIP with our FACTORCL objectives on classifying images and figurative language. Our approach shows strong results as compared to standard fine-tuning and continued pre-training.
Task	IRFL
Zero-shot CLIP [61]	89.2 ± 0.0%
SimCLR [13]	91.6 ± 0.0%
Cross+Self [81, 89]	91.1 ± 1.2%
FACTORCL-IndAug	91.6 ± 1.3%
FACTORCL-SSL	93.8 ± 1.4%
Fine-tuned CLIP [61]	96.4 ± 0.0%
SupCon [41]	87.7 ± 4.7%
FACTORCL-SUP	98.3 ± 1.2%
In the experiments, we use a 2-layer MLP with 10 hidden layer size for the tabular data modality, and map it to a vector of size 10. The time-series modality is encoded using a GRU with hidden size 30 and followed by a linear layer which projects the output to embeddings of size 15. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
2. CMU-MOSEI [93] is the largest sentence-level multimodal sentiment and emotion benchmark with 23,000 monologue videos. It contains more than 65 hours of annotated video from more than 1,000 speakers and 250 topics. Each video is labeled with a sentiment intensity ranging from -3 to 3. In our experiments, we cast the intensity values to a binary classification on whether the sentiment is positive or negative. MultiBench [45] provides access to the extracted features of the vision, text, and audio modalities, and in our experiments, we use the vision and text features for doing contrastive learning.
In our experiments, we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
3. CMU-MOSI [91] is a similar dataset for multimodal sentiment analysis created from 2, 199 YouTube videos clips. The data focuses on videos that reflect the real-world distribution of speakers expressing their opinions in the form of monologues. The sentiment intensities are labeled continuously from -3 to 3. Again we cast the label into a binary classification on whether the sentiment is positive or negative, and we used the extracted vision and text features for contrastive learning.
In our experiments we encode both the vision and text modalities using Transformer encoders with 5 heads and 5 layers, and map them to 40-dimensional embeddings. We train the model for 100 epochs using the Adam optimizer with a 1e-4 learning rate.
4. UR-FUNNY [27] is the first large-scale dataset for humor detection in human speech. The dataset consists of samples from more than 16,000 TED talk videos with speakers from diverse backgrounds sharing their ideas. The laughter markup is used to filter out 8,257 humorous punchlines from the transcripts. The context is extracted from the prior sentences to the punchline. Using a similar approach, 8,257 negative samples are chosen at random intervals where the last sentence is not immediately followed by a laughter marker. The task is to classify whether there is humor or not using the vision and text modalities.
Table 8: Additional experiments on CIFAR10 [43] and MNIST [19] datasets using our FACTORCL objectives on image classification.
Task	CIFAR10	MNIST
SimCLR [13]	87.0%	98.84%
SupCon [41]	92.7%	99.38%
FACTORCL-SUP	91.3%	99.21%
Table 9: We verify our conditional lower and upper bound estimators on a synthetic dataset with fixed dimension of representation  $d_{z}$  and varying number of samples  $n$ .
Number of samples (×103), dz = 20	5	10	20	50
CCMI (MI-Diff + Classifier)	2.03	2.06	2.15	2.20
Conditional InfoNCE	2.19	2.20	2.20	2.20
Conditional InfoNCE-CLUB	3.45	3.53	2.98	2.86
True CMI	2.32	2.32	2.32	2.32
5. MUSTARD [12] is a corpus of 690 videos for research in sarcasm detection from popular TV shows including Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholics Anonymous. It contains audiovisual utterances together with the textual context. We use the preprocessed features of the vision and text modalities for doing contrastive learning and performing sarcasm detection.
6. IRFL [88] is a dataset for understanding multimodal figurative languages. It contains 6,697 matching images and figurative captions (rather than literal captions) of three types of figurative languages: idiom, simile, and metaphor. The original data for the matching task is provided in the form of 1 caption, 3 distractor images, and 1 matching image. We convert it into a fusion task by only collecting the matching image and text pairs and assigning labels using the type of figurative language it belongs to.
For this dataset, we do not train from scratch. Instead, we performed continued pretraining using our proposed objectives on pretrained CLIP [61] models. We used the CLIP-VIT-B/32 model and its pretrained image and text encoders. We performed training for 10 epochs using the Adam optimizer with a 1e-6 learning rate.
D.3 Additional analysis and results
Fusion experiments: In Table 6 and 7 we present more detailed results on the Multibench [45] and IRFL [88] datasets computed from 5 independent runs. FACTORCL significantly outperforms the baselines that do not capture both shared and unique information in both supervised and self-supervised settings, particularly on MUSTARD (where unique information expresses sarcasm, such as sardonic facial expressions or ironic tone of voice), and on MIMIC (with unique health indicators and sensor readings). There are also big improvements on the two sentiment analysis datasets MOSEI and MOSI, with  $6.8\%$  and  $21.9\%$  increases respectively when compared to SupCon [41].
In Table 7, we also see that FACTORCL substantially improves the state-of-the-art in classifying images and figurative captions which are not literally descriptive of the image on IRFL, outperforming zero-shot and fine-tuned CLIP [61] as well as continued pre-training baselines on top of CLIP. While the supervised version gives the best results overall, the self-supervised version with our proposed unique augmentations also performs better than independent augmentations, indicating that in the case without label information, we should always try to find and use unique augmentations when possible. In our experiments, we use word masking for text augmentations. For independent image
Table 10: We verify our conditional lower and upper bound estimators on a synthetic dataset with varying dimensions of representation  $d_{z}$  and fixed number of samples  $n$ .
Dimension dz, n = 2 × 104	1	10	20	50	100
CCMI (MI-Diff + Classifier)	2.30	2.18	2.15	1.98	1.67
Conditional InfoNCE	2.18	2.20	2.20	2.26	2.30
Conditional InfoNCE-CLUB	3.70	2.95	2.98	2.79	2.86
True CMI	2.32	2.32	2.32	2.32	2.32
Table 11: We probe whether the InfoMin assumption from Tian et al.,  $I(Z_{1};Y|X_{2}) = 0$  and  $I(Z_{2};Y|X_{1}) = 0$ , is reasonable for Theorem 1. Compared to the shared information  $I(X_{1};X_{2})$ ,  $I(Z_{1};Y|X_{2})$  is much smaller and closer to zero, indicating that the InfoMin assumption is reasonable, and Theorem 1 holds in practice.
I(X1,Y;X2)	I(X1;X2)	I(Z1;Y|X2)	I(X2,Y;X1)	I(X2;X1)	I(Z2;Y|X1)
12.69	12.29	0.40	11.31	10.92	0.38
augmentations, we use cropping, flipping, and color jittering. The unique augmentation simply removes the cropping operation, as illustrated in Figure 4 in the main text.
Additional experiments on high shared information and low unique information: In Table 8 we include additional results using our method on the CIFAR10 [43] and MNIST [19] datasets. Our method outperforms the self-supervised contrastive learning on both datasets as expected, and roughly maintains the same performance as supervised contrastive learning. Therefore, in cases with abundant shared information (two modalities with high shared information or two different views generated from augmentations), our method recovers the performance of existing methods that do not capture unique information.
Experiments on CMI estimator verification: In Table 9 and Table 10, we include experiment results which verify that computing the conditional MI lower and upper bounds via concatenation indeed yields reliable estimates. In particular, we aim to verify that the Conditional InfoNCE objective gives a lower bound of the CMI, and the Conditional InfoNCE-CLUB objective gives an upper bound of the CMI. We follow the experiment setups in [55], presenting the true CMI and results of our estimators on synthetic data with fixing dimension of representation  $Z$  and varying samples  $n$ , and fixing samples  $n$  and varying  $d_{z}$ . The specific implementations used for conditional InfoNCE and conditional InfoNCE-CLUB can be found in Equation 13 and Equation 14, respectively. The results indicate that our Conditional InfoNCE gives estimations smaller than the true CMI, and Conditional InfoNCE-CLUB gives estimations greater than the true CMI. The performances are comparable to estimators in [55], suggesting that our method yields valid and competitive lower and upper bounds for CMI.
Empirical verification on InfoMin assumption: To verify the InfoMin assumption [72]  $(I(Z_{1};Y|X_{2}) = I(Z_{2};Y|X_{1}) = 0)$ , we use the same synthetic dataset as in Table 1 and measure  $I(Z_{1};Y|X_{2})$ . The results are shown in Table 11: we get  $I(X_{1};X_{2}) = 12.29$  and  $I(Z_{1};Y|X_{2}) = 0.4$ .  $I(Z_{1};Y|X_{2})$  is much smaller and closer to zero than  $I(Z_{1};Y|X_{2})$ , indicating that the InfoMin assumption holds in practice.
Compute resources: All experiments in this paper are run on a single NVIDIA A100 GPU. It takes about 10 to 12 GPU hours to train the model on the CIFAR10 [43] for 300 epochs, and all the other experiments can be finished within 1 GPU hour using our specified hyperparameters.