SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Privacy and security: There may be privacy risks associated with making predictions from multimodal data of recorded human behaviors and medical data (i.e., the datasets used in our experiments for analysis of sentiment, emotions, personality, sarcasm, and humor, as well as disease prediction from medical data). We have followed best practices in maintaining the privacy and safety of these datasets: (1) the creators of these video datasets have taken the appropriate steps to only access public data that participants or content creators have consented for public release (creative commons license and following fair use guidelines of YouTube) [12, 27, 93], (2) MIMIC has been rigorously de-identified in accordance with Health Insurance Portability and Accountability Act (HIPAA) such that all possible personal information has been removed from the dataset [38], (3) all video data was also anonymized and stripped of all personal (e.g., personally identifiable information) and protected attributes (e.g., race, gender), (4) all models trained on affect recognition datasets use only pre-extracted non-invertible features that rely on general visual or audio features such as the presence of a smile or magnitude of voice which cannot be used to identify the speaker [91, 93], and (5) we studied the videos collected in these affective computing datasets and found no offensive words used or personal attacks recorded in the video. Finally, we only use these datasets for research purposes and emphasize that any multimodal models trained to perform prediction should only be used for scientific study and should not in any way be used for real-world harm.
Social biases: We acknowledge risks of social bias due to imbalanced datasets, resulting in potential biases surrounding gender, race, and ethnicity, among others [9, 46]. We note that our FACTORCL approach has a close link with conditional CL [51], which can also be adapted to condition on sensitive attributes and therefore reduce bias. Studying these research questions is an important direction for future work.
Future work: We discuss more limitations and potential future work in this direction. Firstly, optimizing our objectives using better MI lower and upper bounds such as in Guo et al. [26] and Sordoni et al. [69], could improve the performance for inputs of higher dimension and complex modality. Next, the current data augmentation method requires one to pick augmentations to approximately satisfy Definition 4; future work could extend InfoMin [72] to automatically generate data augmentations to satisfy Definition 4, or leverage future progress in multimodal generative models for data augmentation. Lastly, future work could quantify whether shared or unique information is more important for different tasks and reweight the terms in the FACTORCL objective to suit the tasks.
B Analysis of Multi-view Contrastive Learning
Multi-view shared information describes the extent and dimensions in which information can be shared across different views. The presence of shared information is often in contrast to unique information that exists solely in a single modality, and can be formalized via information theory:
Definition 5. (Shared information) Given  $X_{1}$  and  $X_{2}$ ,  $I(X_{1};X_{2}) = \int p(x_{1},x_{2})\log \frac{p(x_{1},x_{2})}{p(x_{1})p(x_{2})}$  measures the degree of information-theoretic shared information between  $X_{1}$  and  $X_{2}$ .
Definition 6. (Task-relevant shared information) Given  $X_1$ ,  $X_2$ , and a target  $Y$ ,  $I(X_1;X_2;Y) = I(X_1;X_2) - I(X_1;X_2|Y) = \int p(x_1,x_2)\log \frac{p(x_1,x_2)}{p(x_1)p(x_2)} -\int p(x_1,x_2|y)\log \frac{p(x_1,x_2|y)}{p(x_1|y)p(x_2|y)}$  measures
the amount of task-relevant shared information between  $X_{1}$  and  $X_{2}$  for predicting  $Y$ .  $I(X_{1};X_{2}|Y)$  represents the task-irrelevant shared information.
Learning shared information via contrastive learning: Current approaches for multi-view contrastive learning model shared information  $I(X_{1};X_{2})$  (and subsequently task-relevant shared information  $I(X_{1};\bar{X}_{2};Y)$  during downstream task fine-tuning), without modeling unique information.
$$
Z _ {1} = \underset {Z _ {1} := f _ {\theta} \left(X _ {1}\right)} {\arg \max } I \left(Z _ {1}; X _ {2}\right), Z _ {2} = \underset {Z _ {2} := f _ {\theta} \left(X _ {2}\right)} {\arg \max } I \left(X _ {1}; Z _ {2}\right). \tag {19}
$$
Optimizing for  $I(X_{1};X_{2})$  is performed via a surrogate loss during self-supervised pre-training (where we do not have access to the label  $Y$ ) by maximizing the InfoNCE objective:
$$
\operatorname {I N F O N C E} = \sup  _ {f} \mathbb {E} _ {\substack {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2}\right) \\ x _ {2} ^ {-} \sim p \left(x _ {2}\right)}} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+}\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-}\right)} \right], \tag{20}
$$
Oord et al. [58] show that  $I(X_{1};X_{2}) \geq \log k - \mathcal{L}_{\mathrm{NCE}}(X_{1};X_{2})$  where  $\mathcal{L}_{\mathrm{NCE}}(X_1;X_2)$  is negative of INFONCE and is the loss to minimize (rather than maximize) in training. NCE falls into a broader class of contrastive learning methods [13, 15, 28, 41, 61] that model the ratio between joint densities  $p(x_{1},x_{2})$  and product of marginal densities  $p(x_{1})p(x_{2})$  using positive and negative samples [57, 59, 60, 79, 84] or probabilistic classifiers [55, 77], all of which can also be used to capture shared information.
Tian et al. [71] argues that the optimal view of contrastive learning is also minimal: the minimal representations only extract relevant information of the contrastive task (maximizing the shared part) and throw away other information. Therefore, from this minimal assumption, we have  $I(Z_{1};Y|X_{2}) = 0$  and  $I(Z_{2};Y|X_{1}) = 0$  as minimal  $Z_{1}$  and  $Z_{2}$  only captures task-relevant information from the shared part. By conditioning on  $X_{1}$  or  $X_{2}$ , the shared part is removed, and  $Z_{1}$  and  $Y$  (or  $Z_{2}$  and  $Y$ ) do not share information.
Lastly, we restate the multi-view non-redundancy from Definition 2:
Definition 7. (Multi-view non-redundancy)  $\exists \epsilon >0$  such that  $I(X_{1};Y|X_{2}) > \epsilon$  or  $I(X_{2};Y|X_{1}) > \epsilon$ .
We would like to compare and clarify the differences between the multiview redundancy assumption in Eq.(1) and the multi-view nonredundancy in Def. 7. The multiview redundancy assumption in Eq.(1) states that the task-relevant information from the unique part is minimal  $(\leq \epsilon)$ . The multiview non-redundancy states the opposite: the task-relevant information from the unique part is nonzero and nonminimal, as it is not bounded by  $\epsilon$ . Next we briefly clarify the relationship between these two assumptions and the InfoMin assumption:  $I(Z_{1};Y|X_{2}) = I(Z_{2};Y|X_{1}) = 0$ . InfoMin is about representation  $Z$  while the redundancy assumptions are only about data  $X$ . InfoMin states that the optimal (sufficient and minimal) representation learns task-relevant information only from the shared part, as we discussed in the paragraph above. We empirically checked the two assumptions: Tables 1 and 4 in the main text show that the multiview non-redundancy assumption holds empirically, and Table 11 shows that the InfoMin assumption holds empirically.
We now show the limitations of CL methods, first restating the Theorem here:
Theorem 4. (Suboptimality of standard CL) When there is multi-view non-redundancy as in Definition 7, given optimal representations  $\{Z_1,Z_2\}$  that satisfy Eq.(19 and  $I(Z_{1};Y|X_{2}) = I(Z_{2};Y|X_{1}) = 0$  [72], we have that
$$
I \left(Z _ {1}, Z _ {2}; Y\right) = I \left(X _ {1}, X _ {2}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) = I \left(X _ {1}; X _ {2}\right) - I \left(X _ {1}; X _ {2} \mid Y\right) <   I \left(X _ {1}, X _ {2}; Y\right). \tag {21}
$$
Proof. Since  $Z_{1}$  and  $Z_{2}$  maximize  $I(X_{1};X_{2})$  we have that  $I(Z_{1};X_{2}) = I(X_{1};Z_{2}) = I(X_{1};X_{2})$  so  $I(Z_{1};X_{2};Y) = I(X_{1};Z_{2};Y) = I(X_{1};X_{2};Y)$  and  $I(Z_{1};X_{2}|Y) = I(X_{1};Z_{2}|Y) = I(X_{1};X_{2}|Y)$ .
We now show the relationship between  $I(X_{1},X_{2};Y)$ , which is the total information  $X_{1},X_{2}$  contributes towards predicting  $Y$  in classic supervised learning, with  $I(Z_{1},Z_{2};Y)$ , which is the information that our learned self-supervised representations can contribute towards  $Y$  during supervised fine-tuning. We first derive the relationship between  $I(Z_{1};Y)$  and  $I(X_{1};Y)$ :
$$
\begin{array}{l} I \left(Z _ {1}; Y\right) = I \left(Z _ {1}; X _ {2}; Y\right) + I \left(Z _ {1}; Y \mid X _ {2}\right) (22) \\ = I \left(X _ {1}; X _ {2}; Y\right) + I \left(Z _ {1}; Y \mid X _ {2}\right) (23) \\ = I \left(X _ {1}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) + I \left(Z _ {1}; Y \mid X _ {2}\right) (24) \\ = I \left(X _ {1}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) (25) \\ \end{array}
$$
Given  $X_{1}$ , we further derive a relationship between  $I(Z_{2};Y|Z_{1})$  and  $I(X_{2};Y|X_{1})$ :
$$
\begin{array}{l} I \left(Z _ {2}; Y \mid Z _ {1}\right) = I \left(Z _ {2}; X _ {1}; Y \mid Z _ {1}\right) + I \left(Z _ {2}; Y \mid Z _ {1}, X _ {1}\right) (26) \\ = I \left(X _ {1}; X _ {2}; Y \mid Z _ {1}\right) + I \left(Z _ {2}; Y \mid Z _ {1}, X _ {1}\right) (27) \\ = I \left(X _ {1}; X _ {2}; Y \mid Z _ {1}\right) + I \left(Z _ {2}; Y \mid X _ {1}\right) (28) \\ = I \left(X _ {2}; Y \mid Z _ {1}\right) - I \left(X _ {2}; Y \mid X _ {1}, Z _ {1}\right) + I \left(Z _ {2}; Y \mid X _ {1}\right) (29) \\ = I \left(X _ {2}; Y \mid Z _ {1}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) + I \left(Z _ {2}; Y \mid X _ {1}\right) (30) \\ = I \left(X _ {2}; Y\right) - I \left(Z _ {1}; X _ {2}; Y\right) - I \left(X _ {2}; Y \mid X _ {1}\right) + I \left(Z _ {2}; Y \mid X _ {1}\right) (31) \\ = I \left(X _ {2}; Y\right) - I \left(X _ {1}; X _ {2}; Y\right) - I \left(X _ {2}; Y \mid X _ {1}\right) + I \left(Z _ {2}; Y \mid X _ {1}\right) (32) \\ = I \left(X _ {2}; Y \mid X _ {1}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) + I \left(Z _ {2}; Y \mid X _ {1}\right) = 0 (33) \\ \end{array}
$$
In Eqs.(28) and (30), we use the fact that conditioning on  $Z_{1}$  and  $X_{1}$  jointly reduces to conditioning on  $X_{1}$  since  $Z_{1}$  is deterministically obtained from  $X_{1}$ , and in Eq.(32) we use the definition of learning  $Z_{s}$  to maximize  $I(X_{1};X_{2})$  so  $I(Z_{1};X_{2};Y) = I(X_{1};Z_{2};Y)$ . Finally, adding both terms up,
$$
\begin{array}{l} I \left(Z _ {1}, Z _ {2}; Y\right) = I \left(Z _ {1}; Y\right) + I \left(Z _ {2}; Y \mid Z _ {1}\right) (34) \\ = I \left(X _ {1}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) (35) \\ = I \left(X _ {1}; X _ {2}; Y\right) (36) \\ = I \left(X _ {1}, X _ {2}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) (37) \\ = I \left(X _ {1}; X _ {2}\right) - I \left(X _ {1}; X _ {2} \mid Y\right) (38) \\ \end{array}
$$
givesthe desired result.
Bayes error rate. The Bayes error rate  $P_{e}(Z_{1}, Z_{2}) \coloneqq 1 - \mathbb{E}_{P_{Z_{1}, Z_{2}}} \left[ \max_{y \in Y} P(\hat{Y} = y \mid z_{1}, z_{2}) \right]$  of contrastive representations  $\{Z_{1}, Z_{2}\}$  is given by:
$$
\begin{array}{l} P _ {e} \leq 1 - \exp \left[ I \left(X _ {1}, X _ {2}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) - H (Y) \right] (39) \\ = 1 - \exp \left[ I \left(X _ {1}; X _ {2}; Y\right) - H (Y) \right] (40) \\ \end{array}
$$
Proof. We use the inequality between  $P_{e}$  and  $H(Y|Z)$  [22, 76, 81]:
$$
- \ln (1 - P _ {e}) \leq H (Y | Z), \text {o r e q u i v a l e n t l y ,} P _ {e} \leq 1 - \exp [ - H (Y | Z) ] \tag {41}
$$
If we regard  $Z$  as the joint of  $Z_{1}$  and  $Z_{2}$ , then we have
$$
P _ {e} \leq 1 - \exp \left[ - H \left(Y \mid Z _ {1}, Z _ {2}\right) \right] \tag {42}
$$
We further expand  $H(Y|Z_1,Z_2)$  by definition of mutual information,  $I(X;Y) = H(X) - H(X|Y)$ , Theorem 4, and the  $I(X_{1};X_{2};Y) = I(X_{1};X_{2}) - I(X_{1};X_{2}|Y)$ :
$$
\begin{array}{l} H \left(Y \mid Z _ {1}, Z _ {2}\right) = H (Y) - I \left(Z _ {1}, Z _ {2}; Y\right) (43) \\ = H (Y) - I \left(X _ {1}, X _ {2}; Y\right) + I \left(X _ {1}; Y \mid X _ {2}\right) + I \left(X _ {2}; Y \mid X _ {1}\right) (44) \\ = H (Y) - I \left(X _ {1}; X _ {2}\right) + I \left(X _ {1}; X _ {2} \mid Y\right) (45) \\ = H (Y) - I \left(X _ {1}; X _ {2}; Y\right) (46) \\ \end{array}
$$
Plugging in Eq.(42), we have
$$
\begin{array}{l} P _ {e} \leq 1 - \exp \left[ - H \left(Y \mid Z _ {1}, Z _ {2}\right) \right] (47) \\ = 1 - \exp \left[ - \left(H (Y) - I \left(X _ {1}, X _ {2}; Y\right) + I \left(X _ {1}; Y \mid X _ {2}\right) + I \left(X _ {2}; Y \mid X _ {1}\right)\right) \right] (48) \\ = 1 - \exp \left[ - H (Y) + I \left(X _ {1}, X _ {2}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) \right] (49) \\ \end{array}
$$
$$
\begin{array}{l} P _ {e} \leq 1 - \exp \left[ - H \left(Y \mid Z _ {1}, Z _ {2}\right) \right] (50) \\ = 1 - \exp \left[ - \left(H (Y) - I \left(X _ {1}; X _ {2}; Y\right)\right) \right] (51) \\ = 1 - \exp \left[ - H (Y) + I \left(X _ {1}; X _ {2}; Y\right) \right] (52) \\ \end{array}
$$
resulting in the Bayes error rate as desired.
C FACTORIZED CONTRASTIVE LEARNING
C.1 Contrastive estimators
Theorem 5. (Contrastive estimators for  $I(X_1;X_2)$ ) Defining the NCE estimator and NCE-CLUB estimator as follows,
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) = \mathbb {E} _ {\substack {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2}\right) \\ x _ {2} ^ {-} \sim p \left(x _ {2}\right)}} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+}\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-}\right)} \right] \tag{53}
$$
$$
I _ {\text {N C E - C L U B}} \left(X _ {1}; X _ {2}\right) = \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2}\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {+}\right) \right] - \mathbb {E} _ {\substack {x _ {1} \sim p \left(x _ {1}\right) \\ x _ {2} ^ {-} \sim p \left(x _ {2}\right)}} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {-}\right) \right] \tag{54}
$$
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) \leq I \left(X _ {1}; X _ {2}\right) \leq I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2}\right). \tag {55}
$$
Proof. The lower bound  $I_{\mathrm{NCE}}(X_1;X_2) \leq I(X_1;X_2)$  follows from Oord et al. [58]: optimizing the objective leads to an optimal critic  $f^{*} = \log p(x_{2}|x_{1}) + c(x_{2})$  [60] or without loss of generality  $f^{*} = \log p(x_{1}|x_{2}) + c(x_{1})$ , where  $c(\cdot)$  is an arbitrary deterministic function. Plugging the optimal critic into the  $I_{\mathrm{NCE}}(X_1;X_2)$  gives the result:  $I_{\mathrm{NCE}}(X_1;X_2) + \log N \leq I(X_1;X_2)$  [58, 60].
Next, the original  $I_{\mathrm{CLUB}}(X_1;X_2)$  [16] is defined as:
$$
I _ {\text {C L U B}} \left(X _ {1}; X _ {2}\right) := \mathbb {E} _ {p \left(x _ {1}, x _ {2}\right)} \left[ \log p \left(x _ {2} \mid x _ {1}\right) \right] - \mathbb {E} _ {p \left(x _ {1}\right) p \left(x _ {2}\right)} \left[ \log p \left(x _ {2} \mid x _ {1}\right) \right]. \tag {56}
$$
As mutual information is symmetric w.r.t  $x_{1}$  and  $x_{2}$ :  $I(X_{1};X_{2}) = I(X_{2};X_{1})$ , without loss of generality, we have:
$$
I _ {\text {C L U B}} \left(X _ {1}; X _ {2}\right) = I _ {\text {C L U B}} \left(X _ {2}; X _ {1}\right) = \mathbb {E} _ {p \left(x _ {1}, x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) \right] - \mathbb {E} _ {p \left(x _ {1}\right) p \left(x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) \right] \tag {57}
$$
The formulation above assumes  $p(x_1|x_2)$  is known, which is satisfied when we use the optimal critic  $f^{*} = \log p(x_{1}|x_{2}) + c(x_{1})$  from  $I_{\mathrm{NCE}}(X_1;X_2)$ . Plugging the optimal critic  $f^{*}$  into  $I_{\mathrm{CLUB}}(X_1;X_2)$ , we obtain a desired upper bound  $I_{\mathrm{NCE - CLUB}}(X_1;X_2)$  of  $I(X_{1};X_{2})$ :
$$
\begin{array}{l} I _ {\text {N C E - C L U B}} \left(X _ {1}; X _ {2}\right) = \mathbb {E} _ {p \left(x _ {1}, x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) + c \left(x _ {1}\right) \right] - \mathbb {E} _ {p \left(x _ {1}\right) p \left(x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) + c \left(x _ {1}\right) \right] (58) \\ = \mathbb {E} _ {p \left(x _ {1}, x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) \right] + \mathbb {E} _ {p \left(x _ {1}, x _ {2}\right)} \left[ c \left(x _ {1}\right) \right] - \mathbb {E} _ {p \left(x _ {1}\right) p \left(x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) \right] - \mathbb {E} _ {p \left(x _ {1}\right) p \left(x _ {2}\right)} \left[ c \left(x _ {1}\right) \right] (59) \\ = \mathbb {E} _ {p \left(x _ {1}, x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) \right] - \mathbb {E} _ {p \left(x _ {1}\right) p \left(x _ {2}\right)} \left[ \log p \left(x _ {1} \mid x _ {2}\right) \right] (60) \\ \geq I \left(X _ {1}; X _ {2}\right). (61) \\ \end{array}
$$
Eq.(59) is from the linearity of expectation, Eq.(60) is from the fact that  $c(x_1)$  is not a function of  $x_2$  and therefore  $\mathbb{E}_{p(x_1,x_2)}[c(x_1)] = \mathbb{E}_{p(x_1)p(x_2)}[c(x_1)] = \mathbb{E}_{p(x_1)}[c(x_1)]$ , and Eq.(61) is directly from the original  $I_{\mathrm{CLUB}}(X_1;X_2)$  paper [16].
C.2 Unimodal and multimodal augmentations
We first restate the definitions of optimal single-view and multi-view augmentation:
Definition 8. (Optimal unimodal augmentation)  $X_1'$  is an optimal unimodal augmentation for  $X_1$  when  $I(X;X') = I(X;Y)$ , which implies that the only information shared between  $X$  and  $X'$  is task-relevant with no irrelevant noise.
Definition 9. (Optimal multimodal augmentation)  $X_1'$  and  $X_2'$  are optimal multimodal augmentation for  $X_1$  and  $X_2$  when  $I(X_1, X_2; X_1', X_2') = I(X_1, X_2; Y)$ , which implies that the only information shared between  $X_1, X_2$  and  $X_1', X_2'$  is task-relevant with no irrelevant noise.
When are these assumptions satisfied?  $I(X;X^{\prime}) = I(X;Y)$  holds when all information shared between  $X$  and  $X^{\prime}$  is task-relevant, which implies that the augmentation keeps task-relevant information constant while changing task-irrelevant information. In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background. To satisfy  $I(X_{1},X_{2};X_{1}^{\prime},X_{2}^{\prime}) = I(X_{1},X_{2};Y)$ , by the chain rule of MI, we augment in two steps:
$$
\text {U n i m o d a l} X _ {1} ^ {\prime} \text {s . t .} I \left(X _ {1}; X _ {1} ^ {\prime}\right) = I \left(X _ {1}; Y\right), \tag {62}
$$
$$
\text {U n i q u e} X _ {2} ^ {\prime} \text {s . t .} I \left(X _ {2}; X _ {2} ^ {\prime} \mid X _ {1}\right) = I \left(X _ {2}; Y \mid X _ {1}\right). \tag {63}
$$
Table 5: More examples of optimal single-view and multi-view augmentations.
Dataset	X1	X2	X'1	Standard Aug	Unique Aug
X'2	X''2
Cartoon	Caption	Image	Word Masking	Crop + Flip + Resize	Flip + Resize
MIMIC	Signals	Tables	Time Warping	CutMix [90] on All Features	CutMix on Nonclinical Features
MOSEI	Transcripts	Video+Audio	Word Masking	Noise Injection on Any Frames	Noise Injection on Silent Frames
UR-FUNNY	Transcripts	Video+Audio	Word Masking	Noise Injection on Any Frames	Noise Injection on Silent Frames
MUsTARD	Transcripts	Video+Audio	Word Masking	Noise Injection on Any Frames	Noise Injection on Silent Frames
the second step is the unique augmentation: after observing  $X_{1}$ , we create augmented  $X_{2}^{\prime}$  from  $X_{2}$  to keep the task-relevant information but meanwhile do not affect any information from  $X_{1}$ . In Table 5, we include some more examples of how unique augmentations could be designed across different datasets.
Final objectives: If Definitions 8 and 9 are both satisfied, we can substitute contrastive estimators in the following equations:
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} | Y\right) = \mathbb {E} _ {p (y)} \left[ \mathbb {E} _ {x _ {1}, \underset {x _ {2} \sim p (x _ {2} | y)} {x _ {2} ^ {+} \sim p (x _ {1}, x _ {2} | y)}} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+} , y\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-} , y\right)} \right] \right] \tag {64}
$$
$$
I _ {\text {N C E - C L U B}} \left(X _ {1}; X _ {2} | Y\right) = \mathbb {E} _ {p (y)} \left[ \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2} \mid y\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {+}, y\right) \right] - \mathbb {E} _ {x _ {1} \sim p \left(x _ {1} \mid y\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {-}, y\right) \right] \right] \tag {65}
$$
by replacing  $I(X_{i};Y)$  terms with  $I(X_{i};X_{i}^{\prime})$  and replacing  $I(X_{1};X_{2}|Y)$  terms with  $I(X_{1};X_{2}|X_{1}^{\prime},X_{2}^{\prime})$ :
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) = \mathbb {E} _ {p \left(x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ \mathbb {E} _ {\substack {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \\ x _ {2} ^ {-} \sim p \left(x _ {2} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)}} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+} , x _ {1} ^ {\prime} , x _ {2} ^ {\prime}\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-} , x _ {1} ^ {\prime} , x _ {2} ^ {\prime}\right)} \right] \right] \tag{66}
$$
$$
\begin{array}{l} I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2} | X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) = \mathbb {E} _ {p \left(x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2} | x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {+}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \right] \right. \\ \left. - \mathbb {E} _ {\substack {x _ {1} \sim p \left(x _ {1} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \\ x _ {2} \sim p \left(x _ {2} \mid x _ {1} ^ {\prime}, x _ {2}\right)}} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {-}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \right] \right] \tag{67} \\ \end{array}
$$
C.2.1 Implementing conditional CL via kernel
We restate our objectives below:
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) = \mathbb {E} _ {p \left(x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ \mathbb {E} _ {x _ {1}, x _ {2} ^ {\prime} \sim p \left(x _ {1}, x _ {2} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+} , x _ {1} ^ {\prime} , x _ {2} ^ {\prime}\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-} , x _ {1} ^ {\prime} , x _ {2} ^ {\prime}\right)} \right] \right] \tag {68}
$$
$$
\begin{array}{l} I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2} | X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) = \mathbb {E} _ {p \left(x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2} | x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {+}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \right] \right. \\ \left. - \mathbb {E} _ {\substack {x _ {1} \sim p \left(x _ {1} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \\ x _ {2} \sim p \left(x _ {2} \mid x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)}} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {-}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right) \right] \right] \tag{69} \\ \end{array}
$$