SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FACTORIZED CONTRASTIVE LEARNING: Going Beyond Multi-view Redundancy
Paul Pu Liang $^{1*}$ , Zihao Deng $^{2*}$ , Martin Q. Ma $^{1*}$   
James Zou $^{3}$ , Louis-Philippe Morency $^{1}$ , Ruslan Salakhutdinov $^{1}$
$^{1}$ Carnegie Mellon University,  $^{2}$ University of Pennsylvania,  $^{3}$ Stanford University pliang@cs.cmu.edu, zihaoden@cs.cmu.edu, qianlim@cs.cmu.edu
In a wide range of multimodal tasks, contrastive learning has become a particularly appealing approach since it can successfully learn representations from abundant unlabeled data with only pairing information (e.g., image-capture or video-audio pairs). Underpinning these approaches is the assumption of multi-view redundancy - that shared information between modalities is necessary and sufficient for downstream tasks. However, in many real-world settings, task-relevant information is also contained in modality-unique regions: information that is only present in one modality but still relevant to the task. How can we learn self-supervised multimodal representations to capture both shared and unique information relevant to downstream tasks? This paper proposes FACTORCL, a new multimodal representation learning method to go beyond multi-view redundancy. FACTORCL is built from three new contributions: (1) factorizing task-relevant information into shared and unique representations, (2) capturing task-relevant information via maximizing MI lower bounds and removing task-irrelevant information via minimizing MI upper bounds, and (3) multimodal data augmentations to approximate task relevance without labels. On large-scale real-world datasets, FACTORCL captures both shared and unique information and achieves state-of-the-art results on six benchmarks.
Learning representations from different modalities is a central paradigm in machine learning [48]. Today, a popular learning method is to first pre-train general representations on unlabeled multimodal data before fine-tuning on task-specific labels [10, 40, 47, 48, 50]. These current multimodal pretraining approaches have largely been inherited from prior work in multi-view learning [13, 58] that exploit a critical assumption of multi-view redundancy: the property that shared information between modalities is almost exactly what is relevant for downstream tasks [70, 73, 76]. When this assumption holds, approaches based on contrastive pre-training to capture shared information [13, 41, 61, 72], followed by fine-tuning to keep task-relevant shared information [76], have seen successful applications in learning from images and captions [61], video and audio [3], speech and transcribed text [58], and instructions and actions [21]. However, our paper studies two fundamental limitations in the application of contrastive learning (CL) to broader real-world multimodal settings (see Figure 1 for a visual depiction and experimental results showing the performance drop of CL):
1. Low shared information relevant to tasks: There exists a wide range of multimodal tasks involving small amounts of shared information, such as between cartoon images and figurative captions (i.e., not literal but metaphoric or idiomatic descriptions of the images [52, 88]). In these situations, standard multimodal CL will only receive a small percentage of information from the learned representations and struggle to learn the desired task-relevant information.
2. High unique information relevant to tasks: Many real-world modalities can provide unique information not present in other modalities. Examples include healthcare with medical sensors or robotics with force sensors [45, 49]. Standard CL will discard task-relevant unique information, leading to poor downstream performance.
* First three authors contributed equally.
Figure 1: Left: We define  $S = I(X_1;X_2;Y)$  as task-relevant shared information and  $U_{1} = I(X_{1};Y|X_{2})$ ,  $U_{2} = I(X_{2};Y|X_{1})$  as task-relevant unique information. Right: On controllable datasets with varying ratios of  $S$ ,  $U_{1}$ , and  $U_{2}$ , standard CL captures  $S$  but struggles when there is more  $U_{1}$  and  $U_{2}$ . Our FACTORCL approach maintains best performance, whereas SimCLR [13] and SupCon [41] see performance drops as unique information increases, and Cross+Self [33, 36, 44, 89] recovers in fully unique settings but suffers at other ratios.
In light of these limitations, how can we design suitable multimodal learning objectives that work beyond multi-view redundancy? In this paper, starting from the first principles in information theory, we provide formal definitions of shared and unique information via conditional mutual information and propose an approach, FACTORIZED CONTRASTIVE LEARNING (FACTORCL for short), to learn these multimodal representations beyond multi-view redundancy using three key ideas. The first idea is to explicitly factorize shared and unique representations. The second idea is to capture task-relevant information via maximizing lower bounds on MI and remove task-irrelevant information via minimizing upper bounds on MI, resulting in representations with sufficient and necessary information content. Finally, a notion of task relevance without explicit labels in the self-supervised setting is achieved by leveraging multimodal augmentations. Experimentally, we evaluate the effectiveness of FACTORCL on a suite of synthetic datasets and large-scale real-world multimodal benchmarks involving images and figurative language [88], prediction of human sentiment [91], emotions [93], humor [27], and sarcasm [12], as well as patient disease and mortality prediction from health indicators and sensor readings [38], achieving new state-of-the-art performance on six datasets. Overall, we summarize our key technical contributions here:
1. A new analysis of contrastive learning performance showing that standard multimodal CL fails to capture task-relevant unique information under low shared or high unique information cases.
2. A new contrastive learning algorithm called FACTORCL:
(a) FACTORCL factorizes task-relevant information into shared and unique information, expanding contrastive learning to better handle low shared or high unique information.
(b) FACTORCL optimizes shared and unique information separately, by removing task-irrelevant information via MI upper bounds and capturing task-relevant information via lower bounds, yielding optimal task-relevant representations.
(c) FACTORCL leverages multimodal augmentations to approximate task-relevant information, enabling self-supervised learning from our proposed FACTORCL.
2 Analysis of Multi-view Contrastive Learning
We begin by formalizing definitions of four types of information: shared, unique, task-relevant, and task-irrelevant information in multimodal data. To formalize the learning setting, we assume there exist two modalities expressed as random variables  $X_{1}$  and  $X_{2}$  with outcomes  $x_{1}$  and  $x_{2}$ , and a task with the random variable  $Y$  and outcome  $y$ . We denote  $X_{-i}$  as the other modality where appropriate.
Shared and unique information: We formalize shared and unique information by decomposing the total multimodal information  $I(X_{1},X_{2};Y)$  into three conditional mutual information (MI) terms:
$$
I \left(X _ {1}, X _ {2}; Y\right) = \underbrace {I \left(X _ {1} ; X _ {2} ; Y\right)} _ {S = \text {s h a r e d}} + \underbrace {I \left(X _ {1} ; Y \mid X _ {2}\right)} _ {U _ {1} = \text {u n i q u e n e s s i n} X _ {1}} + \underbrace {I \left(X _ {2} ; Y \mid X _ {1}\right)} _ {U _ {2} = \text {u n i q u e n e s s i n} X _ {2}}, \tag {1}
$$
where  $I(X_1, X_2; Y) = \int p(x_1, x_2, y) \log \frac{p(x_1, x_2, y)}{p(x_1, x_2) p(y)} dx_1 dx_2 dy$  is the total MI between the joint random variable  $X_1, X_2$  and the task  $Y$ ,  $S = I(X_1; X_2; Y) = I(X_1; X_2) - I(X_1; X_2|Y) = \int p(x_1, x_2) \log \frac{p(x_1, x_2)}{p(x_1) p(x_2)} dx_1 dx_2 - I(X_1; X_2|Y)$  is the task-relevant shared in-
formation,  $I(X_{1};X_{2}|Y) = \int p(x_{1},x_{2},y)\log \frac{p(x_{1},x_{2}|y)}{p(x_{1}|y)p(x_{2}|y)} dx_{1}dx_{2}dy$  is the task-irrelevant shared information, and  $U_{1} = I(X_{1};Y|X_{2})$ ,  $U_{2} = I(X_{2};Y|X_{1})$  denote unique task-relevant information.
Limitations of CL: Current approaches for CL maximize mutual information  $I(X_{1};X_{2})$  (and subsequently task-relevant shared information  $I(X_{1};X_{2};Y)$  during supervised fine-tuning), without modeling unique information. These methods generally learn a pair of representations [73, 76],
$$
Z _ {1} = \underset {Z _ {1} := f _ {\theta} \left(X _ {1}\right)} {\arg \max } I \left(Z _ {1}; X _ {2}\right), \quad Z _ {2} = \underset {Z _ {2} := f _ {\theta} \left(X _ {2}\right)} {\arg \max } I \left(X _ {1}; Z _ {2}\right). \tag {2}
$$
For example,  $Z_{1}$  could encode images  $X_{1}$  and  $Z_{2}$  encodes text  $X_{2}$  via maximizing a lower bound on  $I(X_{1};X_{2})$  using the NCE objective [58]. The NCE objective falls into a broader class of contrastive learning methods [13, 15, 28, 41, 61] that model the ratio between joint densities  $p(x_{1},x_{2})$  and product of marginal densities  $p(x_{1})p(x_{2})$  using positive and negative samples [57, 59, 60, 79, 84] or probabilistic classifiers [55, 77]. It has been shown that contrastive learning works well under the assumption of multi-view redundancy [4, 31, 70, 71, 76]:
Definition 1. (Multi-view redundancy)  $\exists \epsilon >0$  such that  $I(X_{1};Y|X_{2})\leq \epsilon$  and  $I(X_{2};Y|X_{1})\leq \epsilon$
In other words, the task-relevant information in data is mostly shared across both views and the unique information is at most a small  $\epsilon$ . From a representation perspective, Tian et al. [72] further introduces the assumption that the optimal representation is minimal and sufficient, where all learned task-relevant information is shared information:  $I(Z_{1};Y|X_{2}) = I(Z_{2};Y|X_{1}) = 0$ . While the multiview redundancy is certainly true for particular types of multimodal distributions, it crucially ignores settings that display multi-view non-redundancy and unique information can be important, such as when health indicators, medical sensors, and robotic visual or force sensors each provide unique information not present in other modalities [45, 49].
Definition 2. (Multi-view non-redundancy)  $\exists \epsilon >0$  such that  $I(X_{1};Y|X_{2}) > \epsilon$  or  $I(X_{2};Y|X_{1}) > \epsilon$ .
Under multi-view non-redundancy, we show that standard CL only receives a weak training signal since it can only maximize a lower bound on shared information  $I(X_{1};X_{2})$ , and struggles to learn task-relevant unique information. We formalize this intuition with the following statement:
Theorem 1. (Suboptimality of standard CL) When there is multi-view non-redundancy as in Definition 2, given optimal representations  $\{Z_1,Z_2\}$  that satisfy Eq.(2 and  $I(Z_{1};Y|X_{2}) = I(Z_{2};Y|X_{1}) = 0$  [72], we have that
$$
I \left(Z _ {1}, Z _ {2}; Y\right) = I \left(X _ {1}, X _ {2}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) = I \left(X _ {1}; X _ {2}\right) - I \left(X _ {1}; X _ {2} \mid Y\right) <   I \left(X _ {1}, X _ {2}; Y\right). \tag {3}
$$
Correspondingly, the Bayes error rate  $P_{e}(Z_{1}, Z_{2}) \coloneqq 1 - \mathbb{E}_{p(z_{1}, z_{2})} \left[ \max_{y \in Y} P(\hat{Y} = y \mid z_{1}, z_{2}) \right]$  of contrastive representations  $\{Z_{1}, Z_{2}\}$  for a downstream task  $Y$  is given by:
$$
\begin{array}{l} P _ {e} \leq 1 - \exp \left[ I \left(X _ {1}, X _ {2}; Y\right) - I \left(X _ {1}; Y \mid X _ {2}\right) - I \left(X _ {2}; Y \mid X _ {1}\right) - H (Y) \right] (4) \\ = 1 - \exp \left[ I \left(X _ {1}; X _ {2}; Y\right) - H (Y) \right] (5) \\ \end{array}
$$
We include proofs and a detailed discussion of the assumptions in Appendix B. Based on Eq.(3),  $I(Z_{1},Z_{2};Y)$  decreases with higher task-relevant unique information  $I(X_{1};Y|X_{2})$  and  $I(X_{2};Y|X_{1})$ ; we call this the difference  $I(X_{1},X_{2};Y) - I(Z_{1},Z_{2};Y)$  the uniqueness gap. The uniqueness gap measures the loss in task-relevant information between the input and encoded representation: as task-relevant unique information grows, the uniqueness gap increases. In addition,  $I(Z_{1},Z_{2};Y)$  also drops with lower  $I(X_{1};X_{2})$  (i.e., two modalities sharing little information to begin with), or with higher  $I(X_{1};X_{2}|Y)$  (i.e., when the shared information is mostly task-irrelevant). Similarly, in Eq.(5), the Bayes error rate of using  $\{Z_{1},Z_{2}\}$  for prediction is directly related to the task-relevant information in  $\{Z_{1},Z_{2}\}$ : error on the downstream task increases with higher unique information and lower shared information.
3 FACTORIZED CONTRASTIVE LEARNING
We now present a suite of new CL objectives that alleviate the challenges above and work at all ranges of shared and unique information. At a high level, we aim to learn a set of factorized representations  $Z_{S_1}, Z_{S_2}, Z_{U_1}, Z_{U_2}$  representing task-relevant information in  $X_1$  shared with  $X_2$ , in  $X_2$  shared with  $X_1$ , unique to  $X_1$ , and unique to  $X_2$  respectively. As common in practice [61, 72], we define
Figure 2: FACTORCL: We propose a self-supervised CL method to learn factorized representations  $Z_{S_1}$ ,  $Z_{S_2}$ ,  $Z_{U_1}$ , and  $Z_{U_2}$  to capture task-relevant information shared in both  $X_1$  and  $X_2$ , unique to  $X_1$ , and unique to  $X_2$ . By starting with information-theoretic first principles of shared and unique information, we design contrastive estimators to both capture task-relevant and remove task-irrelevant information, where a notion of task-relevance without explicit labels is afforded by a new definition of multimodal augmentations  $X_1'$ ,  $X_2'$ . Lower bounds are in green and upper bounds are in red.
neural networks  $f_{\theta}$  with trainable parameters  $\theta$  to extract representations from inputs  $X_{1}$  and  $X_{2}$ . Learning these parameters requires optimizing differentiable and scalable training objectives to capture task-relevant shared and unique information (see overview in Figure 2):
$$
Z _ {S _ {1}} = \underset {Z _ {1} = f _ {\theta} (X _ {1})} {\arg \max } I \left(Z _ {1}; X _ {2}; Y\right), \quad Z _ {S _ {2}} = \underset {Z _ {2} = f _ {\theta} (X _ {2})} {\arg \max } I \left(Z _ {2}; X _ {1}; Y\right), \tag {6}
$$
$$
Z _ {U _ {1}} = \underset {Z _ {1} = f _ {\theta} (X _ {1})} {\arg \max } I \left(Z _ {1}; Y \mid X _ {2}\right), \quad Z _ {U _ {2}} = \underset {Z _ {2} = f _ {\theta} (X _ {2})} {\arg \max } I \left(Z _ {2}; Y \mid X _ {1}\right). \tag {7}
$$
where  $I(Z_{1};X_{2};Y) = I(Z_{1};X_{2}) - I(Z_{1};X_{2}|Y)$  is the shared information and  $I(Z_{2};X_{1};Y) = I(Z_{2};X_{2}) - I(Z_{2};X_{1}|Y)$  is the unique information. One important characteristic of our framework is that when unique information is zero:  $I(X_{1};Y|X_{2}) = 0$  and  $I(X_{2};Y|X_{1}) = 0$ , or all shared information is task-relevant:  $I(X_{1};X_{2};Y) = I(X_{1};X_{2})$ , our framework recovers standard CL as in Eq.(2). However, as we have previously indicated and will show empirically, these assumptions can easily be violated, and our framework enlarges Eq.(2) to cases where unique information is present.
The learned  $Z_{s}$  can then be used as input to a linear classifier and fine-tuned to predict the label for multimodal classification or retrieval tasks. However, the shared and unique MI terms above are often intractable in practice. In the next section, we will build up our method step by step, eventually showing that each term in Eqs.(6-7) can be approximated as follows:
$$
S = I \left(X _ {1}; X _ {2}; Y\right) \geq I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) - I _ {\mathrm {N C E} - \mathrm {C L U B}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) \tag {8}
$$
$$
U _ {i} = I \left(X _ {i}; Y \mid X _ {- i}\right) \geq I _ {\mathrm {N C E}} \left(X _ {i}; X _ {i} ^ {\prime}\right) - I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2}\right) + I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) \tag {9}
$$
where  $I_{\mathrm{NCE}}$  and  $I_{\mathrm{NCE-CLUB}}$  are scalable contrastive estimators (Section 3.1) and  $X_1', X_2'$  are suitable data augmentations (Section 3.2) on each modality. Overall, these equations can be interpreted as both positive and negative signals to learn representations for  $S$  and  $U$ . For shared information  $S$ , the estimator maximizes task-relevant shared information via  $I_{\mathrm{NCE}}(X_1; X_2)$  while removing task-irrelevant shared information via a novel upper bound  $-I_{\mathrm{NCE-CLUB}}(X_1; X_2|X_1', X_2')$ . For unique information  $U_i$ , we capture task-relevant uniqueness via  $+I_{\mathrm{NCE}}(X_i; X_i')$  while non-unique information is removed via  $-(I_{\mathrm{NCE-CLUB}}(X_1; X_2) - I_{\mathrm{NCE}}(X_1; X_2|X_1', X_2'))$ . In the following sections, we derive this final objective step-by-step: (1) approximating the MI objectives in  $S$  and  $U$  with CL estimators, (2) relaxing the dependence on labels  $Y$  with self-supervised data augmentations, finally (3) discussing overall training and implementation details of end-to-end self-supervised learning.
3.1 Supervised FACTORCL with shared and unique information
To capture shared and unique information via an objective function, we will need to maximize lower bounds for all terms with a positive sign in Eq.(8) and (9)  $(I(X_{1};X_{2}),I(X_{i};Y),I(X_{1};X_{2}|Y))$  and minimize upper bounds for all terms with a negative sign  $(I(X_{1};X_{2}),I(X_{1};X_{2}|Y))$ . Our first theorem derives general lower and upper bounds for MI terms as variants of contrastive estimation:
Figure 3: Estimated  $I_{\mathrm{NCE}}$  lower bound [58] and our proposed upper bound  $I_{\mathrm{NCE-CLUB}}$  on sample distributions with changing mutual information: our upper bound is tighter, more accurate, and more stable than  $I_{\mathrm{CLUB}}$  upper bound [16], and also comes for 'free' via jointly estimating both lower and upper bounds simultaneously. We find that as dimension increases, the  $I_{\mathrm{CLUB}}$  estimator collapses to zero and no longer tracks true MI.
Theorem 2. (Contrastive estimators for  $I(X_1;X_2)$ ) Defining the NCE and NCE-CLUB estimators,
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) = \mathbb {E} _ {\substack {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2}\right) \\ x _ {2} \sim p \left(x _ {2}\right)}} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+}\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-}\right)} \right] \tag{10}
$$
$$
I _ {\text {N C E - C L U B}} \left(X _ {1}; X _ {2}\right) = \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2}\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {+}\right) \right] - \mathbb {E} _ {\substack {x _ {1} \sim p \left(x _ {1}\right) \\ x _ {2} ^ {-} \sim p \left(x _ {2}\right)}} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {-}\right) \right] \tag{11}
$$
where  $f^{*}(x_{1},x_{2})$  is the optimal critic from  $I_{\mathrm{NCE}}$  plugged into the  $I_{\mathrm{CLUB}}$  objective [16]. We call the proposed plug-in objective Eq.(11)  $I_{\mathrm{NCE - CLUB}}$ , and obtain lower and upper bounds on  $I(X_{1};X_{2})$ :
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) \leq I \left(X _ {1}; X _ {2}\right) \leq I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2}\right). \tag {12}
$$
Proof. The lower bound  $I_{\mathrm{NCE}}(X_1; X_2) \leq I(X_1; X_2)$  follows from Oord et al. [58]: optimizing the objective leads to an optimal critic [60]  $f^* = \log p(x_1 | x_2) + c(x_1)$ , with a deterministic function  $c(\cdot)$ . Plugging optimal critic  $f^*$  into  $I_{\mathrm{NCE-CLUB}}(X_1; X_2)$  cancels out the  $c(x_1)$  term and yields  $I_{\mathrm{NCE-CLUB}}(X_1; X_2)$  and  $I(X_1; X_2) \leq I_{\mathrm{NCE-CLUB}}$ . We include a detailed proof in Appendix C.1.
$I_{\mathrm{NCE - CLUB}}(X_1;X_2)$  gives a desired upper bound of  $I(X_{1};X_{2})$  "for free" while avoiding separately optimizing lower bound and upper bounds. In Figure 3, we show these two bounds in practice across two Gaussian distributions  $X_{1}$  and  $X_{2}$  with varying amounts of MI  $I(X_{1};X_{2})$ . We use the second formulation of  $I_{\mathrm{CLUB}}$  [16], which assumes  $p(x_{1}|x_{2})$  to be unknown. Our upper bound is empirically tighter (see Figure 3) and comes for "free" via jointly maximizing the lower bound  $I_{\mathrm{NCE}}$ . These lower and upper bounds can be seen as new contrastive objectives over positive and negative  $(x_{1},x_{2})$  pairs, enabling a close integration with existing pre-training paradigms. Finally, we can similarly obtain bounds for the conditional MI  $I_{\mathrm{NCE}}(X_1;X_2|Y)\leq I(X_1;X_2|Y)\leq I_{\mathrm{NCE - CLUB}}(X_1;X_2|Y)$ :
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid Y\right) = \mathbb {E} _ {p (y)} \left[ \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2} \mid y\right)} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+} , y\right)}{\sum_ {k} \exp f \left(x _ {1} , x _ {2} ^ {-} , y\right)} \right] \right] \tag {13}
$$
$$
I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2} | Y\right) = \mathbb {E} _ {p (y)} \left[ \mathbb {E} _ {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2} \mid y\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {+}, y\right) \right] - \mathbb {E} _ {x _ {1} \sim p \left(x _ {1} \mid y\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2} ^ {-}, y\right) \right] \right] \tag {14}
$$
These two bounds result in conditional CL objectives [51, 74, 78] - they differ critically from standard CL methods since they capture task-irrelevant shared information that remains between  $X_{1}$  and  $X_{2}$  after observing  $Y$ . This task-irrelevant shared information is removed by minimizing its upper bound. Note that  $f(x_{1},x_{2},y)$  here denotes a different function from  $f(x_{1},x_{2})$  in Eq.(10), as the general forms are different (taking in  $x_{1},x_{2}$  versus  $x_{1},x_{2},y$ ).  $f(x_{1},x_{2},y)$  can be implemented in different ways, e.g.,  $g([x_1,y])^T h(x_2)$  where  $g(),h()$  are trainable encoders and  $[x_{1},y]$  denotes concatenation [69].
3.2 Self-supervised FACTORCL via multimodal augmentations
The derivations above bring about supervised CL objectives with access to  $Y$  [41]. For unsupervised CL [58, 72], we derive similar objectives without access to  $Y$  by leveraging semantic augmentations on each modality. Denote  $X'$  as some augmentation of  $X$  (e.g., rotating, shifting, or cropping). Under
the optimal augmentation assumption from Tian et al. [72] (restated below), replacing  $Y$  with  $X'$  in our formulations enables learning of task-relevant information without access to labels:
Definition 3. (Optimal unimodal augmentation) [72]  $X_1'$  is an optimal unimodal augmentation for  $X_1$  when  $I(X;X') = I(X;Y)$ , which implies that the only information shared between  $X$  and  $X'$  is task-relevant with no irrelevant noise.
This assumption is satisfied when all information shared between  $X$  and  $X'$  is task-relevant, which implies that the augmentation keeps task-relevant information constant while changing task-irrelevant information. In the case of image classification, task-relevant information is the object in the picture, while task-irrelevant information is the background. By performing two separate unimodal augmentations giving  $X_1'$  and  $X_2'$ , we can substitute contrastive estimators in Eqs.(13) and (14), by replacing  $I(X_i;Y)$  terms with  $I(X_i;X_i')$  and replacing  $I(X_1;X_2|Y)$  terms with  $I(X_1;X_2|X_1',X_2')$ :