SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, sampling from  $p(\cdot | x_1', x_2')$  is hard. Since  $X_1'$ ,  $X_2'$  are continuous variables, directly sampling from the conditional distributions  $p(\cdot | x_1', x_2')$  may be difficult; training a generative model  $p_{\theta}(x_1, x_2 | x_1', x_2')$  from augmented data  $x_1', x_2'$  to original data  $x_1, x_2$  can be expensive and nontrivial in a multimodal setup. In this work, we implement the conditioning in  $p(x_1, x_2 | x_1', x_2')$  through concatenation and the details are in Appendix D.1. Here we discuss an alternative solution to this problem introduced by Tsai et al. [78]. It leverages kernel methods for conditional sampling in contrastive learning by assigning weights to each sampled data given the kernel similarity between conditioned variables, avoiding directly sampling from the conditional distributions or training generative models. In our formulation, given multimodal input  $(x_1, x_2)$  with their augmentation  $(x_1', x_2')$ , we can simply use the technique from [78] to estimate  $I_{\mathrm{NCE}}(X_1; X_2 | X_1', X_2')$  and  $I_{\mathrm{NCE-CLUB}}(X_1; X_2 | X_1', X_2')$ , where the kernel measures the similarity between different pairs  $(x_1', x_2')$  of the conditional variable
$X_{1},X_{2}$  . Specifically,
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) = \mathbb {E} _ {p \left(x _ {1}, x _ {2}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ \log \frac {\exp f \left(x _ {1} , x _ {2} ^ {+}\right)}{\exp f \left(x _ {1} , x _ {2}\right) + n * \left[ K _ {X _ {1} \perp X _ {2} \mid X _ {1} ^ {\prime} , X _ {2} ^ {\prime}} \right] _ {i i}} \right] \tag {70}
$$
$$
I _ {\text {N C E - C L U B}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) = \mathbb {E} _ {p \left(x _ {1}, x _ {2}, x _ {1} ^ {\prime}, x _ {2} ^ {\prime}\right)} \left[ f ^ {*} \left(x _ {1}, x _ {2}\right) - \log \left[ K _ {X _ {1} \perp X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}} \right] _ {i i} \right] \tag {71}
$$
where  $K_{X_1 \perp X_2 | X_1', X_2'} = K_{X_1 X_2} (K_{X_1' X_2'} + \lambda \mathbf{I})^{-1} K_{X_1' X_2'}$  and  $[K_{X_1 \perp X_2 | X_1', X_2'}]_{ii}$  is the  $i$ th row and  $i$ th column of  $K_{X_1 \perp X_2 | X_1', X_2'}$ .  $K_{X_1 X_2}$  is a kernel similarity matrix between  $X_1$  and  $X_2$ , and  $K_{X_1' X_2'}$  is a separate kernel similarity matrix between  $X_1'$  and  $X_2'$ .  $f^*$  is the optimal solution of Eq.(70). By leveraging the similarity  $K_{X_1' X_2'}$  between conditional variables  $X_1'$  and  $X_2'$ ,  $K_{X_1 \perp X_2 | X_1', X_2'}$  transforms the similarity scores between  $X_1$  and  $X_2$  under unconditional distributions into similarity scores under conditional distributions. Note that the expectations in Eqs.(70) and (71) are taken over the joint distribution  $p(x_1, x_2, x_1', x_2')$ , which comes naturally after augmenting both modalities  $X_1$  and  $X_2$ . This method could effectively alleviate the problem of sampling from conditional distributions in our formulation. We refer the reader to Tsai et al. [78] for more details.
C.3 Final estimators in FACTORCL
Theorem 6. (Contrastive estimators for shared and unique information). Under assumptions on single-view augmentations  $I(X_{1};Y) = I(X_{1},X_{1}^{\prime})$  (Definition 8) and optimal multi-view augmentation  $X_{2}^{\prime}$  such that  $I(X_{1},X_{2};X_{1}^{\prime},X_{2}^{\prime}) = I(X_{1},X_{2};Y)$  (Definition 9), we can define contrastive objectives for task-relevant shared and unique information with:
$$
S = I \left(X _ {1}; X _ {2}; Y\right) \geq I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) - I _ {\mathrm {N C E} - \mathrm {C L U B}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) \tag {72}
$$
$$
U _ {i} = I \left(X _ {i}; Y \mid X _ {- i}\right) \geq I _ {\mathrm {N C E}} \left(X _ {i}; X _ {i} ^ {\prime}\right) - I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2}\right) + I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) \tag {73}
$$
Proof. The objectives follow from the fact that  $I_{\mathrm{NCE}}(X_1;X_2)$  and  $I_{\mathrm{NCE}}(X_1;X_2|X_1',X_2')$  are lower bounds of  $I(X_1;X_2)$  and  $I(X_1;X_2|Y)$  respectively, and  $I_{\mathrm{NCE - CLUB}}(X_1;X_2)$  and  $I_{\mathrm{NCE - CLUB}}(X_1;X_2|X_1',X_2')$  are upper bounds of  $I(X_1;X_2)$  and  $I(X_1;X_2|Y)$  respectively:
$$
S = I \left(X _ {1}; X _ {2}; Y\right) = I \left(X _ {1}; X _ {2}\right) - I \left(X _ {1}; X _ {2} \mid Y\right) \tag {74}
$$
$$
\geq I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) - I _ {\mathrm {N C E} - \text {C L U B}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) \tag {75}
$$
$$
U _ {i} = I \left(X _ {i}; Y \mid X _ {- i}\right) = I \left(X _ {i}; Y\right) - \left(I \left(X _ {1}; X _ {2}\right) - I \left(X _ {1}; X _ {2} \mid Y\right)\right) \tag {76}
$$
$$
\geq I _ {\mathrm {N C E}} \left(X _ {i}; X _ {i} ^ {\prime}\right) - \left(I _ {\mathrm {N C E} - \text {C L U B}} \left(X _ {1}; X _ {2}\right) - I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right)\right) \tag {77}
$$
and symmetrically for  $U_{2}$
Now we show that FACTORCL learns both shared and unique task-relevant information. First, we restate the definition of the factorized representations:
$$
Z _ {S _ {1}} = \underset {Z _ {1} = f _ {\theta} (X _ {1})} {\arg \max } I \left(Z _ {1}; X _ {2}; Y\right), \quad Z _ {S _ {2}} = \underset {Z _ {2} = f _ {\theta} (X _ {2})} {\arg \max } I \left(Z _ {2}; X _ {1}; Y\right), \tag {78}
$$
$$
Z _ {U _ {1}} = \underset {Z _ {1} = f _ {\theta} (X _ {1})} {\arg \max } I \left(Z _ {1}; Y \mid X _ {2}\right), \quad Z _ {U _ {2}} = \underset {Z _ {2} = f _ {\theta} (X _ {2})} {\arg \max } I \left(Z _ {2}; Y \mid X _ {1}\right). \tag {79}
$$
where  $I(Z_{1};X_{2};Y) = I(Z_{1};X_{2}) - I(Z_{1};X_{2}|Y)$  is the shared information and  $I(Z_{2};X_{1};Y) = I(Z_{2};X_{2}) - I(Z_{2};X_{1}|Y)$  is the unique information.
Theorem 7. (Optimality of FACTORCL) If  $Z_{S_1}, Z_{S_2}, Z_{U_1}, Z_{U_2}$  perfectly maximize Eqs.(78-79) and the estimations in Eqs.(13-67) are tight, we obtain  $I(X_1, X_2; Y) = I(Z_{S_1}; Z_{S_2}; Y) + I(Z_{U_1}; Y|Z_{S_2}) + I(Z_{U_2}; Y|Z_{S_1})$ , suggesting that FACTORCL learns both shared and unique task-relevant information.
Proof. Because  $I(X_{1},X_{2};Y) = I(X_{1};X_{2};Y) + I(X_{1};Y|X_{2}) + I(X_{2};Y|X_{1})$ , it is sufficient to show that  $I(X_{1};X_{2};Y) = I(Z_{S_{1}};Z_{S_{2}};Y), I(X_{1};Y|X_{2}) = I(Z_{U_{1}};Y|Z_{S_{2}})$  and  $I(X_{2};Y|X_{1}) = I(Z_{U_{2}};Y|Z_{S_{1}})$ .
First we show  $I(X_{1};X_{2};Y) = I(Z_{S_{1}};Z_{S_{2}};Y)$ . Crucially, by definition of how  $Z_{S_1}$  and  $Z_{S_2}$  are optimized to maximize  $I(X_{1};X_{2};Y)$ , we have that:
$$
I \left(X _ {1}; X _ {2}; Y\right) = I \left(Z _ {S _ {1}}; X _ {2}; Y\right) = I \left(Z _ {S _ {2}}; X _ {1}; Y\right). \tag {80}
$$
We can then obtain
$$
\begin{array}{l} I \left(X _ {1}; X _ {2}; Y\right) = I \left(X _ {1}; Z _ {S _ {2}}; Y\right) (81) \\ = I \left(X _ {1}; Z _ {S _ {2}}; Y \mid Z _ {S _ {1}}\right) + I \left(Z _ {S _ {1}}; Z _ {S _ {2}}; X _ {1}; Y\right) (82) \\ = I \left(Z _ {S _ {2}}; Y \mid Z _ {S _ {1}}\right) - I \left(Z _ {S _ {2}}; Y \mid Z _ {S _ {1}}, X _ {1}\right) + I \left(Z _ {S _ {1}}; Z _ {S _ {2}}; X _ {1}; Y\right) (83) \\ = I \left(Z _ {S _ {2}}; Y \mid Z _ {S _ {1}}\right) - I \left(Z _ {S _ {2}}; Y \mid X _ {1}\right) + I \left(Z _ {S _ {1}}; Z _ {S _ {2}}; X _ {1}; Y\right) (84) \\ = I \left(Z _ {S _ {2}}; Y \mid Z _ {S _ {1}}\right) - I \left(Z _ {S _ {2}}; Y \mid X _ {1}\right) + I \left(Z _ {S _ {1}}; Z _ {S _ {2}}; Y\right) (85) \\ = I \left(Z _ {S _ {2}}; Y \mid Z _ {S _ {1}}\right) + I \left(Z _ {S _ {1}}; Z _ {S _ {2}}; Y\right) (86) \\ = I \left(Z _ {S _ {1}}; Z _ {S _ {2}}; Y\right) (87) \\ \end{array}
$$
where Eq.(84) is because  $Z_{S_1}$  are deterministically obtained from  $S_1$  and Eq.(85) is because  $Z_{S_1}$  maximizes the shared information. Finally, we go to Eq.(87)  $I(Z_{S_2};Y|Z_{S_1}) = 0$  as shown in Eqs.(26-33) using the fact that  $Z_{S_1}$  is learned to maximize  $I(X_1;X_2;Y)$  and  $I(Z_{S_1};X_2;Y) = I(X_1;Z_{S_2};Y)$ .
Next, we show  $I(X_{1};Y|X_{2}) = I(Z_{U_{1}};Y|Z_{S_{2}})$ :
$$
I \left(Z _ {U _ {1}}; Y \mid Z _ {S _ {2}}\right) = I \left(Z _ {U _ {1}}; Y \mid Z _ {S _ {2}}, Z _ {U _ {2}}\right) + I \left(Z _ {U _ {1}}; Y; Z _ {U _ {2}} \mid Z _ {S _ {2}}\right), \tag {88}
$$
which is by the chain rule of conditional mutual information. Then we show  $I(Z_{U_1};Y;Z_{U_2}|Z_{S_2}) = 0$ :
$$
I \left(Z _ {U _ {1}}; Y; Z _ {U _ {2}} \mid Z _ {S _ {2}}\right) = I \left(Z _ {U _ {1}}; Z _ {U _ {2}} \mid Z _ {S _ {2}}\right) - I \left(Z _ {U _ {1}}; Z _ {U _ {2}} \mid Y; Z _ {S _ {2}}\right) = 0 - 0 = 0 \tag {89}
$$
This is because Eq.(79) leads to  $I(Z_{U_1};Y|X_2) = I(X_1;Y|X_2)$  and  $I(Z_{U_2};Y|X_1) = I(X_2;Y|X_1)$ . If the estimations in Eqs.(13-67) are tight, by conditioning and by the previously stated  $I(Z_{U_1};Y|X_2) = I(X_1;Y|X_2)$ ,  $Z_{U_1}$  tightly captures information from only  $X_1$  and not in  $X_2$ . The same applies to  $Z_{U_2}$ . We have  $I(Z_{U_1};X_2) = I(Z_{U_2};X_1) = I(Z_{U_1};Z_{U_2}) = I(Z_{U_1};Z_{U_2}|T) = 0$  with  $T$  being an arbitrary random variable because no shared information exists between  $Z_{U_1}$  and  $Z_{U_2}$ . Then we obtain:
$$
\begin{array}{l} I \left(Z _ {U _ {1}}; Y \mid Z _ {S _ {2}}, Z _ {U _ {2}}\right) = I \left(Z _ {U _ {1}}; Y \mid Z _ {S _ {2}}, Z _ {U _ {2}}, X _ {2}\right) + I \left(Z _ {U _ {1}}; Y; X _ {2} \mid Z _ {S _ {2}}, Z _ {U _ {2}}\right) (90) \\ = I \left(Z _ {U _ {1}}; Y \mid X _ {2}\right) (91) \\ \end{array}
$$
We use the fact that conditioning on  $Z_{S_2}, Z_{U_2}$  and  $X_2$  jointly reduces to conditioning on  $X_2$  since  $Z_{S_2}$  and  $Z_{U_2}$  are deterministically obtained from  $X_2$ . Lastly, since Eqs.(78-79) are satisfied,  $Z_{U_1} = \arg \max_{Z_1 = f_\theta(X_1)} I(Z_1; Y|X_2)$  therefore  $I(Z_{U_1}; Y|X_2) = I(X_1; Y|X_2)$ . We have:
$$
I \left(Z _ {U _ {1}}; Y \mid Z _ {S _ {2}}\right) = I \left(Z _ {U _ {1}}; Y \mid X _ {2}\right) = I \left(X _ {1}; Y \mid X _ {2}\right). \tag {92}
$$
The proof for  $I(X_{2};Y|X_{1}) = I(Z_{U_{2}};Y|Z_{S_{1}})$  is similar. We now have shown that  $I(X_{1};X_{2};Y) = I(Z_{S_{1}};Z_{S_{2}};Y)$ ,  $I(X_{1};Y|X_{2}) = I(Z_{U_{1}};Y|Z_{S_{2}})$  and  $I(X_{2};Y|X_{1}) = I(Z_{U_{2}};Y|Z_{S_{1}})$ , adding up all LHS and RHS we have the theorem.
C.4 Extensions to masking and non-contrastive learning
We now show how similar ideas can be extended to other popular self-supervised learning objectives, such as non-contrastive learning [6, 94] and masked pre-training [20, 29]. Importantly, this paper provides a new principle for multimodel self-supervised learning: (1) learning task-relevant information and (2) removing task-irrelevant information from both shared and unique parts across modalities. Our paper focuses on realizing this principle via multi-view information theory and contrastive learning. Below we provide two potential alternatives to realize this principle on non-contrastive and masking methods, respectively:
Non-contrastive learning: Methods such as Barlow Twins [94] and VICReg [6] use invariance and covariance regularizations to maximally preserve shared information in the embeddings across two modalities. However, the embeddings learned may contain only contain task-relevant information from the shared part and not unique parts. To use the principle in this paper to capture more task-relevant information from unique parts, one should perform VIC-regularization on  $X_{1}$  features, on
$X_{2}$  features, and on  $X_{1}, X_{2}$  cross-modal features. When performing VICReg on unimodal features, one should condition on the other modality when performing augmentation. Specifically, similar to the idea of multimodal augmentation in this paper, the augmentation of the second modality should not interfere with the shared part (i.e., do not augment regions referred to by the first modality), making the invariance and covariance regularization of the second modality focus on the augmented modality-unique features. This makes the model learn unique modality features that are not captured by the joint embedding from standard independent augmentations.
Masking: Conceptually, masking [20, 29] can be interpreted as leveraging unmasked regions in the same modality to predict masked regions or leveraging the other modality to predict the masked region. However, the learned representation may not be all task-relevant. To use the principle in this paper to exclude task-irrelevant information and capture more task-relevant information from unique parts, we can perform conditional masking, where masking is conditioned on augmented views (similar to the multimodal augmentation in the paper, where the conditioned views are approximating the labels). As a result, only unique regions in the second modality can be masked out, making the model capture more unique information from the second modality by masked prediction. Here we have only provided high-level intuitions of extensions to these methods, and future work should explore these ideas in more detail.
Objective Formulation and Architecture
In Algorithm 2 in the main text, we see the sketch for doing contrastive learning with our proposed objectives. To implement all algorithms used in our ablation experiments, we start with two encoders  $e_1(\cdot)$  and  $e_2(\cdot)$ , which takes samples  $x_1$  and  $x_2$  from the modalities  $X_1$  and  $X_2$ , and outputs corresponding representations  $z_1$  and  $z_2$ . We also have a critic function  $f_{\theta}(\cdot, \cdot)$  parametrized by  $\theta$  which takes  $z_1$  and  $z_2$  as inputs and returns a scalar. A popular way to perform contrastive learning aims to maximize  $I_{\mathrm{NCE}}(X_1; X_2)$ , where
$$
I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right) = \mathbb {E} _ {\substack {x _ {1}, x _ {2} ^ {+} \sim p \left(x _ {1}, x _ {2}\right) \\ x _ {2} ^ {-} \sim p \left(x _ {2}\right)}} \left[ \log \frac {\exp f _ {\theta} \left(e _ {1} \left(x _ {1}\right) , e _ {2} \left(x _ {2} ^ {+}\right)\right)}{\sum_ {k} \exp f _ {\theta} \left(e _ {1} \left(x _ {1}\right) , e _ {2} \left(x _ {2} ^ {-}\right)\right)} \right]. \tag{93}
$$
In our algorithms, we follow the derivations in Eqs.(8-9) to maximize each  $I_{\mathrm{NCE}}$  objective and minimize each  $I_{\mathrm{NCE - CLUB}}$  objective. Therefore, for each objective, we add two additional MLP heads on top of the two encoders and create a separate critic which takes in the outputs of the MLP heads instead of the encoders. In all the experiments, we adopt the concat critic design [58, 60, 68], where  $f_{\theta}(x,y) = h_{\theta}([x,y])$  with  $h_\theta$  being an MLP.
FACTORCL-SUP: In the supervised version of CL which uses label  $Y$ , the objective we aim to maximize is formulated as
$$
\begin{array}{l} \mathcal {L} _ {\text {F A C T O R C L - S U P}} = I _ {\text {N C E}} \left(X _ {1}; X _ {2}\right) - I _ {\text {N C E - C L U B}} \left(X _ {1}; X _ {2} \mid Y\right) (94) \\ + I _ {\mathrm {N C E}} \left(X _ {1}; Y\right) + I _ {\mathrm {N C E}} \left(X _ {2}; Y\right) (95) \\ - I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2}\right) + I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid Y\right). (96) \\ \end{array}
$$
Each  $I_{\mathrm{NCE}}$  and  $I_{\mathrm{NCE-CLUB}}$  term in this objective is calculated using its own critic as discussed above. The conditional terms involving the label  $Y$  are implicitly modeled by directly concatenating  $Y$  to the outputs of both heads before feeding into the critic. To obtain the learned representations  $Z_{S_1}$ , we concatenate the outputs of the heads on top of the encoder  $e_1$  that correspond to the terms  $I_{\mathrm{NCE}}(X_1;X_2)$  and  $I_{\mathrm{NCE-CLUB}}(X_1;X_2|Y)$ . To obtain  $Z_{U_1}$ , we concatenate  $e_1$ 's head outputs from the terms  $I_{\mathrm{NCE}}(X_1;Y)$ ,  $I_{\mathrm{NCE-CLUB}}(X_1;X_2)$ , and  $I_{\mathrm{NCE}}(X_1;X_2|Y)$ .  $Z_{S_2}$  and  $Z_{U_2}$  are obtained in a similar fashion, except we use the outputs from  $e_2$ 's heads instead of  $e_1$ .
FACTORCL-SSL: In the self-supervised version of CL which uses augmentations  $X_1'$  and  $X_2'$  of the input modalities, the objective we aim to maximize is formulated as
$$
\begin{array}{l} \mathcal {L} _ {\text {F A C T O R C L - S S L}} = I _ {\text {N C E}} \left(X _ {1}; X _ {2}\right) - I _ {\text {N C E - C L U B}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) (97) \\ + I _ {\mathrm {N C E}} \left(X _ {1}; X _ {1} ^ {\prime}\right) + I _ {\mathrm {N C E}} \left(X _ {2}; X _ {2} ^ {\prime}\right) (98) \\ - I _ {\mathrm {N C E - C L U B}} \left(X _ {1}; X _ {2}\right) + I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right). (99) \\ \end{array}
$$
Figure 5: An illustration of conditioning by concatenation in the implementation of FACTORCL. Conditioning is done by concatenating  $Z_{1}$ , the encoded representation of  $X_{1}$ , and  $Z_{1}'$ , the encoded representation of  $X_{1}'$ . A similar operation is performed for  $X_{2}$  and  $X_{2}'$ . The concatenated vectors are then fed to MI estimators, such as  $I_{\mathrm{NCE}}$  and  $I_{\mathrm{NCE-CLUB}}$  (the figure illustrates  $I_{\mathrm{NCE}}$ ).
Here the conditional terms are conditioned on the augmentations  $X_1'$  and  $X_2'$ , and we can similarly model it by concatenating the head outputs of  $X_1'$  to  $X_1$  and the head outputs of  $X_2'$  to  $X_2$  before feeding into the critic. We use Figure 5 to illustrate this. The way to obtain the learned representations is the same as described in FACTORCL-SUP.
Estimation of CMI: To estimate the conditional mutual information (CMI)  $I(X_{1};X_{2}|X_{1}^{\prime},X_{2}^{\prime})$ , we can estimate the lower or upper bounds of true CMI [51, 55, 69]. However, direct sampling from the conditional distribution  $p(x_1,x_2|x_1',x_2')$  can be expensive because we should consider a different conditional distribution  $p(x_{1},x_{2}|x_{1}',x_{2}')$  for each data pair  $x_1',x_2'$ . Sordoni et al. [69] address this by concatenating the conditioning variable with the input in the critic:  $\phi (x_{1},x_{2},c)$ , and showing that Conditional InfoNCE (Eq.(15)) is a lower bound and estimator of CMI. This estimator can be made more exact by further importance sampling [69]. However, adding importance sampling [69] or using more accurate estimators [26] comes with a trade-off in complexity. Since we focus on capturing unique information to learn a scalable multimodal representation instead of accurately estimating the CMI, we leveraged a simpler version of the estimator from Sordoni et al. [69]: generating multiple augmented pairs from  $x_{1},x_{2}$ , and concatenating the input  $x_{1},x_{2}$  and each augmented pair  $x_1',x_2'$  to define samples from the conditional distribution  $p(x_{1},x_{2}|x_{1}',x_{2}')$ . We argue that since augmentations do not significantly change the semantics of images,  $p(x_{1},x_{2}|x_{1}',x_{2}')$  could be approximated by  $p(x_1'',x_2''|x_1',x_2')$  where  $x_1''$ ,  $x_2''$  are other augmented pairs in addition to  $x_1',x_2'$ . In this submission, we use one pair of augmented samples for consistency, but our code easily supports increasing the number of augmented pairs that can improve the accuracy of CMI estimation.
Regardless, our existing one-pair implementation can already show that our estimators are empirically comparable to CMI estimators with guarantees such as Mukherjee et al. [55] (Table 9), and our estimators empirically satisfy that the lower bound is smaller than the true CMI, and the true CMI smaller than the upper bound, i.e., Conditional InfoNCE  $\leq$  true  $\mathrm{CMI} \leq$  Conditional InfoNCE-CLUB (also in Table 9). We refer the reader to Sondoni et al. [69] for tighter bounds for CMI.
OurCL-SUP: For this ablation, we remove the factorization and only learn  $Z_{1}$  for  $X_{1}$  and  $Z_{2}$  for  $X_{2}$ . The objective we use is the same as that of FACTORCL-SUP. The only difference is that we now take  $e_1(x_1)$  and  $e_2(x_2)$  as the learned representations for inputs  $x_{1}$  and  $x_{2}$ .
OurCL-SSL: This is a similar ablation for FACTORCL-SSL where we remove the factorization. The objective is the same as that of FACTORCL-SSL and we use  $e_1(x_1)$  and  $e_2(x_2)$  as the learned representations for inputs  $x_1$  and  $x_2$ .
Training Strategy: In regular contrastive learning using  $I_{\mathrm{NCE}}$  as the only objective, we can simply perform gradient descent to minimize  $I_{\mathrm{NCE}}$ , updating all parameters in the encoders, MLP heads, and critics. However, training any of the four methods above also involves the minimization of the  $I_{\mathrm{NCE-CLUB}}$  objectives, which require the optimal critic  $f^{*}$  from  $I_{\mathrm{NCE}}$ , as stated in Eq.(11). Therefore, within each iteration during our training, we need to first obtain the optimal critics for the  $I_{\mathrm{NCE-CLUB}}$  terms using the  $I_{\mathrm{NCE}}$  objective. We outline the training strategy using a sampling method in Algorithm 3. In this algorithm,  $\mathcal{L}_{\mathrm{FACTORCL}}$  can be either  $\mathcal{L}_{\mathrm{FACTORCL-SUP}}$  or  $\mathcal{L}_{\mathrm{FACTORCL-SSL}}$ , and  $\mathcal{L}_{\mathrm{NCE}}$  is the summation of  $I_{\mathrm{NCE}}$  objectives for the  $I_{\mathrm{NCE-CLUB}}$  terms. In particular, we have
$$
\mathcal {L} _ {\mathrm {N C E}} = \left\{ \begin{array}{l l} I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid Y\right) + I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right), & \text {i f} \mathcal {L} = \mathcal {L} _ {\text {F A C T O R C L - S U P}}; \\ I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2} \mid X _ {1} ^ {\prime}, X _ {2} ^ {\prime}\right) + I _ {\mathrm {N C E}} \left(X _ {1}; X _ {2}\right), & \text {i f} \mathcal {L} = \mathcal {L} _ {\text {F A C T O R C L - S S L}}. \end{array} \right. \tag {100}
$$
Algorithm 3 CL training with sampling
Require: Multimodal dataset {X1, X2}.
θ, φ← Initialize network parameters.
while not converged do
{x1, x2}← Sampled batch from {X1, X2}
θ← Update parameters by maximizing LFACTORCL
for i = 1 to k do
{x1', x2'}← Sampled batch from {X1, X2}
φ← Update parameters by maximizing LNCE
end for
end while
return θ, φ
We define  $\phi$  to be the parameters of critics for the  $I_{\mathrm{NCE - CLUB}}$  terms, and  $\theta$  corresponds to all the rest parameters in the network (parameters of encoders, heads, and critics for  $I_{\mathrm{NCE}}$  terms). In the outer loop, we update  $\theta$  using the main learning objective  $\mathcal{L}$ . In the inner loop, we update  $\phi$  using the  $\mathcal{L}_{\mathrm{NCE}}$  objective, which learns the optimal critics  $f^{*}$  needed to compute the  $I_{\mathrm{NCE - CLUB}}$  terms. Ideally in the inner loop we would update  $\phi$  until convergence so we get a good approximation to the optimal critic. In practice we found sampling just one batch by setting  $k = 1$  in Algorithm 3 works pretty well. Using only one iteration does not have a big impact on the convergence and still produces promising results. More importantly, it significantly reduces the time required for training, and allows our algorithms to have comparable running time to existing contrastive learning methods.
D.2 Datasets
Gaussian datasets for MI estimation: As shown in Figure 3 in the main text, we first demonstrate the quality of our proposed upper bounds  $I_{\mathrm{NCE-CLUB}}(X_1; X_2)$  on a toy Gaussian dataset. We obtain the samples  $\{(x_i, y_i)\}$  from 4 multivariate Gaussian distribution with dimensions  $\{20, 50, 100, 200\}$ . In each dataset, we set the ground truth MI values to be  $\{2, 4, 6, 8, 10\}$ , and so we can compute the correlation  $\rho$  needed for achieving these MI values using the ground truth MI formula for Multivariate Gaussian:  $I(X, Y) = -\frac{d}{2}\log (1 - \rho^2)$ . At each true MI value we sample 4000 times using a batch size of 64.