Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities
Authors: Adriel Saporta
Aahlad Manas Puli
Mark Goldstein
Rajesh Ranganath
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal
contrastive learning
representation learning
total correlation
Primary area: probabilistic_methods
TL;DR: Symile is a simple contrastive learning approach that captures higher-order information between any number of modalities and provides a flexible, architecture-agnostic objective for learning modality-specific representations.

Abstract: Contrastive learning methods, such as CLIP, leverage naturally paired data—for example, images and their corresponding text captions—to learn general representations that transfer efficiently to downstream tasks. While such approaches are generally applied to two modalities, domains such as robotics, healthcare, and video need to support many types of data at once. We show that the pairwise application of CLIP fails to capture joint information between modalities, thereby limiting the quality of the learned representations. To address this issue, we present Symile, a simple contrastive learning approach that captures higher-order information between any number of modalities. Symile provides a flexible, architecture-agnostic objective for learning modality-specific representations. To develop Symile's objective, we derive a lower bound on total correlation, and show that Symile representations for any set of modalities form a sufficient statistic for predicting the remaining modalities. Symile outperforms pairwise CLIP, even with modalities missing in the data, on cross-modal classification and retrieval across several experiments including on an original multilingual dataset of 33M image, text and audio samples and a clinical dataset of chest X-rays, electrocardiograms, and laboratory measurements. All datasets and code used in this work are publicly available at https://github.com/rajesh-lab/symile.

Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities
Adriel Saporta"
Aahlad Puli
Mark Goldstein
Rajesh Ranganath
New York University
Abstract
Contrastive learning methods, such as CLIP, leverage naturally paired data-for example, images and their corresponding text captions-to learn general repre- sentations that transfer efficiently to downstream tasks. While such approaches are generally applied two modalities, domains such as robotics, healthcare, and video need to support many types of data at once. We show that the pair- wise application of CLIP fails to capture joint information between modalities, thereby limiting the quality of the learned representations. To address this issue, we present Symile, a simple contrastive learning approach that captures higher- order information between any number of modalities. Symile provides a flexible, architecture-agnostic objective for learning modality-seecffic representations. To develop Symile's 's objective, we derive lower bound on total correlation, and show that Symile representations for any set of modalities form a sufficient statistic for predicting the remaining modalities. Symile outperforms pairwise CLIP, even with modalities missing in the data, on cross-modal classification and retrieval across several experiments including text and audio samples and a clinical dataset of chest X-rays, electrocardiograms. and laboratory measurements. All datasets and code used in this work publicly available at https: / /gg thub. com/ ra jesh- ab/ symil. Le.
original multilingual dataset of 33M image,
Introduction
Contrastive learning leverages naturally paired data to learn general representations that transfer efficiently to downstream tasks [3, 35, 53]. common contrastive approach is to maximize the mutual information between the paired modalities, ensuring that the learned representations retain sensitivity to all correlations between them. While SimCLR [12] popularized the use the mutual information estimator InfoNCE [38] for data augmentations, CLIP [40] applied the approach to dis- tinct modalities for example, images and their corresponding text captions-wher where representations learned using any encoder for each modality.
While contrastive approaches are generally applied to two modalities, there a rapidly expanding range of domains that require the integration of many types of data at once. For example, in robotics, agents combine information from visual, proprioceptive. and tactile sensors [18, 28]: healthcare providers analyze various types of patient data including imaging, biosignals, and genomics [10, 29]; and video encompasses RGB frames, audio waveforms, and text transcripts [55]. One strategy for handling multimodal data has been to design specialized architectures capable of processing all data types at once, which limits their general applicability and increases operational complexity [2, 47]. Another common approach is to apply two-modality contrastive objectives, such as CLIP, to pairs of available modalities [15, 44].
In this paper, we show that, despite its popularity, the pairwise application of CLIP fails to cap- ture higher-order conditional information between modalities, thereby limiting the quality of the
"Correspondence to: Adriel Saporta <adriel@nyu.edu>.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

representations it learns. For instance, given three modalities b, and pairwise CLIP captures dependencies between a and b, b and c, and a and c, yet cannot capture any conditional dependen- cies, such as between a and b given c. We show in Section 2.2 that even in simple one-dimensional controlled setting where the target b is perfectly predictable from a and c, CLIP performs no bet- ter than random chance. Effective contrastive learning for more than two modalities requires model-agnostic approach capable of learning modality-speciff representations- captures higher-order information between any number of modalitiess-unlke CLIP.
CLIP-yet also
Methodological contributions.
This paper presents Symile, simple contrastive learning approach that captures higher-order information between any number of modalities. Symile provides a flexible, cechitecture-agnostie objective for learning modality-speciin representations. To develop Symile's 's objective, we derive total correlation estimator, employing a generalization of inner products to more than two vectors that allows for the simultaneous contrasting of all modalities and enables zero-shot applications such as classification and retrieval. We then show that the representations produced by Symile for any set of modalities form a sufficient statistic for predicting the remaining modalities not considered in the set. Because it targets total correlation, Symile captures strictly more information than CLIP, guaranteeing performance that matches or surpasses CLIP, except in cases where it known that only pairwise statistics are relevant. Given that such prior knowledge is rarely available, Symile should be favored over CLIP.
Empirical contributions. We demonstrate that Symile outperforms pairwise CLIP on cross-modal classification and retrieval across several experiments including on a multilingual dataset of images, text and audio of over 33M examples and a clinical dataset of chest X-rays, electrocardiograms, and laboratory measurements. We show that Symile retains its advantage over pairwise CLIP even with modalities missing in the data. We publicly release both the multilingual and the clinical datasets, which are specifically designed to test model's ability to capture higher-order information between three distinct nigh-dimensional data types.
2 Background and motivation
In this section, first provide background the original CLIIP objective for two modalities, and describe how it has been extended to additional modalities. We then present a simple problem set up for three modalities that illustrates where pairwise contrastive objectives fall short.
2.1 Pairwise contrastive learning
Given batch of (x. y) pairs, separately encoded by f2 and fe. respectively. contrastive objectives such as CLIP maximize the similarity between representations of correctly paired (poxitive) samples and minimize the similarity between representations of incorrectly paired (negative) samples.
As is now standard in contrastive learning. in order to construct a batch of data, each modality is treated as the anchor in turn and used to construct a set of positive and negative samples. Letting т E IR be a temperature parameter, the CLIP objective when x is the anchor modality is the categorical cross-entropy of correctly classifying the positive pair out of N possible pairs:
1 N
exp (x,) 1 (x.)) log Σラ= exp [f (x:)) ())
(x-+)) (0,T)
(1)
The final CLIP objective is an average of the losses in cach direction: 2 [[(x->>) + ((y-)x)(0 (0,T)] The dot product in Equation (1) serves as a scoring function that is trained assign high values to positive pairs, which are sampled from the joint distribution Px.y, and low values to negative pairs, which are sampled from the product of marginals PxPy-
C (0, =
Contrastive methods are typically designed to maximize the mutual information between and y. which is defined as the Kullback-Leible divergence from the joint distribution to the product of the marginal distributions: I(x; y) DKL. (p(x. y) p(x)p(y)). It has been shown that Equation (1) maximizes a lower bound on the mutual information between x and y [38, 39). This information maximization ensures that the learned representations preserve all correlations between the modalities, which is essential for downstream tasks.
Incorporating additional modalities. In order to learn a joint embedding space for more than two modalities, existing work has applied the CLIP objective in a pairwise fashion [1, 2, 9, 11, 14, 21,

---

33, 34, 43, 44, 47, 52]. For example, Guzhov et al. [19] extend CLIP to incorporate audio alongside image and text, and ImageBind [15] uses CLIP to align image embeddings with embeddings from five other modalities. In the simplest case, for three modalities, the pairwise CLIP loss corresponds to
CLIP (0,7) clr (0,7) cly (0, r) + ClLP (0,
CLIP can either be fine-tuned for downstream tasks operate zero-shot classifier by computing the similarities between the query embedding from one modality and each candidate embedding from the other modality. In the case of more than modalities, this generalizes the pairwise similarities. The resulting similarity scores are used to rank the candidates, and the candidate with the highest similarity the query is chosen [40].
across
2.2 A simple one-dimensional problem for three binary modalities
While contrastive objectives were originally designed for two modalities, the naive pairwise extension of CLIP to additional modalities warrants a deeper analysis. To explore this further, we propose a simple problem setup for the following data generating process:
a, b ~ Bernoulli(O.5). c a XOR b.
Using the pairwise CLIP objective, we fit three affine linear models perform the zero-shot classifi- cation task of predicting whether b is 0 or given a, C. See Appendix for additional details.
Even in this simple one-dimensional controlled setting where the target b is perfectly predictable from a and c, CLIP performs no better than random chance, with an accuracy of 0.5.
CLIP failure analysis. can be shown that even though the variables a, b, e are jointly depen- since is a deterministic function of a and b t-ethey are pairwise independent (Appendix A):
((a; b) = I(b; c) = I(a: c) = O, I(a; ba O.
This explains CLIP's poor performance for the above XOR experiment: the objective maximizes lower bound on the mutual information between pairwise terms, and therefore was not designed to capture higher-order dependencies such the dependence between and b given c. Capturing conditional dependencies like this will require the formulation of a new contrastive learning objective.
3 Learning Symile representations
Instead of targeting the mutual information between pairs of modalities, we target the total correlation between any number of modalities, learning what we call Symile² representations.
Total correlation [50]-the higher-order generalization mutual information is defined as the Kullback-Leible divergence from the joint distribution to the product of the marginal distributions:
TC( x = DKL (p(x1,,..... x || p(x1) )(xx)))
In words, total correlation a symmetric statistical measure that captures the amount of information shared in set of random variables. A higher total correlation implies more dependency among the variables, and a total correlation of zero indicates that the variables are independent.
Total correlation can be decomposed into a summation of mutual information terms. For example, in the case of three random variables,
3- TC(X, y,z)
yx::y) + I(2; (T) + I(x; 2)] + [I(x; I(y;x,
Symile target
= 2 [(x:)) + I(y; I(; [(x [z) + Iyy x) I(x;2|y)
(2)
pairwise information (CLIP target)
higher-onder information
While, discussed, contrastive learning was designed to capture the shared information between modalities, Equation (2) indicates that when there are more than two modalities, the scope of what to capture should extend beyond pairwise information to include conditional interactions (Figure I).
2To specific, we use "righer-order information" to mean information between two random variables given any number of additional random variables in the conditioning set. Symile stands for S Symmetric MultlLinear Embeddings.
3

---

Because it targets total correlation, Symile captures strictly more information than CLIP, guaranteeing performance that matches or surpasses CLIP. except in cases where only pairwise statistics are relevant, with no higher-order interactions whatsoever. In such cases, Symile may be less sample efficient, tracks both pairwise and higher-order information. Unless there prior knowledge that the downstream task relies solely on pairwise statistics, Symile should be chosen over CLIP.
To illustrate when such higher-order information might be relevant, consider again the XOR exper- iment outlined in Section 2.2. Because all the pairwise information terms between b, and are zero, the conditional mutual information
I(acb) b)
I(ac c)
I(ac
f(a: c)
terms constitute the only dependence between the variables to track.
I[b.c a)
I[a,c c b)
The XOR experiment represents an extreme case where the CLIP target is zero, but most real-
b
I(b;c)
I(b:c)
world applications will exhibit combination of
CLIP: pairwise information
Symile: all information
both pairwise and higher-order information. For
example, in order to diagnose acute pancreatitis, one might consider patient's 's clinical history of abdominal pain, elevated levels of digestive enzymes. and imaging results consistent with inflammation. While each of these modalities would provide useful information about the likelihood of pancreatitis (i.e., pairwise information between the modality and the diagnosis is non-zero), none of them alone would be diagnostic of the condition. Similarly, in the case of Parkinson's 's disease, clinical evaluation provides valuable information, along with imaging and blood tests to rule out other conditions, but clinicians rely on the integration of all modalities.
Figure I: formation captured by CLIP (only pairwise) and Symile (both pairwise and higher-order).
An illustrative comparison of the in-
3.1 Deriving multi-sample lower bound total correlation
In order to eventually derive a contrastive objective by maximizing total correlation, we first establish multi-sample lower bound total correlation. This lower bound and, in the next section, the Symile objective are illustrated using three modalities for simplicity, but both can be extended to an arbitrary number of modalities, as shown in Appendix B.
Given a batch N (x, y, z) triples, let
(3) denote the index of the positive triple in the batch. Our goal is estimate TC(x, y. z) given one positive triple sampled from the joint distribution, and 1 negative triples sampled from the product of marginals:
i~ Uniform((1,...,,
x. Vo Zi Px.y. (x. Yo, Z;), X. X.yj p(x)py (y)+:)p2 (Z)zs))
be the sets of all samples of and z, respectively, this sampling procedure describes the following distribution:
Letting YN
(y J.y and ZN [z,, I.N.
y.2
Y.u from posstive sample
negutive samplex
pxx, NN, ZN i= p(x) Py (Y.,
Il (y))))) juts
Il Pa (z5)
(4)
We derive the following lower bound in Appendix B:
Theorem 3.1 (Total Correlation Lower Bound), Given the distributions in Equations (3) and (4), for any value of i and any scoring function g. a multi-sample contrastive lower bound on total correlation is
exp exp.((x,ti ys. log
TC(x,y, 2 log N +
E
(S)
plx. N..YY.C. ZN I=1) 2j=1 exp exp.exxpx....... Y,.
As described in Section 2.1, in contrastive learning each modality is sequentially treated as the anchor, with batch of corresponding positive and negative samples generated for each. Theorem 3.1 treats the anchor modality, but by symmetry holds when y acts as the anchor modality.

---

(a) Symile pre-training
(b) Zero-shot prediction
CANDIONIES
RANGE ENCODER
IAD AARAAA VLLU AARA (AAJA
TEXT ENCODER
Eacer Linad
TEXT ENCODER
T
QULIRYY
(ATTT)) (AMT-) (ATTT)) (ATTT))
W LESpor WET WOIY"
AUDIO ENCODER
Ho walhin fo
AUDIO ENCOGER
A
Figure 2: Symile pre-training and zero-shot prediction the Symile-M3 multilingual dataset. (a) Given a batch of triples, Symile maximizes the multilinear inner product (MIP) of positive triples (in yellow along the diagonal of the cube) and minimizes the MIP of negative triples. (b) The model selects the candidate image with the highest similarity to the query audio and text.
Notice that the term inside the expectation in Equation (5) is the categorical log likelihood of correctly identifying the index of the positive triple in the batch, where the scoring function (or critic) g is trained to assign high value to positive samples and low value to negative samples. In Appendix E, we show that the optimal scoring function g" is equal to the instantaneous total correlation up to additive constants:
Lemma 3.2. For some K. > O, the y that maximizes the lower bound
log exp gX. yo. z:)
TC(x,y,2) 2 log +
E
N.
|1:))
E/=1 exp exp ,,)
is
Px, x Xx y. p(x Py (y (z)
9 (xxxy,2) = x 12-1oy log
We show in Appendix B.3 that, as N gets larger, the total correlation lower bound closes for the optimal scoring function g". This implies demands more computation but results in a tighter bound.
petpuuaatiooalstttiittiii trade-off: larger batch size
3.2 The Symile objective
We now derive the Symile loss by maximizing the total correlation lower bound in Theorem 3.1.
Instead of using the dot product as a scoring function, as CLIP does, Symile uses its generalized form: set of vectors. We call this the multilinear inner product (MIP): { = Eb= I1 Ti As a scoring function, the MIP strikes a balance between computational simplicity and expressive power: it represents generalizations of the dot product to more than two modalities, and the vector multiplication ensures it is expressive enough to model any joint statistic.'
the coordinate-wise sum of the glement-wiss product
of the simplest possible
Given a batch of N' positive triples (x, (xi, y y' z,),, and letting R+ be temperature parameter, the Symile loss is the negative of an empirical estimate of the expected log likelihood Equation (5):
Y;, zz), each with N corresponding negative triples
g(x-t.22 (0,T)=
1
N'
exp (()2(()))) 16 (()))))
log
2
exp
Σ exp (21)
*Note that the MIP measure similarity defined by the joint distribution of modalities, rather than a measure of the geometric similarity of the modalities' representations. For example, a large MIP for Symile representations ry, Iz indicates that the sample (x, y. z) has high probability under the joint likelihood; it provides information about whether 1x, Fy, Fz equal to another.
5

---

Minimizing Equation (6) optimizes the lower bound on total correlation by maximizing the MIP of positive tuples and minimizing the MIP of negative tuples (Figure 2a). See Appendix B.4 for the Symile objective generalized to any number of modalities.
As is done with CLIP, the final Symile loss is average of the loss terms where each modality is treated as the anchor in turn:
c(x.y.2) 2) (0, (xxxx) po txx)
(0, (0,7) p(- (0, (0,7)]
Symile
Efficient negative sampling. In the sampling procedure described in Section 3.1, negatives samples for the non-anchor modalities are drawn independently for each positive triple, which can be intensive in terms of both computation and memory. Instead, for efficiency, negative sampling can be approximated within a batch by forming negative tuples from non- matchat combinations of the non-anchor modalities.
Algorithm 1 Pseudocode for implementation of Symile with ((V) negative sampling
compute [n, m] logits from x ty, 2)
def get_ get_logitte y, x):
MIP_poa Ix
(uu #[n]
y_shuffled
(n)
R_shufrted > [.pp. wwnndor.perntatiiin (n2]
BiP_neg x ty_shuéfled v s_ehuffled; vT + n]
return np.shere tap.eye (nl, XIP_pGs, KIP_neg]
Approximating negatives within batch is straightforward with two modalities, but in the
v, 0, er 12-normall enbeddings, each [n, din) case of more than two modalities, both how neg- atives are formed and how many are used be- come design choices. At one extreme, one could generate N negative triples for each posi- tive by considering all possible combinations of the two remaining non -anchor modalities. This approach, which we call ( can be compu- tationally and memory intensive. Instead, any subset of these negatives can be used for sampling. For instance, a more efficient approach, which we refer to as ONN,, involves randomly permuting the non non-anchor modalities within the batch, providing each data point with N - negatives. The cube in Figure 2a illustrates the O approach and Algorithm presents pseudocode for the O(N) approach, both for three modalities.
symile_losssC, u, M):
legits_v_ur np:exp(t) get_logits M]
1egita_o_vv ep.exp(t) get_logits
legita_M_vu ax.exp get_logits
labels np.arange (n))
Iesa_v_uM - oe_loss (logsta_v_uw, labela]
less_u_ve # ce_less (logits_u_yw, labels)
1esa_M_v0 = celicas (logsts_M_ve, labels)
return (Loss_y_ux + loss_j__w loss_j_vvlt/3
Missing data. The Symile objective is defined for data in which all modalities are observed. However, in practice, datasets often include samples where not all modalities are available. This raises the question: during training how should one incorporate data points for which only a subset of modalities is observed? Symile can be easily adapted to such missingness by adding extra dimensions to the encoder inputs that indicate whether or not a modality is missing, ensuring that missing data points are out-of-support. This approach allows Symile to model dependencies between whichever modalities are observed within a sample. We show in Section 5.2 that Symile retains its advantage over pairwise CLIP even with modalities missing in the data.
3.3 Learning sufficient statistics with Symile
An important property of Symile is that it learns sufficient statistics, which is central to the represen- tations' effectiveness for downstream tasks.
Theorem 3.3 (Symile Sufficient Statistics). Let x, be three random variables whose optimal representations when trained using Symile are f fy) (y), fz (z). respectively. The element-wise product of any subset of the representations is sufficient statistic for predicting the remaining random variables.
For example, (x) ④ (z) is a sufficient statistic for predicting y, which can be expressed using the following conditional independence statement:
y IL x, 2 (x) O (z).
The proof can be found in Appendix G. The independence statement in Theorem 3.3 tells us that the element-wise product the Symile representations of any subset of modalities contains all the information required to predict the remaining modalities. In other words, once Symile representations have been computed, access to the full data is no longer needed. Theorem 3.3 confirms Symile's ability to learn efficient modality-specine representations for downstream tasks.

---

3,4 Zero-shot prediction using the scoring function
Just as with CLIP, the optimal scoring function g (Lemma 3.2) can be used to predict one of the modalities y Y' using instances of the other modalities 2, 2. If ply) is uniformly distributed, then the scoring function can be used to rank the candidates for y: arg maxyey p(y = y x, x2 arg yay 8 (x, U,
However, this zero-shot approach, whether applied to Symile or to CLIP, does not lead to the Bayes optimal prediction and, consequently, does not always yield reliable results when ply) is not uniformly distributed (see Appendix H for a detailed discussion). To address this issue, we can instead compute the desired conditional probability directly using the scoring function:
Theorem 3.4 (Conditional Distribution using the Scoring Function). Let y, 2 be three random vari- ables whose optimal representations when trained using Symile are (x),,, (y), (z), respectively, Let the MIP (xx()), f (y), (z)) be the scoring function. Then,
exp (x))(x) fy),,, ply)
ply x.
Jy exp (f2 (y)),, (2)) ply)dy
The proof is provided in Appendix
If the marginal distribution of y is known, we could then perform zero-shot classification in one of two ways. When the distribution p(y x. z) itself is of interest, as often the case in healthcare [10], we could compute p(y x, z) directly, following Equation (7). Alternatively, if only predictions are needed, we could use
(x),
+ poy)
to rank the possible values for y, as discussed further in Appendix H. If the marginal distribution of y is nor known, then because (x) @ [z) is a sufficient statistic for predicting y (Theorem 3.3),
could instead use (x) f (z) to train simple model to predict any property of y. s(y): p(s(y) (x) @ (z)).
Note that although the above discussion centers Symile, it applies equally to CLIP and its own scoring function, the dot product.
4 Related work
Contrastive learning beyond two modalities. As discussed, previous work has extended con- trastive learning to multiple modalities by applying CLIP to pairs of available modalities. Tian et al. [49] distinguish between two such pairwise approaches: core view and full graph. The core view strategy fixes one modality and then averages the loss terms between that primary modality and each of the other modalities [I. 11, 44]. ImageBind [15] exemplifies this approach, using CLIP to align image embeddings with embeddings from five other modalities: text, audio, depth, thermal, and motion sensor data. One advantage of this strategy is that it avoids the need for datasets with all modalities (though each dataset must still align with a primary modality). As discussed in Sections 3.2 and 5.2, Symile representations can be learned even with modalities missing in the data.
The full graph strategy- which we have referred to as pairwise CLIP in this paper-is is to consider all (M) contrastive losses [9, 14, 33, 34, 433. For example, Guzhov et al. [19] extend CLIP to include audio with text-to-image, text-to-audio, and image-to-audic losses. While this pairwise strategy captures strictly more information than the one used by ImageBind, neither pairwise approach is able to capture the higher-order information that Symile does.
Pairwise CLIP has also been applied to architecture-specife fusion models that simultaneously process modalities to capture cross-modas interactions [2, 21, 52]. For example, Shvetsova et al. [47] train a Transformer to accept any number of modalities, using a weighted sum of contrastive losses across all input combinations. Such fusion approaches face a combinatorial explosion not only in the number of weighting coefficients to tune, but also in the number of forward passes required per batch. In contrast, Symile is ruchitecture-agnosti and can learn modality-specinn representations.
Targeting higher-order information with contrastive learning. The use of contrastive meth- ods to target higher-order information has been explored primarily within the context of multiple augmentations of the same data. For instance, Bai et al. [5] derive total correlation estimator by
7

---

1.00
Mutual Information
3
Accuracy
0.75
2
0.50
Symile
I(a: b) [(b; (a: b c) [(;
0.25
CLIP
Random
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 3: The performance gap between Symile and CLIP on binary synthetic data (left) is consequence the changing information dynamics between the variables as P moves from 0 to (right). Mean accuracy is reported across 10 bootstrap samples of the test set.
recursively decomposing total correlation into summation of mutual information terms, to which variational estimators are applied (in contrast, Symile optimizes only a single term when targeting total correlation). They then use their estimator to maximize the total correlation between four text augmentations. Shidani et al. [46] develop pairwise contrastive approach for image representation learning by generalizing lower bound on mutual information to one-vs-rest mutual information across multiple augmentations. Liang et al. [31] maximize the information in two modalities for specific downstream task by targeting higher-order information.
The relationship between these studies and our work is analogous to that between SimCLR [12] and CLIP. SimCLR popularized the use the InfoNCE mutual information estimator for contrastive learning two data augmentations. Building this framework, CLIP applied the approach to distinct modalities, where representations are learned separately for each modality using any encoder. Similarly, while existing work leverages total correlation or mutual information estimators for multi- augmentation contrastive learning, to our knowledge only pairwise applications of CLIP have applied such estimators to more than two distinct modalities. Our work parallels the contributions of InfoNCE and CLIP for cases involving more than two modalities: like InfoNCE, we develop a simple estimator that recovers all possible information between any number of modalities, and like CLIP, show how this estimator can be used to learn modality-specifff representations using any encoder.
Experiments
In this section, we empirically evaluate Symile on cross-modal retrieval tasks in three settings: a syn- thetic dataset, multilingual dataset encompassing text, images, and audio, and clinical dataset with chest X-rays, clectrocardiograms, and blood labs. Throughout our experiments, we use pairwise CLIP as baseline comparison since, as outlined in Section 4, represents the only architectuee-agnosicc approach that applies contrastive objectives to more than two modalities. We release all datasets and code used in these experiments https: h/tttttbb.tbb///////gg thub. com/ ra jaajeessalae lab/ sym
5.1 Synthetic data
Building the illustrative XOR experiment from Section first test Symile on synthetic dataset drawn according to the following sampling procedure:
aj by Bernoulli(O.5), i ~ Bernoulli()), Cj (a, XOR by)² b,) a.-1)
We fit three affine linear functions that map b, R representations ra. rb, Fe E R6 respectively, and evaluate the model's ability to correctly predict ry given the pair (ra.
Results. Figure 3 (left) compares Symile and CLIP across varying values of p. Both models start with mean accuracy 0.0324 + 0.001 (SE) at = As ő increases, Symile's accuracy progressively climbs, reaching perfect accuracy at ÿ + 0.0 (SE). In contrast, CLIP's accuracy remains nearly constant, barely surpassing the baseline random guessing rate of 0.031 (2/32).
This performance gap is a consequence of the changing information dynamics between the variables as P moves from 0 to 1, as shown in Figure 3 (right). When و = 0, b shares no information with a and both models incapable
eithe pairwise or conditionally-rendeeii As increases, the higher-order I(a; b c) and [(c; b) a) rise, driving a corresponding improvement in Symile's s performance. However, because the pairwise I(a; b) and [bb; c) are always zero, there no value of at which CLIP is able to predict Tb from (Ia,
predicting rb from (ra, re).

---

(a) Data generation
(b) Fully-observed data
(c) Data with missingness (w 2)
IAAGE AUCIO ENGLISH
1.00
1.00
Mrs.. Dallway sald
she would buy the Rowers harself
Accuracy
0.75
Symie CLIP
Accuracy
0.75
0.50
0.50
TEXT includes APPLE in ENGLISH
0.25
Rendom
0.25
Symil CLIP
Ganer_apple_ _nf_oooddd..... FE.8
0.00
0.00
100
12.5
4.3
% training data fully observed
Figure 4: (a) Data- Data-generating process for Symile-M3-5. (b) Comparison of Symile and CLIP on the three versions of Symile-M3 (w E {2, 10)). Random chance is 1/1000. Symile successfully leverages joint information between the modalities, whereas CLIP limited to pairwise information, resulting in accuracies bounded by 1/w. (c) Symile outperforms the CLIP baseline on Symile-M3-2 across varying levels of completeness in the training data, Both plots report mean accuracy across 10 bootstrap samples of the test set.
5.2 Symile-M3: multilingual dataset
We now evaluate Symile on multilingual dataset comprising 33 million (audio, image, text) samples. The dataset, Symile-M3, is specifically designed to test a model's ability to capture higher- order information between three distinct high-dimensional data types: by incorporating multiple languages, we construct a task where text and audio are both needed to predict the image, and where, importantly, neither text nor audio alone would suffice.
Dataset design and model setup. Let w represent the number of languages in the dataset. An (audio, image, text) sample is generated by first drawing short one-sentence audio clip from Common Voice [4] spoken in one of w languages with equal probability. An image is drawn from ImageNet [45] that corresponds to one of 1,000 classes with equal probability. Finally, text containing exactly w words is generated based on the drawn audio and image: one of the w words in the text is the drawn image class name in the drawn audio language. The remaining w - words are randomly chosen from the ImageNet class names and written in one of the languages such that there is no overlap in language or class name across the w words in the text. The words are separated by underscores, and their order is randomized. We release three versions of the dataset: Symile-M3- Symile-M3-5, and Symile-M3-1D corresponding to 2, 5, and 10 languages (w). Figure 4a shows an example of the data gentaatenertiigg process for Symile-M3 M3-55 For each of the three datasets, 10M training, 500K validation, and 500K test samples were generated.
We use pre-trained encoders, freezing all parameters except for those in the text encoder's embedding layer and first encoder layer, which are fine-tuned. We train three linear projections to map each encoder's representation to the same 8192-dimensional space. The Symile loss is trained with O(N) negative sampling. See Appendix for details.
Evaluation and results. We evaluate the learned representations the zero-shot retrieval task of finding an image of the appropriate class given the audio and text. The most probable image for a given query audio and text pair, selected from all possible candidate images in the test set, is that with the highest similarity score (Figure 2b). Symile-M3 was designed to ensure that neither text nor audio alone would suffice to predict the image. Therefore, success on this zero-shot retrieval task hinges on model's 'ss ability to capture joint information between the three modalities.
As shown in Figure 4b, Symile successfully leverages this joint information, with mean accuracies of 0.939, 0.919, and 0.882 on Symile-M3-2 M3-22 Symile-M3-5, and Symile-M3-10, respectively, calculated across 10 bootstrap samples of the test set, all with standard error less than 4.0 x 10-4 4. In contrast, CLIP, which captures pairwise information between image and text, can only predict an image randomly from among the w class labels present in the text, resulting in mean accuracies of 0.473, 0.187, and 0.094 on Symile-M3-2, Symile-M3-5, and Symile-M3-110 respectively, all with standard error < 3.01 10-4. Because CLIP cannot distinguish between the class labels in the text using the audio language, it can only pick a class label at random, bounding its accuracy by 1/w.
Missing data. We also train Symile on variant of Symile-M3-2 where each modality is indepen- dently missing with probability 0.5 or 0.65, corresponding, respectively, to probabilities 0.125 and 0.043 of a complete data sample in the training set (see Appendix for details). As before, the test set consists of complete triples. As shown in Figure 4c, even when only 12.5% of the training data is

---

complete, Symile achieves mean accuracy of 0.906 + 3.4 x 10 (SE), far outperforming the CLIP baseline accuracy of 0.473, despite the adverse effect of missing modalities. Notably, when less than 5% of the training data is complete, Symile still exceeds the CLIIP baseline.
5.3 Chest X-ray prediction using electrocardiograms and laboratory measurements
Zero-shot retrieval widely used in the evaluation of representation learning for healthcare [6, 22, 29, 51,51,,,, 56]. In this section, evaluate the Symile objective on comprised of chest X-rays, electrocardiograms, and blood labs from MIMIC-IV [17, 24, 27] and MIMIC-CXR [25, 26]. Since ECGs and labs are both safer than CXRs, this experiment explores whether an ECG and labs collected at admission are predictive of a CXR taken shortly thereafter.
clinical dataset
Dataset design and model setup. Each data sample includes an ECG reading and blood labs taken within 24 hours of the patient's admission to the hospital, and CXR taken in the 24- to 72-hour period post-admission (Figure 5a). Our analysis focuses on the 50 most common blood labs, with each sample containing at least one.
(a) Data generation
(b) Zero-shot retrieval
-24
Hospital
244
+72
0.4
Accuracy
0.3
ECG
We split our dataset (11, 622 admissions) into vrain/valiation development set (95% of pa- tients) and a test set (5% of patients), ensuring there is no patient overlap across the splits. Fol- lowing previous work, we use the ResNet-50 and ResNet-18 architectures [20] for the CXR and ECG encoders, respectively, and a three- layer neural network to encode the blood labs. All encoders are trained from scratch, and three linear projections map each encoder's repre- sentation to the same 3192-dimensional space. Given the limited size of the dataset, the Symile loss is trained with (NN2) negative sampling to mitigate overfitting. See Appendix for details.
0.2
0.1
Distelata Leiin
0.0
Symile CLIP Random
Figure 5: (a) Each sample of Symile-MIMIC in- cludes an ECG and blood labs taken within 24 hours of the patient's 's admission to the hospital, and CXR taken in the 24- to 72-hour period post- admission. (b) Retrieval accuracy for identifying the CXR corresponding to given ECG and labs pair. Results are averaged over 10 bootstrap sam- ples, with error bars indicating standard error.
Evaluation and results. We evaluate the learned representations on the zero-shot retrieval task of finding the most probable candidate CXR for a given query ECG and labs pair according to the similarity score. For each query ECG and labs pair in the test set, we sample nine negative CXR candidates from the remaining test samples, so that that each query has a total of 10 candidates: one positive (the true corresponding CXR) and nine negative.
In Figure 5b, we report mean accuracy for Symile and CLIP over 10 bootstrap samples of the test set. While both models surpass random chance (0.1), Symile achieves an average accuracy of 0.435 + 0.007 (SE), outperforming CLIP's 0.387 + 0.003 (SE). These results correspond to a 12.5% increase in accuracy for Symile over CLIP.
Conclusion
This work presents Symile, simple contrastive learning approach that captures higher-order infor- mation between any number of modalities. Symile provides a flexible, architecture-annssii objective for learning modality-specifft representations, maintaining the simplicity of CLIP while delivering superior performance, even in cases of missing modalities. Because it targets total correlation, Symile captures strictly more information than CLIP, guaranteeing performance that matches or surpasses CLIP, except in cases where known that only pairwise statistics are relevant, Given that such prior knowledge is rarely available, Symile should be favored over CL.IP.
Future work. (1) The sigmoid- based loss function SigLIP [54] was recently introduced as memory-efficient alternative to traditional softmax-bass based contrastive objectives. potential avenue for future work would be to adapt Symile, and its use of the multilinear inner product, to this sigmoid loss. (2) The proposed implementation of Symile relies on an approximation for negative sampling, and future work could examine how this approximation scales when applied to settings with more than three modalities. (3) Future work could integrate pre-trained Symile representations into multimodal large language models, enabling them to capture higher-order information between modalities.
10

---

7 Acknowledgements
We are especially grateful to Charley Crissman for his invaluable and meticulous feedback on every aspect of the paper, from the proofs the code. We would like thank Nick Murphy (Pantograph) and Madeleine Murphy for their thoughtful guidance and indispensable support in preparing the illustrative figures. We thank Wanqian Yang for his helpful suggestions and careful editing of the paper. We also thank Leon A. Gatys, Eran Halperin, Andrew C. Miller, Charles Peyser. Pranav Rajpurkar, Ardavan Sacedi, and Jagadish Venkataraman for engaging valuable discussions throughout this work. This work was partly supported by the NIH/NHLB Award R01HL 488248, NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation, and Responsibility for Data Science, NSF CAREER Award 2145542, NSF Award 2404476, ONR N00014-23-1-266 Apple, and Optum.
References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian. Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 24:2420--24221,222 2021.
[2] lean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
[3] Jean-Ban Saan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716 2022.
[4] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A mussively-multiliign speech corpus. arXiv preprint arXiv: 2019.
[5] Ke Bai, Pengyu Cheng, Weituo Hao, Ricardo Henao, and Larry Carin. Estimating total correlation with mutual information estimators. In International Conference on Artificial Intelligence and Statistics, pp. 2147-2164. PMLR, 2023.
[6] Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Garzia Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. Learning to exploit temporal structure for biomedical vision-language processing. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15016- 15027, 2023.
[7] David Barber and Felix Agakov. The im algorithm: variational approach to information maximization. Advances neural information processing systems, 16(320):201, 2003.
[8] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, al. Making the most of text semantics improve biomedical vision lasiuanlanguuge processing. In European conference on computer vision, pp. 1-21. Springer, 2022.
[9] Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, et al. Multimodal clustering networks for self-supervised learning from unlabeled videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8012-8021, 2021.
[10] Irene Y Chen, Shalmali Joshi, Marzyeh Ghassemi, and Rajesh Ranganath. Probabilistic machine learning for healthcare. Annual review of biomedical data science, 4(1):393-4155 2021.
[11] Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: -udio--auoo----aaaeeee omni-perception pretraining model and dataset. arXiv preprint 284 2023.

---

[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
[13] Alexis Conneau. Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek. Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross- coos-llngual representation learning at scale. arXiv preprint drXiv:1911.02116. 2019.
[14] Kevin Duarte, Brian Chen, Nina Shvetsova, Andrew Rouditchenko, Samuel Thomas, Alexander Liu, David Harwath, James Glass, Hilde Kuehne, and Mubarak Shah. Routing with self-attention for multimodal capsule networks. arXiv preprint arXiv:2112.0077 2021.
[15] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
[16] A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components a new research resource for complex physiologic signals. Circulation, 101(23)=e215 2000 (June 13). Circulation PMID:1085218; doi:
Electronic Pages: 10.1 101.....3..
[17] Brian Gow, Tom Pollard, Larry A Nathanson, Alistair Johnson, Benjamin Moody, Chrystinne Fernandes, Nathamiel Greenbaum, Seth Berkowitz, Dana Moukheiber, Parastou Eslami, Tanner Carbonati, Ashish Chaudhari, Elizabeth Herbst, Dana Moukheiber, Seth Berkowitz, Roger Mark, and Steven Horng. Mimic-iv-ecg: Diagnostic electrocardiogram matched subset, 2023. .0.
URL /physsi///iiiiim onet. orgo htt/://ttteet///t// wttpt//miiiiiiiiiiiimm mwm-mmiiiiieeee
[18] Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala, and Lerrel Pinto. See to touch: Learning tactile dexterity through visual incentives. arXiv preprint arXiv:2309. 12300, 2023.
[19] Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980. IEEE, 2022.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. 2016.
[21] Jingjia Huang, Yinan Li, Jiashi Feng, Xinglong Wu, Xiaoshuai Sun, and Rongrong Ji. Clover: Towards unified video-language alignment and fusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11856-14866 2023.
[22] Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Pro Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 3942-3951, 2021.
[23] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
[24] Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Homg. Leo Anthony Celi, and Roger Mark. Mimic-iv, 2023. URL ht https:://// http:///yyyyitt..eeeoee org wwwtent/ttttt/cccccccc ht///mimiiii//i/ cilmiciiv// 2/.
[25] Alistair Johnson, Matthew Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg chest radiographs with structured labels, 2024. URL. https: http://wwpp.ttc www.physionet...mmmmmm wwg.eiinnottooooooooooooooo content /ttm//mmiaaiaa//ee /p.g2.1.0.
12

---

[26] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Yifan Peng, Zhiyong Lu. Roger G Mark. Seth J Berkowitz, and Steven Horng. Mimic-cxt-jng, large publicly available database of labeled chest radiographs. arXiv preprint (rXiv:1901.07042, 2019.
[27] Alistair HW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
[28] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: large-scale un-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12944 12945, 2024.
[29] Sravan Kumar Lalam, Hari Krishna Kunderu, Shayan Ghosh, Harish Kumar, Samir Awasthi, Ashim Prasad, Francisco Lopez-Jimenez, Zachi Attia, Samuel Asirvatham, Paul Friedman, et al. Ecg representation learning with multi-modal ehr data, Transactions on Machine Learning Research, 2023.
[30] Eunjung Lee, Saki Ito, William R Miranda, Francisco Lopez-Jimenez, Garvan C Kane, Samuel Asirvatham, Peter A Noseworthy, Paul A Friedman, Rickey E Carter, Barry A Borlaug, et al. Artificial intelligence-enabled eog for left ventricular diastolic function and filling pressure. npj Digital Medicine, 7(1):4, 2024.
[31] Paul Pu Liang, Zihao Deng, Martin Q Ma, James Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy, Advances in Neural Information Processing Systems, 36, 2023.
[32] lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Leurning Representations, 2019.
[33] Sijie Mai, Ying Zeng, Shuangjia Zheng, and Haifeng Hu. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis. IEEE Transactions on Affective Computing, 2022.
[34] Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Alireza Dirafzoon, Aparajita Saraf, Amy Bearman, and Babak Damavandi. Imu2clip: Multimodal contrastive learning for imu motion sensors from egocentric videos and text. arXiv preprint arXiv:2210. 14395, 2022.
[35] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-ing image pre-training. In European Conference on Computer Vision, pp. 529-544. Springer, 2022.
[36] Kevin P Murphy. Probabilistic machine learning: an introduction. MIT press, 2022.
[37] Radford M Neal. Meme using hamiltonian dynamics. arXiv preprint 1206 2012.
[38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 080710377778 2018
[39] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, pp. 5171 -- 5180. PMLR, 2019.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning. pp. 8748-8763. PMLR, 2021.
[41] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. International Conference Machine Learning, pp. 28492-28518 PMLR, 2023.
13

---

[42] Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models, In Interna- tional conference on machine learning, pp. 324-333. PMLR, 2016.
[43] Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, et al. Avlnet: Learning audio-vistio language representations from instructional videos. arXiv preprint urXiv:2006.0919, 2020.
[44] Ludan Ruan, Anwen Hu, Yuqing Song, Liang Zhang, Sipeng Zheng, and Qin Jin. Accommo- dating audio modality in clip for multimodal processing. arXiv preprint arXiv: Siti:2303.0659,, 2023.
[45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115()::1111 2015. doi: 10. 1000//s1263--15-000
[46] Amitis Shidani, Devon Hjelm, Jason Ramapuram, Russ Webb, Eeshan Gunesh Dhekane, and Dan Busbridge. Poly-view contrastive learning. arXiv preprint 2iti::20030049900 2024.
[47] Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Roge- rio S. Feris, David Harwath, James Glass, and Hilde Kuehne. Everything at once multi-modal fusion transformer for video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20020, June 2022.
[48] Raghav Singhal, Mark Goldstein, and Rajesh Ranganath. Where diffuse, how dif- fuse, and how to get back: Automated learning for multivariate diffusions. arXiv preprint arXiv:2302.07261, 2023.
[49] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
[50] Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1):66 82, 1960.
[51] Kevin E Wu, Howard Chang, and James Zou. Proteinclip: enhancing protein language models with natural language. bioRxiv, pp. 2024-05, 2024.
[52] Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to languaae representation alignment. arXiv preprint arXiv: 06430, 2022.
[53] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18123-18133, 2022.
[54] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv preprint arX1::2303./5343, 2023.
[55] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunn tuned audio-visual language model for video understanding. arXiv preprint arXiv:2336.00885, 2023.
[56] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale -peeife pretraining for biomedical vision-language processing. arXiv preprin! arXiv:23033000915 2(3):6, 2023.
[57] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2-25, PMLR, 2022.
14

---

A Pairwise independence in binary XOR experiment
In this section, we show that the three variables in the XOR experiment in Section 2.2 are pairwise independent.
Let
(emoouli(0.5)
c
a XOR
First, will show that (ernoull(O0.)::
P(c= 1)= = Σ
b)
a.b
0.25
Σ P(c=1la=abbi
= 0.25
[P
- 1.b 1))
0.25
[0+1+1+0
0.5.
Next, we will show that c a ~ Bernoulli(0.3):
P(c= 1/a) = c- 1)P(c 1) 0.5.
P(a)
By symmetry, since (em5oull(O0.),, then e b (ernoull(O.5)..
15

---

B Total correlation lower bound
Our goal in this section is to derive a lower bound on TC (MM)
We start by describing in Appendix B.1 the sampling procedure for a batch of (mi..... mm) tuples. In Appendix B.2, we derive the desired lower bound in Theorem 3.1 (our proof was inspired by Poole et al. [39]'s derivation of the InfoNCE lower bound, which does not rely on approximation used by Oord et al. [38]). In Appendix B.3, we show that the bound is closed at optimality. Finally, we use the lower bound to define the Symile objective in Appendix B.4.
B.1 Sampling procedure
We start by describing the sampling procedure for the batch of N M-ttupls. In contrastive learning, the objective of matched data. In order to construct these samples, each modality is treated as the anchor in turn, and then for each anchor modality corresponding set of positive and negative samples is generated.
differentiate between positive and negative samples constructed from given batch
Let y be arbitrary in f1...., let my denote the anchor modality, and let m denote the M 1 non-anchor modalities. Let
i i Uniform((1......)
(8)
denote the index of the positive M-tuple in the batch.
We draw m, from p( pm,.) and m m.3 from Pm m, m y. m,). We call (m m m- our
positive tuple.
For each non-anchor modality myyr, we draw samples of may from Pm, (me,) ), so that there N 1 total negative tuples (m.,, m_yj-
Let M mm 7,m m be the set of all samples of non- non-anchor modalities in the batch.
This sampling procedure describes the following distribution:
m from
m_ from positive sample
nogative samples
i=
(m Pm
/m.
m
Pmu (me.)
(9)
Letting M (m J. be the set of all samples of modality my in the batch, the following properties hold by Lemma C.I:
p(m,) | i) p(m.)
1) II Pmu (me,) = p(MM).
B.2 Lower bound on total correlation
We now derive lower bound on TC(M)...., mx), which we express using the following notation for convenience:
TC mM) TC (me)et)) DKL. (p(m., m_)) ((m,) p(mm))
En
Theorem B.1 (Total Correlation Lower Bound). Given the distributions Equations (8) and (9), for any value of and any scoring function g. a multi-sampe sample contrastive lower bound total correlation is
exp e x.) J.
TC (melet)) 2 log N +
E
log
p(m,M_,, lime)
ΣN exp expg(mm,)) m.)))
Proof. By Lemmas C.I and D.I, have
TC(M, (me TC(M, (Mrtet =)
by Lemma D.I
16

---

DKL ( My i = i) p(m, ii= i) II Edy
p(M, i= i))
DKL (p(m, M i) p m II p(M)))
by Lemma C.I
Exty
total correlation (TC) likelihnod ratio
p(M m
log
p(m, M
i=i) 1I p(M))
We call the above likelihood ratio in blue the total correlation (TC) likelihood ratio. We introduce a variational approximation qM- m,, i i) that has the same support as p (M. m, = i):
(M
myy i = i)
2(M
my. =
TC (me)et))=
log
p(m, M
I=i)
(
my, i
Il Mi p(M))
p PMM
my, 1= i)
(MM m m i=
log
+
E
log
p(m, .M l=()
9 M
m,, i=
p p(m, .MM
I=t)
Ie p(M))
DKL (p((M
m,,i- i) || g| my, i))
p(m, i=t)
log ((M 1eyyy((M, my i=
my, i = i)
E
p(m,, .M
imi)
(M
log
(10)
M
He ((M))
since the Kulbback-Leibler divergence is always non-negative. Note that Equation (10) is the total correlation variant of Barber & Agakov [7]'s lower bound mutual information.
We choose set
(M_, my i= 1) ne Ie,, p(M,)
exp f(i, my M_,)
(11)
((
where
C(my; 1ley, p(M))
exp f(i m,, M_,)
is normalizing constant,
exp.(m, m m...))
S(i, my M = log
(12)
N 25=1 exp exp.gmm, m))
and is an arbitrary function.
Plugging Equation (11) into Equation (10) gives
He p(M)) C(m,,))
exp f(i, m,,,,,1.. M
TC {me)e4))
E p(m,,MM.. M Mi.iiii))
log
11ey, p(Me)
E MM
f(i, my M M,))
E p(m,)
log C(m,
(13)
li=i)
Since log(t) _/2 + log a for all b, > O, we see that
log C(m.: 8) - C(m,, + log - 1
1 C(my; 1),
e
which, continuing from Equation (13), gives us
(0 (14)
TC(M,, (melet7) 2
E
f(i, m, N.mmM.,)) M.,)
E v(m.)
p(m, M_, ||=)
17

---

Substituting the formulas for f and C' into Equation (14),
(C (me)een))
E p(m, M
1 + log
exp myy m-y.a)
1 E
exp (1+ log
exp gln I m_3.d)
(1=()
N 2===1 exp g m m- m-))))
W
C p(m.)) Hesy pides)
N
ク== exp g
exp expg(my m_ m_,,4)
exp expg(my mn 7,
E (1=1)
log
plm,,M
2j=1 ( m -7d)
p(mn)) Meegy p(Me)
2 expg m_5))
Now take the expectation of this bound over p(i):
g p(1)
[TC(m,, (melex>)]
exp g (m., m m_
exp.g m m_7,k)
R
1
log
p(h
plmn M
1=m)
N En=1 expg(m,, m 75)
1
p(m,) Iee p(M))
N Σn exp ( m
TC (melley)
exp exp.gmm, m_..)
exp i m
2 1
log
E
p(m,, M 7.0)
2nj=1 exp gg m 7(7)
p(pp(mm)) Herr pM,
N
N 2سj= exp
exp ( m_y.i)
i= expg(my: m
=1+
E
log
E
P(my. M 9.1)
N
2i N ((pg(myy..... m m-))
p(mn)) p(Me)
N
2
expg(m, mーرر
exp g (my m-
log
p(m,..M M 713)
N
2-j=1 exp (m,, m.).)
exp g m
=log N +
E
log
p(m,, M 1)
exp (m m
exp
log N plm,, 11=0)
E
log شز= exp g
(15)
Notice that the index does not change the expected value in Equation (15). To see why, consider two values and i':
exp (mp. m...
E p(m, .M
log
i=1)
2 exp ( m.ys)
log ( m. m..)))
p(m, Pm
m. (m m,))
Pme (me,s)
dm, dM-... (16)
Σソ= exp (m,, m m.)))
-f p(m,))m
exp ( m-y.e)
Im (m
my)
Pme (me.j)
log
dm, dM (17)
ا== exp
exp g m. m
E MM limss)
log
E exp (mm.. m m-)))
Swapping the names of integration variables does not change the integral from Equation (16) to Equation (17).
Therefore, continuing from Equation (15), the lower bound can be written for any value of i as
exp (m, m..s)
TC(M,, {me)e47) 2 log N
log Σ)= exp g (my m m
p(m,, M
limi)
exp expg(mm.. m
log N
E M M
log
y li=))
En exp g (m, m m
The extra negative samples are auxiliary random variables for computation in that these random variables do not appear the target total correlation. This is analogous to the auxiliary random variables used in approximating posteriors and probabilistic modeling [37, 42, 48).
18

---

B.3
Closing the lower bound
There are two inequalities the derivation for the total correlation lower bound in Theorem :: Barber & Agakov gap in Equation (10) and the log ratio gap in Equation (14). In this section, show that each of these bounds is closed at optimality.
The Barber & Agakov gap in Equation (10) is closed when
DKL (pL( m = i) || (MM,, m,, 1 = i))
Therefore, closing the Barber & Agakov gap requires
p(M my, i = i) q(M 1m,
Ile,, p(Me) C(m..) exp fii, m,,
p(M my, i=
1 exp f(i, m. M.)
1l44 (M,)
(m,,,))
pMM
my, 1 =
log
f(i, my M M,,) - log C(m,::
11 p(M))
p(M,, my.i i He p(M))
[(i, (i, M_,) log
+ log C(m,,
(18)
The log ratio gap Equation (14) is closed when
C(m,. =c.
Then by Equation (18), the lower bound is closed if
p(M my i == i)
fi, my M,,) log
+ 1.
(19)
M p( (M+)
By Equation (12),
exp g m. m mm
[(i, n,m,,,M.. M_,) = + log
N j=1 exp (m m m.)
log p(M,, my i = +1
by Eq. 19
11 p(M))
exp m. m m-
log p(M_,, 1= i)
log
N Eド= exp g (m, m-.j)
M p(Me)
exp po (m, m )
pMM,, i= Iley, ((Me)
N 1 exp m. m-)))
1
p(m,, Mi.li=/ i)
p(m,) ITet, p(Me)
p(m, 10 Im,
mm (m p(m..)
He Ilg (me
by Lemma C.I
1= Pene (m)
p(m,)) 1 mm 7. m,)
p(m,))
184n Pm, (ma,s)
19

---

Therefore, we need
exp g m mm
pm,m (m, m m
(20)
N ブ== exp (m, m-
p(m)) Me Pme (m
Let
Pm., m. my m7,1) g m- = log k p(m,) Lega Pene (m,,s)
(21)
Pm, in (my m g,3) p(m)) Met Peu, (me,i)
expg m- m-
Pm, m. m m7.))
E Hege, Pmay (m)..)
exp m-g.))
E p(m,) Heyy Pan, (mr..)
p p(m,)
eeg Pmy (me,;
R.
(22)
Informally, for large enough N,
1
exp g m,,, i m-m))
exp exp g(,,, m m-y,).
N
p(m,) Ie Pm, (my,))
Therefore, have
exp g m
exp expg(m, m_..)
N Eラ= exp g m m m- Ep(m,)) ) Iler, pmù (m) exp exp (my, m_
exp (my, m 7.1))
by Eq. 22
1
Pm, m (my, m p (m) Ne Pmo (me,)) (my m m_
exp log
by Eq. 21
k
Pm,,.m
p(m,,) Heype Pmn (ma.i
required by Equation (20).
The solution for the scoring function g in Equation (21) equals the g", derived Lemma E.I, that maximizes the total correlation lower bound.
B.4 The Symile objective
Given a batch of N' positive tuples (m m) each with N - 1 corresponding negative tuples (my,4, m' m'_..)) ). and letting 5 e R be temperature parameter, the Symile loss is the negative of an empirical estimate of the expected log likelihood in the lower bound in Theorem B.
(mm -m -0,-)((007)) (0,
N
exp (m..))
log
N
1=1
exp (m)(m)))) 1 (m (mm,,))) //))//r) /T)
Σ j=1
exp (my (2
20

---

C Batch sampling procedure properties
Lemma C.1 (Batch Sampling Procedure Properties). Suppose a batch of N M- -tuuless sampled according the data generating process outlined in Appendix B.1 where
~ Uniform((1.......)
pmm,, M,,) i = 6) = ((mm,)pm. Im, (m y. my)
Pma (me,)
(23)
Let M441 = (men be the set of all samples of modality my in the batch. The following properties hold:
p(m, i = = =p(m,,
p(M i = = Pm. (me.) p((M))
p(m,, m m-7d = (m m, (m- m,) Pm, m_, (m, m m-
Proof. C.1
Derive p(m, 1i = =p).))
pmm, i)
p(m,, M
i) dM
M
=
p(m,) Pm
m. (m
Pm, (mr R) dM-;
by Eq. 23
M
p p(m-))mm mm (m my)
Pm; (m dM 7.4 dm
M ritt E44
p(m)))
my dm--.. Pm |m, (m
m
-(m,))
C.2
Derive M i= --= =I= Pm; (me,)) p(M.).
Let M M, denote the set of all samples of all non-anchor modalities excluding the modality m.
[ /
P(M 1i=
p(m,, M-, dM-
M, dm,
my MM, M,
{
p(m, ppm m. (m- m
Pmy (me.s) dM- My Mydm, by Eq. 23
m. M MM,,i0 MM,
II
Prme (mE,s)
p(m-))mm m. (m (m. m.)
my (m_y/mm \mp)
II
Pm, (ma,s) d(M Meljsi d(m \ me) dmyy
[ (Mee) (y.E)) 3#1
Pone (me.))
my (m-y) p(m-)Pm_ m. (m-y.i m.) d(m \ mm)) dm,
371
Pmn (ma,s)
371
N
Pmn (ma.))
(24)
1m
21

---

We take the expectation over p(i) of both sides of Equation (24) to get
p
i=
Pmn (ms
m()
p()
==1
p(M)
Pme (m.,))
P()) j=1
-(M li=))
by Eq. 24
C.3
Derivep(m,, m 70- i i) ((m))Pm-. 1m, (m m,) pm,. (my, m=y.)).
p(m,, 7. 1= i) =
p(my, M 1= 1) (MM tostt
M-
p(my)pm.. my (m
Pmy (me.s) dM dM- by Eq. 23
M
=p(my PPm my (m- my)
IIII
Pm. (mr,)) dM
= =p(m, Pm m. (m- m,)
= -mm (m, v.))
22

---

D Total correlation for a batch
Lemma D.1 (Total Correlation for a Batch of Tuples). Suppose a batch of N M -tuples is sampled according to the data generating process outlined in Appendix B.1 where
~ Uniform((1......
Pm (mes)
pmm, M i = 2) p(m, Pm
Im,
mm
E4 571
We claim that for any value i ofi
TC (me)e+) TC (Mr | i= i).
Proof. By the definition of conditional total correlation,
TC(m, (M | =i= i) DKL (p(m,, M., l = i) || p( p(m, = i) II ((M i))
ES9
p(my,, M
1)
E
log
p(m, M
i=i))
(m,, i = i) bey M(M |
p m. M_, i i=
E
log
by Lemma C.I
p(m,, M
imi)
pm,)
II p(M))
p(m, Pm m, (m 7, mm,) Nez IT Pen (me,))
E
log
by Lemma C.I
p(m,, M limi)
p(m,) Loyy N
kim Peoc (me,,)
p(m, Pm Im, (m- |m,)
E
log
p(m,) Lezy Pme
(me))) (me,) Pme (m,.))
mm m_y
DRI. (pm,,
(m, m m_ ll p(m)) II
bxy
= mm (metee)).
23

---

E Scoring function as total correlation likelihood ratio estimator
this section, we show that the optimal scoring function is equal to the log total correlation likelihood ratio up additive constants.
Lemma E.1 (Scoring Function as Total Correlation Likelihood Ratio Estimator). Suppose batch of N M-tuples is sampled according to the data generating process outlined in Appendix B.I. For some O, the g that maximizes the lower bound
exp my x.4)
TC (me)et)) log N
E M-, li=1)
log
Ej=1 exp gl m, -7
is
Pm, m_ (my m_
g" g' m_ = n + log
p(m,) Lenn Pm: (m))
Proof. Define
(1 = my m M M.y) exp g my 1 771)
ンド= exp g (m,, mmyJ)
be the categorical cross-entropy of correctly classifying the positive tuple (m,, m-
The maximizer of the log likelihood is the true conditional distribution, which by Lemma F.1 is
Pro Im (m m,,)
Teay Ping (mu..)
p(i positive my, M M_,)
my m 7. m.
j=1
Teyy Ping (me.)
Therefore, solving for the form of the optimal g", we have
p(i = positive my, M..) = pd (i= -=imm,M.... M,.)
Pmi_, Jay (m 7.5 m..)
He Pmy (
exp (m, m 7
Pm
mm m
j=1 exp (m,: m.ns)
Lesy Pm, (m))))
Therefore, optimality, when our model is equal to the true conditional distribution, for some constant K > 0, we have
Pm, m_ m. m p(m,) Lest Pmn (m:)
(m,, m_n) = K + log
24

---

Ratio of total correlation likelihood ratios
Lemma F.1 (Ratio of Total Correlation Likelihood Ratios), Suppose a batch of N M M-tuples is sampled according to the data generating process outlined in Appendix B.1 where
~ Uniform([1......
pmm, M i = 1) p(m, Pm Im,
mm
Pm (mes)
(25)
The true conditional probability that (my i m= the positive tuple among all N samples in the batch can be expressed as ratio of total correlation likelihood ratios:
Pm mm [m- m. m.))
Hey Pmy (my
p(i positive my, M-,)
my (m 7 mg)
Eey Pmy (myg.)
Proof. We first apply the definition of conditional probability and the law of total probability:
p(i positive my, M M- p(m, M y, M_,, positive)
p(m,. M-,)
pmm, M M,,, i positive)
N j= p M M - positive)
((m))pm m,, (m 3. m Hey Iays Per (mea)
by Eq. 25
Σド= p(m, (1mm))pm... Pm
mm, (m ri my)) He Trpe Pene (merr)
Loyy Pme Pme (me,)
Pm.. my (m-ayi m,) Mepr Ieys Pm, (me.a)
E Pme (me,e)
Ligty Pme (me.)
Pm- m. (m و m,)) Il Nr Pens (mer))
1=
Pegy Pm (me.)
1m. (m 7. m.)
Ligly II Pme (maa.)
Me Pm, (m;
(m- v.d m.)
Heyy I Pmne (mer)
Exby Panz (m).)
Pmn 1m mm- m.)
1erge Pm, (Me.s)
Pm.., mm 7 m
===
Ie Payy me..)
25

---

G Symile learns sufficient statistics
Theorem G.1 (Symile Sufficient Statistics). Let m.. mm be M random variables whose optimal representations when trained using Symile are fi (m.).... f' (mm). respectively. The element- wise product of any subset of the representations is sufficient statistic for predicting the remaining random variables.
For example, letting 7 be arbitrary in (l...., M) and letting Ix Sk (ma) indicate the element- wise product of the representations for the remaining M modalities, I (mg) is sufficient statistic for predicting my, which can expressed using the following conditional independence statement:
II (2(m) ley
my
Proof. Since, as discussed in Section 3.2, we use the multilinear inner product (MIP) as the scoring function g, by Lemma E. for some & > 0 at optimality, we have
Pm, m_ (m m_
g" mm. m_ = (1.. (m. )
log
k p(m,) Iop Pmm (ms))
(26)
Consider the case which we are given representations for the modalities that are not m.. The goal is to show
II (m)). Ag7
m
To do so, we will show that
p pmm.)
fi (mx)) = p(m, m
II (mm)).
Reff
Since, conditioned on m. yy. my is independent of any function of mayo
pl p(m. m
II kg9
fl (m²)) p(m, m m
p(m,, m..)
p(m_>)
p(m,, m_.)
[ p(me)
M
p(m_,)
III M
p(m,)
exp (mt)((m
Ie p(me)
by Eq. 26.
(27)
K p(m
Since is distribution,
[ exp (m:)) |. I p(m))
(m:)
dm,
K - p.-pmm pmm_))
m.
[ m.
K p( p(mm)) II p(me)
exp [V (m:) (l
Substituting this back into Equation (27) yields
cxp ((1f((m))))))) ( p(m))
m
(m))
exp [[f] ] ) )mm (28)
m.
Now compute
p(m,)
Il (mx))
E
p(m, m_m.. (mm))
p(m-7 Hes, f7 (m.
kgn
26

---

() p(m)) exp
E
by Eq. 28
p(m-y Mepe
12 (mx))
L.m, exp
(m,)) p ((mm))mmm
exp J3 (mx)) f (m,))
E p(m-y Me (2(m)
Sm, exp [[Ix, (mm)) f (m,) (p
Since m- only appears inside the expectation through Ix ft (m)). and since conditioning on 1k f (m²) being particular value, the term inside the expectation is conditionally constant. Therefore,
exp [[I1 ft (mx)) f (mm))ppmm)
p(my)
II f (m²)) k
S m exp
[[]] f2 (ms&) f (m,, ((m,))dm,
exp
K( (1.)) [mmm)/=a111 (p(m,)))
exp
KU (m.) M pdm,)dm,
m,
p(m, m m_ II f2
by Eq. 28
kpy
This equality establishes that
my
(ma).
27

---

H Zero-shot prediction using the score function
In this section, we discuss the limitations- for both Symile and CI.IP-of using the scoring function for zero-shot prediction and demonstrate how these limitations can be addressed by using the scoring function to directly compute the desired conditional probability.
Recall from Lemma 3.2 that the optimal scoring function g" is equal to the instantaneous total correlation up additive constants:
9 (x.y, = log Px.y X. y. 2)
p(x) (y)pa
Similarly, the optimal scoring function h* for CLIP can be expressed follows [38, 39]:
Px, (x,))
h (x. = log k p(x) (y)
Traditionally, for zero-shot prediction with CLIP, the scoring function is used to rank the candidates for one of the modalities: arg yax ply = y æ²) arg max h (z. y). However, it turns out that this approach for zero-shot prediction does not lead to the Bayes optimal prediction, potentially sacrificing accuracy.
To illustrate the issue, consider scenario in which we have two modalities: disease y and temperature t.. The values these two variables can take outlined in the following joint distribution table:
t
99
100
101
102
p(y)
0.1
0.1
0.3
0.3
0.8
0.1
0.1
0.2
ptt)
0.1
0.1
0.4
0.4
Now, consider patient with a temperature of 101 degrees; our goal predict which disease the patient has. Predictions derived from the conditional distribution achieve optimal accuracy [36]. Therefore, we should predict that the patient has disease a, since
ply = a mltt t 101) ply = a, = 101)
0.3
0.75
p
0.4
and
ply = b t= 101) ply = b, 101)
0.1
0.25.
p(t 101)
0.4
However, were we to apply the standard strategy of using the scoring function for zero-shot classi- fication, we would predict that the patient has disease b, since dividing by the prior probability of disease b upweights its likelihood ratio compared to that of disease
ply a =001
0.75
0.9375
ply a)
0.8
compared to
ply b t 101) ply=
0.25
1.25.
0.2
Why, then, does CLIP perform well in practice? Because the kinds of zero-shot classification tasks for which the dot product is used typically feature an almost deterministic likelihood, where the modality predict has point mass distribution at a single value, with probability zero everywhere else.
For example, in our case, this would mean that p(y = a | = 101) = and p(y = b t 101) = 0.
higher likelihood ratio for disease a compared disease b:
p(y= = a 1011
p(y = b 101)
p(y = a)
0.8
0.2
p(y = 6)
28

---

While zero-shot classification works well when one modality directly determines another (for example, a text caption precisely specifies its corresponding image), in all other instances, the CLIP or Symile scoring function fails to provide reliable predictions.
To address this issue, demonstrate how the Symile scoring function be used compute the desired conditional distribution, which achieves optimal classification accuracy. (While we illustrate this approach for Symile, can be applied similarly CLIP.)
Suppose we want to predict modality y from modalities x, z using zero-shot classification. Recall the multilinear inner product (MIP) as the scoring function. Theorem H.1 establishes that we can compute p(y x, z) directly using the MIP.
from Section 3.2 that
Theorem H.1 (Conditional Distribution using the Scoring Function). Letx, y, z be three random vari- ables whose optimal representations when trained using Symile are (x)x,, fy (y), ;) J2 (z), respectively. Let the MIP (xx(x), 82 (y), f2 (z)) be the scoring function. Then,
exp fg)) Jy exp (f2(x), fy)) f2 ply)dy
ply x.
(29)
Proof. Let (x) @ (z) indicate the element-wise product of the two representations. Since f2 (x) @ (z) determined by x and z,
xx, ply x, 2x (z)) exp
(2)) by Eq.28. 28.
Jy exp
If the marginal distribution y is known, we could then perform zero-shot classification in one of two ways. When the distribution p(y x, 2) itself is of interest, is often the case in healthcare [10], we could compute p(y x, z) directly, following Equation (29). Alternatively, if only predictions are needed, we could use
( (x)) + log p(y)
to rank the possible values for y.
To see why the latter approach works, first notice that
exp (f
log ply x, log Jy exp ((x)
by Eq.29 29
= (y)). (()) +ogply) -xpp-eeexxXXXXXX (6)
logp(y xx 2) + exp (x), (())
= (f) fy (y), (z)) + ply).
(30)
Since the above integral is constant with respect to Equation (30) will produce the same rankings for y as log p(y xx,2) 2).
the marginal distribution of y is not known, then because ft (x) O (z) is a sufficient statistic for predicting y (Theorem 3.3), could instead use [x) O (z) to train simple model to predict any property of y, s(y): p(s(y) fo (x) C (z)).
29

---

I Experiment details
All datasets and code used in this work are publicly available at rajesh-lab/see symile. le.
com/
For all experiments, we use the AdamW optimizer [32]. Following [40], the temperature parameter is directly optimized during training a multiplicative scalar to avoid the need for separate hyperparameter tuning. Experiments were conducted with 16 CPUs, 200GB of RAM, and a single NVIDIA A100 80GB PCIc GPU.
L1 Simulated data: ID
We fit model with three affine linear functions that map the binary data b, to representations Ta bb, Ie E R16, respectively. The zero-shot classification task to predict whether rb=0 or rb=1 is the correct match for given ra, Fe.
1.2 Simulated data: 5D
The synthetic dataset is drawn according to the following sampling procedure:
Cj = (aj XOR 6,)* aat
ag. by (er5o i ~~ Bernoulli(p).
[a1,.... asl,
b = [b1....,,5]] 6s],
c= [cl....,cs
We construct train, val, and test sets of 10K, IK, and 5K samples, respectively. We fit three affine linear functions that map a, b, e. representations ra, Te I6 respectively. These representations are then 1.2-normalized.
Both Symile and CLIP are trained for 100 epochs using batch size of 1000, learning rate of 0.1, and a weight decay of 0.01. The learned temperature parameter is initialized to -0.3. The Symile loss is trained with (N) negative sampling. Checkpoints saved the end of every epoch, and the best model was selected based on the lowest validation loss.
1.3 Symile-M3
Dataset. We use images from the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012-2017 train set [45], which we downloaded from Kaggle. The ImageNet train set has 1,281,167 images from 1,000 categories.
We use audio from the Common Voice Corpus [4]. All languages are from versions 16.0 except for English, which is from version 14.0. Each audio clip in the dataset is an MP3 file that consists of a sentence being read aloud. We remove any audio clips that have duration 0.0 seconds. We use the following languages for each version of Symile-M3:
Sy-ile-M3-2: English, Greek
SymileMM3-5: English, Greek, Hindi, Japanese, Ukrainian
Symile-M3-10: Arabic, Chinese, English, Greek, Hindi, Japanese, Korean, Telugu, Thai, Ukrainian
To generate text, we use Google Cloud's Translation API to translate the ImageNet class names into the relevant language. For the ImageNet class names with identical translations across languages (for example, dog breeds), we manually disambiguate so there is no translation overlap. We publicly release all translations ensure reproducibilityy
For each of the three versions Symile-M3, 10M training, 500K validation, and 500K test samples generated.
Training. Although Symile does not require the use of pre-trained encoders. we use them to accelerate training. For audio, image, and text, we use pre-trained encoders from Whisper [41] (Hugging Face model id opena / weer- operta//cllpp-tt---are--pttttt xlm itg.o Audio is downsampled to 16kHz, expected by Whisper,
CLIP [40] (Hugging Face model id and XIMMRoBERTa [13] (Hugging Face model id
http::///ww...agglea.eoooo////g/ www.aopeeetttttttteeeoaaaaaaaaee 1oca cha Lenge/
htktps:/www.kagggee.. com/
overvies//decriitiion
'nttps://cooud. http:////coouu.gogggl...////ttta
30

---

before being passed to the feature extractor. We freeze the three encoders' parameters except for those in the text encoder's embedding layer and first encoder layer. which are fine-tuned. We train three linear projections to map each encoder's representation to the same 8192-dimensional space, followed by layer normalization.
For each combination of objective (Symile CLIP) and Symile-M3 version (2, 5, or 10), we do grid search over learning rate (le-5, 5e-5, le-4) and weight decay (0, Ie-4, Ie-3). We also tune these hyperparameters for the experiments with missing data. All models are trained for 24 epochs using a batch size of 256. The learned temperature parameter T is initialized to -6. The Symile loss is trained with (N) negative sampling. Checkpoints were saved every two epochs, and the best model was selected based on the lowest validation loss.
Missingness. We evaluate Symile on a variant of Symile-M3-2 where each modality is indepen- dently missing with probability 0.5 0.65, which correspond, respectively, to probabilities 0.125 and 0.043 of a complete data sample.
For audio and image data, we learn two embeddings, one for observed data points and one for missing data points. Each embedding matches the dimension of the last hidden layer of the respective audio or image encoder. When data point is observed, we concatenate its encoder representation and the learned embedding for observed data points, and pass this combined vector into the linear projection head before layer normalization. When a data point is missing, we concatenate the mean encoder representation from the observed training samples and the learned embedding for missing data points, and pass this combined vector into the linear projection head before layer normalization.
For text data, if data point is missing, we pass into the text encoder the tokenized representation of [MISS ING], which is outside of the model's vocabulary.
L.4 SyMile-MIMIC
Symile-MIMIC is clinical dataset comprised of chest X-rays, clectrocardiograms, and blood labs from the MIMIC-IV [16, 17, 24, 27] and MIMIC-CXR [25, 26] datasets. We use admissions and labs from MIMIC-IV v2.2, ECGs from MIMIC-IV-ECG v1.0. 8 and CXRs from MIMIC-CXR v2.0.0.9
Each data sample includes an ECG reading and blood labs taken within 24 hours of the patient's admission to the hospital, and CXR taken in the 24-72 hour period post-admission. For each admission, we choose the carliest CXR, ECG, and labs.
We use CXRs in JPG format, and consider only CXRs with a posteroanterior (PA) or anteroposterior (AP) view. Following Irvin et al. [23], each CXR is scaled such that the smaller edge is set 320 pixels, followed by a square crop (random for training or center for validation and testing). Images are then normalized using the ImageNet mean and standard deviation.
We use 10-second 12-lead ECGs, and remove from consideration any ECGs with NaN values or with signal of all zeros. The ECG signal is normalized to lie within the range [-1,
We focus on the following 50 most common blood laboratory measurements in our dataset, with each data sample containing at least one: Hematocrit, Platelet Count, Creatinine, Potassium, Hemoglobin, White Blood Cells, MCHC, Red Blood Cells, MCV, MCH, RDW, Urea Nitrogen, Sodium, Chloride, Bicarbonate, Anion Gap, Glucose, Magnesium, Calcium Total, Phosphate, INR (PT), PT, PTT, Basophils, Neutrophils, Monocytes, Eosinophils, Lymphocytes, RDW-SD, H. L. I, Alanine Amino- transferase (ALT), Asparate Aminotransferase (AST), Lactate, Alkaline Phosphatase, Bilirubin Total, pH, Albumin, Base Excess, pO2, Calculated Total CO2, pCO2, Absolute Neutrophil Count, Absolute Eosinophil Count, Absolute Monocyte Count, Absolute Basophil Count, Absolute Lymphocyte Count, Creatine Kinase (CK), Immature Granulocytes.
For the labs model, we use a 100-dimensional vector as input: the first 50 coordinates are lab values standardized to percentiles based on the training set's empirical CDF, and the remaining 50 coordinates are binary indicators that denote whether each lab value is missing. When a lab value is unobserved, the mean percentile for that lab is substituted.
"https://physiionets http://www.....eottettttnnni//// hnttpa://physiioeet wtgg/onnentttoooooooooooooo/meoe www.otteet//iiiiiieeeeeeeee Pnttps://physiona org/contente otg////tttttti/mmmoooooooo///22/
31

---

Following previous work [8. 22, 29, 30, 57], we use the ResNet-50 and ResNet-18 18 architectures [20] for the CXR and ECG encoders, respectively, and three-layer neural network to encode the blood labs. All encoders are trained from scratch, and three linear projections map each encoder's representation to the same 8192-dimensional space.
For Symile and CLIP each, we do a grid search over learning rate (5e-5, lc-4, 5e-4, le-3, 5e-3, le-2) and weight decay (Ie-3, le-2, le-1, 2e-1, 5e-1). All models are trained for 80 epochs using batch size of 280. The learned temperature parameter T is initialized to -7. The Symile loss is trained with O(N2) negative sampling to mitigate overfitting. Checkpoints were saved at the end of every epoch, and the best model was selected based on the lowest validation loss.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction are substantiated both theoret- ically and empirically in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
In Section 3, we outline limitations, clearly state all theoretical assump- tions, and discuss the computational and memory trade-off's of various negative sampling approaches.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
33

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
All theoretical assumptions and proofs are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are described in Appendix I, and the code and data used for the experiments have been released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
34

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code and data used for the experiments have been released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /ttp//ips....o cc/ public/ guiguuieeee wwedddeeeeooeeeeeeeee CodeSuumiis ss sisnooiicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See NeurlPS code and data submission guidelines (https: //nips cc/ ctips.cc/pubbic.. gwwpc/////u///eeeeeeeeeocccceeee Cooee as onPoli olycy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Experimental details are fully described in Section 5 and Appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
35

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard error is reported for all experiments Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Details on the compute resources used are provided in Appendix I.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tee:///nerrr htt///eurip...// cc/ httpa////c//////tcccccccceeeeeea
36

---

Answer: [Yes]
Justification: Our work fully conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive societal impacts Symile, particularly healthcare, are discussed in Sections 1, and 5.3.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work relies publicly available, de-identified healthcare data, ensuring full adherence to ethical guidelines and privacy standards.
Guidelines:
The answer NA means that the paper poses such risks.
37

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All sources for the datasets and models used in this work are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper s wiit tchooee com/ da aast tets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Full details for the new datasets are available in Section 5, Appendix and https: / com/ ra ra paajesh-aie fesh-lab/ symile.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
38

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
39


Reviews:
[{"review_id": "GRYgDY1De0", "reviewer": "Reviewer_QTPJ", "summary": "This paper proposes Symile, a new contrastive learning objective that addresses the challenge of contrasting multiple modalities simultaneously by considering total correlation. Symile captures the statistical dependence between multiple variables and uses a generalized inner product to derive a lower bound on total correlation. The representations learned by Symile form sufficient statistics for other modalities, outperforming pairwise CLIP in cross-modal classification and retrieval. This is demonstrated through experiments on a multilingual dataset with 33 million samples and a clinical dataset.\n\n=== After response ==\nI initially assigned a score of 4, but after reviewing the manuscript further and clarifying some misunderstood details addressed in the rebuttal, I decided to adjust my score. Following a discussion of other reviews, I increased my score to 6 (weak accept).", "questions": "#1. It's a great idea to include a simple and clear example to show why summing pairwise contrastive losses might not be best for multiple modalities. However, the failure analysis seems very theoretical. In real-world datasets, the assumption that random variables are jointly dependent but pairwise independent might not be true. Can you provide more realistic examples where real-world datasets fit this assumption?\n\n#2. What is the reason for using the coordinate-wise sum of the element-wise product of a set of vectors instead of the simple dot product for the scoring function?\n\n#3. The only baseline used to compare the proposed method is CLIP. More recent works, especially those focused on representation learning with multiple modalities, should also be included. In my opinion, ImageBind would be selected as its weights are publicly available.\n\n#4. The negative sampling with O(N) compared to O(N^2) seems to be a very crude approximation. Does it affect the performance? From a theoretical perspective, can we analyze the difference between the linear and quadratic sampling schemes in terms of the total correlation lower bound?\n\n#5. Minor comments\nWhat do the color shades in Figure 1 represent?\nIn line 138, the element of Z_n​ should be written in lowercase as z_n.\nIn related works, the authors mention that contrastive learning has been popularized since the release of CLIP. However, in my opinion, the credit for popularizing contrastive learning should go to SimCLR.", "limitations": "The authors have mentioned the limitations, aside from the lack of comparisons. Additionally, in my opinion, there is no potential negative societal impact of this work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Introducing the limitation of pairwise summation of contrastive losses for multiple modalities is interesting, as demonstrated by both a synthetic example and theoretical justification. Addressing this limitation makes the work more principled. Additionally, handling missing data is crucial for practical scenarios.", "weaknesses": "Limited experiments: \n- The paper only compares the proposed method with CLIP. More recent works, especially those focused on representation learning with multiple modalities, should be included for a comprehensive comparison. ImageBind, with its publicly available weights, would be considered as a baseline.\n- The formulation of the Symile model requires complete triplets (e.g., image, text, audio) for training, which makes it difficult to scale up the training data. The current approach does not accommodate missing values in the triplet elements, which is a significant limitation for practical applications. An extension to handle missing values is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback, and are glad that you found our work conceptually and theoretically compelling. We address your questions below, but please let us know if we can provide any further clarifications. If we have successfully addressed your primary concerns, would you be willing to consider raising your score?\n \n**ImageBind as a baseline:** As we discuss in the global response above, the baseline that we currently use in our paper is pairwise CLIP, and ImageBind is an instantiation of pairwise CLIP. That said, your feedback indicates that we could have made this clearer in Related Work, which we have now updated.\n\n**Accommodating data missingness:** It seems there was a misunderstanding here. In our missingness experiment in Section 5.2, we do indeed train Symile with missingness in the training data. On the right-hand side of Fig. 5, we show that Symile outperforms pairwise CLIP on the 2-word subset of Symile-M3 when only 12.5% and 2.7% of the training data has complete triplets! We will update this section so that it's clearer to readers that this experiment was run with missingness in the training data.\n\nWe absolutely agree that in order for Symile to be at all useful for practical applications, it needs to be able to accomodate any amount of missingness in the data, which is why we ran the missingness experiments in Section 5.2. Our approach for handling missingness was specifically designed to be easy to implement for practical applications. Please let us know if there is additional information we can provide to help clarify this misunderstanding.\n\n**Real-world scenarios where Symile outperforms CLIP:** We're glad that you found the XOR example to be clear and illustrative of the types of information that Symile and pairwise CLIP capture, and you are correct that it represents an extreme case that is probably rare in real-world applications. To see why, consider again the total correlation decomposition in line 115 of our paper:\n$$3 \\cdot TC(x,y,z) = 2 \\cdot [MI(x;y)+MI(y;z)+MI(x;z)] + MI(x;y | z)+MI(y;z | x)+MI(x;z | y).$$\nIn the XOR example, the CLIP target is zero, but as you rightly suggest, most real-world cases will contain *some* pairwise information.\n\nBased on the decomposition above, Symile will outperform CLIP in situations where the higher-order (conditional) information terms play a role. This is typically seen in applications when two modes are needed to infer the third.\n\nWe include one such example in Section 5.3 of our paper on the task of predicting chest X-ray from ECG and labs: there is pairwise information to learn, which explains why CLIP performs better than random guessing, but the presence of conditional information allows Symile to ultimately outperform CLIP. As suggested by Reviewer hrNA, we ran preliminary experiments on the VQA dataset and found that Symile outperforms CLIP, illustrating another instance where conditional information benefits Symile.\n\n**Multilinear inner product (Q2):** The dot product is only defined for two modes. The coordinate-wise sum of the element-wise product is one of the simplest possible generalizations of the dot product to more than two modes. Please let us know if we misunderstood the question or can clarify further.\n\n**Negative sampling performance impact and lower bound connection:** The results we report in the paper on the Symile-M3 dataset were run using $O(N)$ negative sampling. We have now also run this experiment on the 2-word subset of Symile-M3 using $O(N^2)$ negative sampling. We find that when trained using $O(N^2)$ negative sampling, Symile demonstrates a marginal improvement in test accuracy over training with $O(N)$ negative sampling (0.938 vs. 0.937). However, Symile trained with $O(N^2)$ negative sampling takes almost twice as long to run 24 epochs (43 vs. 24 hours), but achieves its best validation accuracy in far fewer epochs (4 vs. 16). We have added these results to the Appendix.\n\nAs discussed in Section 5.3, we found that training with $O(N^2)$ negative sampling on the Symile-MIMIC dataset helped mitigate overfitting. We hypothesize that this is because more negative samples create a more challenging learning problem, allowing the model to better learn to differentiate between positive and negative samples.\n\nIn terms of analyzing the difference between the negative sampling schemes in terms of the derived lower bound, in lines 156-157 we write, \"We show in Appendix B.3 that, as $N$ gets larger, the total correlation lower bound closes for the optimal scoring function $g^*$.\" This implies a computational-statistical trade-off: a larger batch size demands more computation but results in a tighter bound.\n\n**Colors in Fig. 1:** The colors are meant to loosely represent the information in each of the three random variables. We carefully considered how best to structure this figure: on the one hand, we wanted a high-level and intuitive illustration of the different types of information that Symile and CLIP each capture (and it seems that Reviewer hrNA appreciated this); on the other hand, we understand that this figure is underspecified. Ultimately, we decided to err on the side of building the reader's intuition.\n\n**SimCLR vs. CLIP in Related Work:** We write in Related Work that CLIP \"popularized the use of a contrastive objective for general image-text representation learning,\" but we agree with you that SimCLR popularized contrastive learning for multiple augmentations for a single modality. We will update this section accordingly.\n\n**$Z_n$ typo:** Thank you for catching this!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LuifecTs1R", "reviewer": "Reviewer_HxmV", "summary": "The paper introduces Symile, a new contrastive learning objective designed to accommodate any number of modalities, addressing the limitations of pairwise CLIP which fails to capture joint and conditional information between modalities. Symile targets total correlation, capturing higher-order dependencies among multiple variables. The paper derives a lower bound on total correlation using a generalized form of inner products and demonstrates that Symile representations form sufficient statistics for the remaining modalities. Experiments on a large multilingual dataset and clinical data show that Symile outperforms pairwise CLIP in cross-modal classification and retrieval tasks, even with missing modalities.", "questions": "See the question in Weaknesses.", "limitations": "The authors have discussed certain limitations in section 6.", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a model-agnostic contrastive learning objective that captures higher-order dependencies among any number of modalities, addressing limitations of existing pairwise CLIP methods. It effectively identifies and addresses a significant limitation of CLIP, presenting a novel and important contribution to the field.\n2. The paper starts with a simple one-dimensional problem to illustrate the failure mode of pairwise CLIP, which aids in understanding and intuition.\n3. Theoretical analysis is provided to further support that Symile learns sufficient statistics.", "weaknesses": "I wonder if more explanation could be provided on the rationale of the MIP. For example, if we have 3 vectors and consider one coordinate, imagine two cases: (1) the values are x= -1, y= -1, z= 1, whose product is 1, and (2) the values are x= -1, y= -1, z= -1, whose product is -1. In this example, case (2) apparently has higher similarity between the 3 values than case (1), but case (1) ends up with a larger MIP value, which seems undesirable to me.\n\nFor the experiments, since I am not very familiar with the state of the art in learning with three or more modalities, I'll leave it to other reviewers to judge whether the benchmarks used and the comparisons are sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and questions! We're thrilled that you found our work valuable and our examples and theoretical contributions effective. Please let us know if we can provide any further clarifications. If our response has resolved your questions, we would greatly appreciate it if you would consider raising your score.\n\n**Explanation for multilinear inner product (MIP):** We chose the MIP as a scoring function for several reasons. We were drawn to its simplicity: the MIP is one of the simplest possible generalizations of the dot product to more than two modalities in terms of computation. The scoring function also needs to be expressive enough to model any joint statistic, which requires that the vectors be multiplied together. Therefore, the MIP strikes a nice balance between computational simplicity and expressiveness.\n\nWe understand your confusion regarding the interpretation of the MIP—the MIP is not a measure of how *geometrically* similar vectors are. Suppose we have three modalities $\\mathbf{x}, \\mathbf{y}, \\mathbf{z}$ whose Symile representations are $\\mathbf{r_x}, \\mathbf{r_y}, \\mathbf{r_z}$. If the MIP of $\\mathbf{r_x}, \\mathbf{r_y}, \\mathbf{r_z}$ is large, then that indicates that $(\\mathbf{x}, \\mathbf{y}, \\mathbf{z})$ has high probability under the joint likelihood. It says nothing about whether or not $\\mathbf{r_x}, \\mathbf{r_y}, \\mathbf{r_z}$ are equal to one another. In other words, the MIP is a measure of similarity imbued by the joint distributition of the modalities.\n\nWe're glad you brought this up, and have included this clarification in the paper, as it will benefit our readers.\n\n**Baselines:** You'll see that we addressed the issue of baselines and benchmarks in our global response, and in our responses to the other two reviewers. Please let us know if you have any further questions here that we can address.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCFDWhJTRj", "reviewer": "Reviewer_hrNA", "summary": "This paper shows that CLIP-style pairwise contrastive objectives on multiple modalities (more than two) fail to capture the total dependencies of different modalities, and proposes to use total correlation, a higher-order generalization of mutual information as the new objective to optimize multimodal contrastive learning. The authors provide derivations of the lower bound on total correlation, the optimal scoring function, the practical objective, and proof of statistical sufficiency. On a simulated dataset, a newly created multilingual dataset, and a Chest X-ray dataset, the authors illustrate the limitation of existing CLIP-style pairwise objectives and the effectiveness of the proposed approach.", "questions": "There are no error bars in Figures 5 and 6, will the authors intend to provide error bars for them?\n\nThere is a quite related paper on capturing total information and conditional independence in a multimodal setup. The reviewer would appreciate some comparisons with it:\nLiang, Paul Pu, et al. \"Factorized contrastive learning: Going beyond multi-view redundancy.\" NeurIPS 2023.", "limitations": "The authors include limitations of the work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clarity: the paper is very well written. The motivation, problem formulation, derivation, and novelty of the paper are clearly stated. Figures 1 and 2 are helpful for the reader to understand the concepts.\n\nQuality: the theoretical derivations seem valid to the reviewer upon checking. The theoretical results of the lower bound, optimal scoring function, and statistical sufficiency make this work concrete. The derivations in Appendix A seem correct, and the reviewer did not check the derivations in other Appendix sections. This paper also introduces a new multilingual dataset, Symile-M3, including 33 million (audio, image, text) samples.\n\nSignificance: the paper points out that pair-wise CLIP losses cannot capture conditional dependencies between multiple variables, and provides a simple and effective solution to address this. The problem of leveraging contrastive learning on more than two modalities is worth studying.", "weaknesses": "Originality: the problem of capturing dependencies of multiple variables beyond pairwise mutual information is well-studied. There are many solutions proposed already, albeit not in an explicitly multimodal setup, as discussed by the authors in Lines 226-231. However, the reviewer reckons that the methods such as Bai et al. can be extended to the setting involving three or more modalities, e.g., substituting the $x^1, ..., x^m$ by different modalities in the Sample-based TC estimator from Bai et al. The authors did not include any such baseline results on the datasets. The reviewer would appreciate clarifications on this.\n\nQuality: the authors did not justify clearly why they curated a new dataset (and a multilingual one) instead of using existing multimodal datasets, e.g., VQA, GQA, How2, HowTo100M, etc. Lacking comparisons on standard datasets weakens the submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your helpful and constructive comments! We are glad that you found our contributions well motivated and clearly explained, and that you consider our solution to be simple and effective. We have addressed your questions regarding suitable baselines for Symile both below and in the global response above, but do let us know if there are any further clarifications we can provide. If we have addressed your main concerns, would you be willing to consider raising your score?\n\n**Bai et al. as a baseline:** While Symile optimizes only a single term in targeting total correlation (TC), Bai et al. derive TC estimators by recursively decomposing TC into a summation of MI terms, to which variational estimators are applied, using two linear separation paths:\n$$\\text{Line-like: } TC(X_{1:i+1})=TC(X_{1:i})+I(X_{1:i};x_{i+1})$$\n$$\\text{Tree-like: } TC(X_{i:j})=TC(X_{i:\\lfloor(i+j)/2\\rfloor})+TC(X_{\\lfloor(i+j)/2\\rfloor+1:j})+I(X_{i:\\lfloor(i+j)/2\\rfloor};X_{\\lfloor(i+j)/2\\rfloor+1:j})$$\nThe authors apply the tree-like TC estimator to multi-augmentation contrastive learning, where a text encoder is trained by maximizing the TC between augmentations of the text. For this experiment, the authors use InfoNCE to estimate the MI terms and find that their estimator does not perform much better than pairwise InfoNCE.\n\nAs we discuss in the global response, Bai et al. and others have leveraged TC estimators for contrastive learning on multiple augmentations of the same data, but to our knowledge no one has applied such estimators to more than two distinct modalities. We also explain in the global response why pairwise CLIP is the most suitable baseline for Symile, but we recognize that we could have provided a better framing for how we situate our work in comparison to Bai et al., and have updated Related Work accordingly.\n\n**Justification for new dataset:** We created the Symile-M3 dataset for two reasons:\n1. There should exist a dataset specifically designed to evaluate a model's ability to capture higher-order information between modes.\n2. We wanted a dataset comprised of distinct data types for more than two modes.\n\nBy incorporating multiple languages, we were able to design a task where two modes (text and audio) were needed to predict the third (image), and where, importantly, neither text nor audio alone would suffice for predicting image. That said, we agree that it is important to run experiments on real-world datasets, which is why we ran the healthcare experiment on the existing MIMIC dataset in Section 5.3.\n\nWe considered video data, but found that we could not use datasets such as How2 and HowTo100M because the text is a direct transcription of the audio. When one mode is a function of another, the dataset effectively has only two modes, and total correlation between three modes degenerates to mutual information between two modes.\n\nWhile we had not originally considered datasets like VQA because they incorporate only two data types, we agree that including results on standard datasets would strengthen our paper. Based on your suggestion, we ran preliminary experiments using available encoders on the VQA dataset where the three modes are 1) an image, 2) a text question (e.g. \"Who is wearing glasses?\"), and 3) a text answer (e.g. \"woman\"). On the task of predicting one of 18 possible answers given the image and question, Symile outperformed CLIP with a mean test accuracy of 0.590 vs. 0.558. We intend to run this experiment more extensively, and will add the findings to our paper.\n\n**Comparison with Liang et al.:** While Liang et al. use contrastive learning objectives to optimize conditional MI terms, their approach is restricted to handling only two modalities.\n\nFor modes $X_1$ and $X_2$, Liang et al. propose two contrastive objectives (supervised and self-supervised) that apply lower bounds to maximize task-relevant information and upper bounds to minimize task-irrelevant information. The supervised objective introduces a task $Y$:\n$$\nL_{\\text{FactorCL-SUP}}=I_{\\text{NCE}}(X_1;X_2)-I_{\\text{NCE-CLUB}}(X_1;X_2|Y)+I_{\\text{NCE}}(X_1;Y)+I_{\\text{NCE}}(X_2;Y)-I_{\\text{NCE-CLUB}}(X_1;X_2)+I_{\\text{NCE}}(X_1;X_2|Y)\n$$\nwhere $I_{\\text{NCE}}$ is the InfoNCE objective and $I_{\\text{NCE-CLUB}}$ is another contrastive objective derived from an upper bound on mutual information that depends on the optimal scoring function from $I_{\\text{NCE}}$.\n\nInstead of using explicit task labels $Y$, the self-supervised objective uses augmentations $X_1'$ and $X_2'$ to approximate task-relevant information:\n$$L_{\\text{FactorCL-SSL}}=I_{\\text{NCE}}(X_1;X_2)-I_{\\text{NCE-CLUB}}(X_1;X_2|X_1',X_2')+I_{\\text{NCE}}(X_1;X_1')+I_{\\text{NCE}}(X_2;X_2')-I_{\\text{NCE-CLUB}}(X_1;X_2)+I_{\\text{NCE}}(X_1;X_2|X_1',X_2').$$\nUnlike FactorCL, Symile does not require prior knowledge of some specific downstream task defined by either explicit labels or targeted augmentations. FactorCL seeks to decompose the information in two modes for a specific task; Symile targets all information across any number of modes to learn general representations without any decomposition, resulting in computational benefits.\n\nWhile Symile optimizes a single objective with which all parameters can be updated at once, training either of the FactorCL objectives involves the minimization of each of the $I_{\\text{NCE-CLUB}}$ objectives, which in turn requires the optimal critic $f^∗$ from $I_{\\text{NCE}}$. Therefore, within each iteration during FactorCL training, one needs to first obtain the optimal critics for the $I_{\\text{NCE-CLUB}}$ terms using the $I_{\\text{NCE}}$ objective. Symile is able to capture higher-order information for more than two modes while avoiding such computational complexity.\n\nIn summary, FactorCL targets the information in two modes for a specific task, and in support of this target makes use of higher-order information. We have added this to Related Work.\n\n**Error bars:** We will provide error bars for all experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive and careful comments, which will significantly strengthen our work. We are glad they found that:
* our paper "effectively identifies and addresses a significant limitation of CLIP, presenting a novel and important contribution to the field" (HxmV)
* our work "provides a simple and effective solution to address this [limitation]" (hrNA)
* "the problem of leveraging contrastive learning on more than two modalities is worth studying" (hrNA) and "[a]ddressing this limitation makes the work more principled" (QTPJ)
* "The paper is very well written. The motivation, problem formulation, derivation, and novelty of the paper are clearly stated." (hrNA)

### Common reviewer feedback

A shared point that arose concerned appropriate baselines for Symile. We sincerely appreciated these questions—they helped to refine our presentation of Symile's contributions to representation learning. We discuss the baselines that came up below.

**ImageBind**

Reviewer QTPJ writes, "ImageBind, with its publicly available weights, would be considered as a baseline." The baseline that we currently use in our paper is pairwise CLIP, and ImageBind is pairwise CLIP. To see why, let $I$ represent images with learned embeddings $q$ and let $M_\ell$ represent one of the other five modalities with learned embeddings $k_\ell$. Notice that ImageBind uses the InfoNCE loss:
$L_{\mathcal{I},\mathcal{M_\ell}} = -\log \frac{\exp{(q_i^\top k_{\ell,i}} / \tau)}{\exp{(q_i^\top k_{\ell, i}} / \tau) + \sum_{j \neq i} \exp{(q_i^\top k_{\ell, j}} / \tau)}.$
ImageBind then trains by minimizing this loss summed across the other modes indexed by $\ell$. *This is pairwise CLIP where only pairs where the image mode appears are kept.*

In our experiments, we compare Symile to pairwise CLIP—the ImageBind approach—and find that Symile outperforms CLIP. 

**Contrastive representation learning on multiple augmentations**

Reviewer hrNA rightly points out that previous work has explored contrastive learning in the context of multiple views of the same data. For example, Bai et al. (2023) derive two total correlation (TC) estimators, one of which they apply to multi-augmentation contrastive learning where a text encoder is trained by maximizing the TC between four augmentations of the text (see our individual response to hrNA below for details). Shidani et al. (2024), which we mention in Related Work, develop a (pairwise) contrastive approach using more than two augmentations for image representation learning. Reviewer hrNA also mentioned Liang et al. (2023): while they use contrastive learning objectives to optimize conditional MI terms, their approach is restricted to only two modalities.

The relationship between our work and contrastive methods for multi-augmentations on a single modality is comparable to that between CLIP (Radford et al., 2021) and SimCLR (Chen et al., 2020). While SimCLR popularized the mutual information (MI) estimator InfoNCE (van den Oord et al., 2018) for contrastive learning on two augmentations of the same data, CLIP used this estimator to handle distinct modalities. Similarly, Bai et al. (and many others before them) leverage TC or MI estimators for contrastive learning on multiple augmentations of the same data, but to our knowledge no one has made use of such estimators for more than two distinct modalities—except for pairwise CLIP, which we do use as a baseline. Further, these estimators typically require specific encoders that ingest multiple modes of data, which limits their use.

The contribution of our paper is a combination of the contributions of InfoNCE and CLIP for more than two modes of data. Our paper (1) provides a single contrastive loss that works with any encoders and recovers higher-order information for more than two modes (like InfoNCE for two modes) and (2) demonstrates the value of this estimator for representation learning on more than two distinct modes of data (like CLIP for two modes).

Given our work's emphasis on targeting TC for representation learning for distinct modalities using any type of encoder, we view pairwise CLIP as the most suitable baseline because pairwise CLIP is intended for distinct data modalities and allows for the use of any encoders for those modalities. We also show why targeting TC yields good representations by, for example, showing that Symile yields sufficient statistics; this analysis holds for any TC estimator that at optimality yields a likelihood ratio. That said, it is certainly an interesting and worthwhile question to ask how our approach compares to other TC estimators—but it is outside of the scope of our paper.

### Contributions
We now summarize the contibutions of our work.

**Theoretical:**
- Show that, despite its popularity, the pairwise use of CLIP fails to capture higher-order information between modalities, thereby limiting the quality of the representations it learns
- Propose the use of total correlation for representation learning with more than two modalities of data by noting it captures higher-order information
- Derive a multi-sample lower bound on total correlation to build Symile, a simple contrastive learning objective that, unlike other total correlation estimators, accommodates any number of modalities and allows any model to produce representations for each modality
- Prove that representations produced by Symile for any set of modalities form sufficient statistics for the remaining modalities not considered in the set

**Empirical:**
- Demonstrate that Symile outperforms pairwise CLIP on cross-modal classification and retrieval in a multilingual dataset of images, text and audio of over 33M examples and a clinical dataset of chest X-rays, electrocardiograms, and laboratory measurements
- Show that Symile retains its advantage over pairwise CLIP even with modalities missing in the data

Meta-review: This paper introduces Symile, a novel contrastive learning objective designed to capture higher-order dependencies among multiple modalities, addressing limitations of pairwise CLIP approaches. The authors provide theoretical foundations, deriving a lower bound on total correlation and demonstrating that Symile representations form sufficient statistics for remaining modalities.

Given the paper's novel contributions, strong theoretical foundations, and comprehensive empirical validation, along with the authors' thorough addressing of reviewer concerns, I recommend accepting this paper for publication.
Decision: Accept (poster)