The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities
Authors: Yongwei Che
Benjamin Eysenbach
Venue: ICLR 2025 Poster (2025)
Keywords: theory
contrastive learning
probabilistic graphical models
multi-modal learning
reinforcement learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: While internet-scale data often come in pairs (e.g., audio+image, image+text), we often want to perform inferences over modalities unseen together in the training data (e.g., audio+text). Prior work has addressed this issue by learning multiple contrastive embedding spaces between existing modality pairs, implicitly hoping that unseen modality pairs will end up being aligned. This theoretical paper proves that this hope is well founded, under certain assumptions. Starting with the proper Bayesian approach of integrating out intermediate modalities, we show that directly comparing the representations of data from unpaired modalities can recover the same likelihood ratio. Our analysis builds on prior work on the geometry and probabilistic interpretation of contrastive representations, showing how these representations can answer many of the same inferences as probabilistic graphical models. Our analysis suggests two new ways of using contrastive representations: in settings with pre-trained contrastive models, and for handling language ambiguity in reinforcement learning. Our numerical experiments study the importance of our assumptions and demonstrate these new applications.

Published as a conference paper at ICLR 2025
THE "LAw" OF THE UNCONSCIOUS CONTRASTIVE LEARNER: PROBABILISTIC ALIGNMENT OF UNPAIRED MODALITIES
Yongwei Che
Benjamin Eysenbach
Princeton University
Princeton University
yongwe yongweic&pri c&princet.on
eysenbachh@rinncetoon edu
ABSTRACT
While internet-scale data often comes in pairs (e.g., audio/image, image/text), we often want to perform inferences over modalities unseen together the training data (e.g., audio/text). Empirically, this can often be addressed by learning multiple contrastive embedding spaces between existing modality pairs, implicitly hoping that unseen modality pairs will end up being aligned. This theoretical paper proves that this hope is well founded, under certain assumptions. Starting with the proper Bayesian approach of integrating out intermediate modalities, we show that directly comparing the representations of data from unpaired modalities can recover the same likelihood ratio. Our analysis builds on prior work on the geometry and probabilistic interpretation of contrastive representations, showing how these representations can answer many of the same inferences as probabílistic graphical models. Our analysis suggests two new ways of using contrastive representations: in settings with pre-trained contrastive models, and for handling language ambiguity in reinforcement learning. Our numerical experiments study the importance of our assumptions and demonstrate these new applications.
Code: hht.f//ss/tthttoooooooo//./////t
INTRODUCTION
Much of the appeal of contrastive learning is that gives a pplug-n-play' approach for swapping one modality for another. Because representations from different modalities are trained to be aligned when representing the same object, the hope is that (say) a language representation and image representation of the same scene can be used as substitutes. This property is practically appealing for at least two reasons. First, it allows us to make use of pre-trained models. If you have model that wants to make use of (say) language input and you have access pre-trained image-language contrastive model, you might simply train your model on the pre-trained image representations and hope that will continue to work when you swap in the language representations. Second, it allows you to merge datasets that have different modalities, in the same way that a data scientist might merge two SQL tables. This is especially useful in settings where we have datasets with pairs of modalities, where collecting datasets with triplets of modalities (c.g., image, audio, and text) is challenging. While it seems intuitive that this splug-n-play approach should work, to the best of our knowledge prior work has not provided a rigorous argument for why it works or when it should fail. The main contribution of our paper answer these questions, and to provide an alternative algorithm for when this "naive" approach fails.
Our analysis based on two ideas. The first idea. building on prior work (Poole et 2019 Ma
Collins 2018), is to think about contrastive learning probabilistically. Rather than thinking about representations as encoding "this image maps to this text," we think about them as encoding a likelihood that this text is more likely to describe this image, but could plausibly also describe this other image. The second idea, building on top of other prior work (Wang & Isolal (2200). is to consider the marginal distribution over the representations. We follow prior works in making the assumption that the marginal distribution is either (I) Gaussian and (2) Uniform over the hypersphere.
The main contribution of this paper is a probabilistic analysis of multimodal contrastive learning. The first part of our analysis is to show that, when certain assumptions are satisfied, the "plug-n- -n- play" approach is principled, correctly inferring the posterior mean. We also introduce approach

---

Published as a conference paper at ICLR 2025
mnals
Domes antiaae
Acart Spunds
Muscaal Insommmntt
Dassm
Whiggng
Image
Language
Audio
Figure 1: Aligning vision and audio by integrating over the intermediate 'language' modality. While prior work has shown that aligning modalities ++ and ++ results representations that can compare modalities A and C, it remains unclear when and why this is guaranteed to work. This paper provides the assumptions under which this approach is principled, and our analysis unlocks new ways of comparing unpaired modalities and new applications for contrastive learning.
that works agnostic to the marginal distribution of the representations. The second part looks at applications of our results. We start by verifying our theoretical results with experiments over synthetic datasets. One such experiment includes setting where users want to bridge modalities A and C' using pre-trained or black-box models that connect A and C' through an intermediate modality B. Then, extend our results large pre-trained contrastive models over image, language, and audio. Finally, we explore an application of our method in language-conddttinee reinforcement learning, testing our method's 's ability to handle natural ambiguity in language.
2 RELATED WORK
Our work builds on wide range of prior work, ranging from representation learning, multimodal machine learning, and probabilistic graphical models. Contrastive learning methods aim to learn compact representations of examples from
more modalities (Chen et al. 2020 Hadsell et al. 2006; Mikolov et al., 2013), and have found applications ranging from ecology to NLP to reinforcement learning. These methods typically trained pairs of examples, with the aim of acquiring representations that are similar for paired examples and dissimilar for unpaired examples While these pairs are often generated via data augmentation (Oord al. 2018). our work builds on prior work that looks at paired examples coming from different modalities (c.g., an image accompanied by its caption) (Elizalde et al. 2023)] Girdhar et al. 2023; Zhu et al., 2023). Prior work has shown that these methods can work effectively with more than two modalities Xue jtt al. (023); Guzhov 2 et al. (202T), even in settings where there are no paired examples from certain modalities (Girdhar et al. [223]] Zhu et al. 2023 Shah et al. 2023 Yuan et al. 2023 Ferraro et al. 2023). However, from a theoretical perspective it remains unclear why these sorts of "plug-n -1 -nplay" methods can combine modalities for which there no training examples, problem that ordinarily requires marginalization (which is very challenging on the high-dimensional inputs to which these methods have been applied). Are these methods implicitly performing this marginalization? Are they correctly accounting for uncertainty?
Our paper will aim to answer some of these questions by building top of ideas from two lines of prior work. First, we will exploit the geometry of contrastive representations (Wang & Isola 2020, [Eysenbach et al., 2024. borrowing (theoretically-motvvaeeed) assumptions about the asymptotic behavior of contrastive representation distributions. Second, we build on prior work (Poole et al. 2019) that provides a probabilistic interpretation of contrastive learning: when trained with a cross entropy loss, the learned representations encode probability ratio. When combined, these two ideas allow us to prove that reasoning across unpaired modalities takes the form of a certain integral over representations, which admits closed form solution. Our theoretical analysis will explain why the "plug-n-play" approach often works, elucidate its assumptions, and provide new ways of combining unpaired modalities that work under fewer assumptions.
Our work will not consider the successful line of prior work that studies negative-free contrastive learning methods (Grill et al. 2020 Chen al. 202T).
2

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
This section formally defines the problem we are studying: performing probabilistic inferences across pairs of modalities that do not simultaneously occur in the training data.
3.1 PROBLEM STATEMENT
Formally, assume there exists three data modalities A, B, C, and have large-scale paired datasets for A + B and B + C, but little to no data of A + C. Our goal is to learn three encoders (A(),)n(B)) (C that map each modality into a d-dimensional latent representation.
Our focus will be on applying contrastive methods to train these three encoders. Prior works jointly train encoders Φ @B and encoders dB ++ Φc under contrastive losses /Zhu al. (2023); Girdhar et al. )023)) Using this training objective, these works achieve strong results in diverse applications. However, these works take the emergent alignment between unpaired modalities for granted. The alignment result is trivial if there exists a one-to-one mapping over all modality triplets (A, B, C),, however, is unclear whether the method maintains the correct probabilistic information between the unpaired data modalities.
3.2
CONTRASTIVE LEARNING
Given two modalities A and B, let p(a, b) be the joint distribution over a E A and b E B. We will learn critic function that assigns high score to "aligned" pairs f(a, ~ p(a, b) and low score to "unaligned" pairs f(a, b) ~ (b)) This critic function is typically some distance between learned representations. We will use (A(+): A + Rd denote the encoder that maps input a to a d d-dimensional representation, and likewise for (.) and (c(+). We will learn these encoders by comparing the similarity between the produced representations using a function f( (b))..
We train these encoders by sampling pairs (a, b+) p(a, b) of aligned examples (e.g.. . an image and its corresponding caption). We will also sample "un-aligned" pairs from the product of the marginal distributions: (a. b-) p(a) p(b). Both these aligned and unaligned examples used to compute the symmetrized infoNCE objective (Radford et al. (00II) which we optimize to train our encoders:
N log
Σj=1 ef(asdb)) )
eflands)
(ar.))
]
min 6A.9
+ log
E==1 Ey efaas
where the batch of training examples (a;, by) N sampled from the joint distribution p(a, b).
3.3
KEY ASSUMPTIONS
We outline two key assumptions necessary for our analysis, followed by a third assumption on the marginal distribution of our representations. The first assumption is a restriction on the relationships of the data modalities for our analysis to hold. The remaining assumptions build off prior work in contrastive learning.
Assumption 1. Desired modalities A and C are conditionally independent given the intermediate "bridge" modality B. Formally, A 1 C B
While this assumption may appear limiting, proves necessary for meaningful analysis the relationship between (A, C') within our problem setting. That is, given three random variables A, B, C. and oracle access to any probabilities involving (A, B) and (B, C') separately, but not to any probabilities involving all three variables together, it is generally impossible to uniquely determine P(CIA) without additional assumptions. For a full proof, refer Lemma in the Appendix.
Assumption Applying contrastive learning to the symmetrized InfoNCE objective results in representations that model a density ratio: (B())(A))en(B)
p(B(A)
K p))(B)
We denote ΦM the data encoder for modality M, f(.)) latent representations to a real number, f Rdd x Rd - R, and K is constant representing the approximation error. This assumption is well justified by prior work (Poole al. 2019 Ma & Collins
function that maps d-dimensional
3

---

Published as a conference paper at ICLR 2025
2018), which proves that contrastive representations converge this probability ratio under some assumptions. Nonetheless, we state this as an "assumption" to clarify that it could be violated in practice (e.g., if data is limited).
The last assumption is needed in just part of our analysis, when understanding the settings where the "plug-n-play" heuristic used by prior work is formally justified. The assumption is that the representations learned by contrastive learning have a certain marginal distribution.
Assumption 3. Contrastive learning under symmetrized InfoNCE with an inner product f(x y) & IJ y and unit-norm latent vectors acquires representations whose marginal distribution p(ő) & p()) (ors (a) = dr is uniform over the hypersphere sd-1: p( (($6-1).
Prior work (Wang & Isola 2020) provides a theoretical basis for why this assumption should hold. We also test whether Assumption 3 holds in practice. See Section 6.2.2
4 BAYESIAN REASONING OVER CONTRASTIVE REPRESENTATIONS
We show that representations learned jointly under InfoNCE correctly infer density ratios. Assump- tions and 2 will allow evaluate the direct connection between two unpaired modalities by using the Bayesian approach of integrating over the intermediate modality. Then, under Assumption we can use the marginal distribution the intermediate representation to reach a closed-form solution for the relationship between our disparate modalities.
3
4.1 MARGINALIZATION WITH CONTRASTIVE REPRESENTATIONS
The first part our analysis shows how to evaluate the correct marginal density ratio between unpaired data modalities A and C.
Lemma 1. Let @ (a), òB (b), (c(c) be three encoders trained with contrastive learning on paired data p(a, b) and p(b. c). Assume that the encoder pairs (0x(a), ón (b)) and (øn(b), @c(c)) satisfy The marginal density ratio
Assumption 2 and that our modalities A, B, C satisfy Assumption 7
between A and C can be expressed as
pCC A) K1 K2 Eóg PCC)
f(bB. oc) (oa.d
in ap- where K1, K2 respectively denote the constant multiplicative enrors of (6 ΦB), (ØB, Φc) proximating the marginal density ratio from Assumption
This result holds for any choice of f() Φ2) that is consistent with Assumption This expression what proper Bayesian would do, if she were allowed to make additional assumptions (i.e., Assumption 33. Note that the expression is reminiscent of doing message passing (Yedidia et al. 2003) on graphical model + B ++ C, except where the node potentials tell about probability ratios instead of probabilities.
Proof. The proof follows by using Assumptions I and together with Bayes' Rule: p(C A)
p(C) = { p(C| B) p(B A)
p(C) p(B) pBB)dB
{ K1 c(dc)))) K2 (osinaa) p(B) dB K₂ K2 Bøn [cllterbe)+
Sec. Dcc.5)esccne desscribes a practical and efficient approach for approximating this expectation with Monte Carlo samples. It is worth noting that Lemma] iis not what is used today; this expression is different from directly comparing the representations ((A) and ((C), a baseline that we will compare against and discuss at length in the subsequent sections. However, in the following section, we will show that a version of the direct dot-product comparison is theoretically justified under additional assumptions. This will enable to obtain an expression for the probability ratio without the need for expectation. For subsequent derivations, we assume a constant approximation error and omit the Ki K2 term for clarity.
4

---

Published as a conference paper at ICLR 2025
4.2 BUILDING INTUITION VIA THE TRIANGLE INEQUALITY
Before presenting our main result, we attempt to build intuition for why directly comparing the representations of Φf A) and ((C) would ever be good idea. We will do this in the setting where the representations are normalized, and measure the representation similarity as f(01, = o1 Φ2 our subsequent results will also examine different choices. With this choice of critic, we can get lower bound on the log ratio:
log P(C) = log (B) (o((A)T((B)-oC) v[O)] 2 log Es(B)
(e(l)) [ = ((A)
The first inequality follows from the triangle inequality (note that the vectors are normalized). We removed the expectation in the third expression because the term inside the expectation does not depend on d(B).
While this identity provides some intuition into why there should be a connection between the inner products and the log ratio, it not the identity that we want: we want to estimate the log ratio itself, not lower bound on it. Additionally, while the analysis above is tight in the setting where oi A) and (CC are orthogonal (akin to the Pythagorean theorem), this would imply that A and C are independent, contradicting
intuition that modality A should tell us something about modality C. Thus, we will need a different proof technique for our main result.
4.3 THE "LAwW" OF THE UNCONSCIOUS CONTRASTIVE LEARNER
This section provides the main theoretical result of the paper, showing that a commonly heuristic is justified under some assumptions. Prior work often assumes that the dot product between repre- sentations from unpaired modalities meaningfully describes their similarity, somehow marginalizing over any uncertainty. Our proof here provides some theoretical justification for that claim, alongside the necessary assumptions.
Lemma 2 (The "Law" of the Unconscious Contrastive Learner). Let ( (c(() be three encoders trained with contrastive learning on paired data p(a, b) and p(b, c) to output normalized rep- resentations (i.e., || = 1) using f f01, 02) = $1 Φ2. Assume that the underlying joint distribution pla, c) satisfies Assumption [ Assume that the encoder pairs (á @B(b)) and (db(b), oc(c))
satisfy Assumption 2
and assume that all three encoders satisfy Assumption
PÍCIA)
Then, the probability ratio is a deterministic function of the inner product,
P(C)
a(da(A) (C()), where g(x) = (2m)p/31 10/2 1(x) [m
is monotonically increasing function of
defined in terms of a modified Bessel function I
This Lemma is important because provides theoretical grounding for the commonly used heuristic of comparing representational similarity between modalities unseen together during training.
Proof. Before starting the proof, we note that it will make use of the von-Mises- Fiisher distri- bution, probability density function over the unit hypersphere that has density f(;; p,,
(c(*))xp(xx)) T z) where the normalizing constant (x) (2m))/²1 1.12 106) is defined in terms
20/-11
Ip the modified Bessel function of the first kind.
Our proof starts by applying Lemma with f(ø1, (22) = -I t2, and then proceeds by using Assump- tion 3 to write the marginal p(o(B))
P(C A)
BO(B)
( (BB)+(()TTCC))
P(C)
(MBB)(((A)+())))) dōn
ga
T ((B) ddB
where [(x(A) + 6(C) and p= [[((A)+ + +CC)ll
Le CalssheonTotB døb-
1
5

---

Published as a conference paper at ICLR 2025
To conclude, note that can be written in terms of the inner product between @a(a) (b) the representations norms equal 1: C=((l(())++(CII)) Cx(v2+0(A) ((B)) - ((((A) (B)).
4.4 EXTENSION TO UNNORMALIZED REPRESENTATIONS
In Appendix B we extend this analysis to the case unnormalized representations with the following Lemma. Here we will assume that the representation distribution is Gaussian, rather than uniform (as in Assumption 3); see Wang Isola (2000) Eysenbach et al. (2024) for a justification this assumption.
Lemma 3. Consider applying contrastive learning with the symmetric infoNCE loss and encoder
A ΦB, dc on paired data p(A, B3) and p(B, C)), using a critic function parametrized as f(r y) & 2212 -ll-- Following prior work, assume that the acquired representations have marginal
distribution that is an isotropic Gaussian: p(6) - [ p(x) (0) = o) = N(mi; 0, 1). Then, the learned encoders QA and &e encode the true probability ratio p(CJA) despite never being trained on pairs of (A, C)) data:
P(C]
pCC A)
K expl-> (I(((A) (08))
(C)
where = 1c+2
c+1 8 = ctI
Note that + oc, we have 5 - 0. The proof can be found in Appendix B
In Lemmas and 3 we recover closed-form expressions for the marginal probability ratio of modalities A and C with just their representations ((A) and o(C') respectively. For the dot product critic, the log probability ratio can be computed as a monotonically increasing transform of the dot product between ((A) and @CC)): and for the negative 12 distance critic, the log probability ratio asymptotically approaches the negative l2 distance between ((A) and ((C) as e increases. We see that the relationship between ((A) and @CC) for both normalized and unnormalized representations closely resemble their respective critic functions! We refer to this phenomenon as the "Law" of the Unconscious Contrastive Learner.
A PRACTICAL ALGORITHM FOR WHEN THE "LAW" DOES NOT HOL.D.
In this section we turn Lemma into practical algorithm for estimating the log ratio p(CIA)
P[C] given
inputs A and C. We will form Monte Carlo approximation of the expectation in Lemma
p(C A) P(C)
K
N E exp(f((A. f(o oc)}
where o; ~ (())(B)) the representation randomly sampled example of modality B. The ap- proximation here reflects sampling eπror for the Monte Carlo summation and function approximation error from contrastive learning. the limit of infinite samples (N) and contrastive representations trained to convergence on infinite data, the approximation becomes exact. Note that this result requires Assumptions I but not Assumption 3
In settings where we want compute this probability ratio for many different values of A and C, we can save compute by precomputing matrix representations +b (ø(B), BBB(B] N
p(C A) P(C)
1 N
N
K
expl (oll((AA.)))) 2.) + f(2, f(01,00)]
This expression then can be computed very efficiently in modern deep learning libraries (PyTorch, JAX) LogSumExp. We study the efficacy of this method in Sec. Sec.|6land show how it can enable new applications in Sections 6.2 and 6.3 For brevity, we will refer to this method Monte Carlo approximation the LogSumExp (LSSE) method.

---

We learn contrastive representations A ΦB, &c using three choices of similarity function f( d2)). For each, we measure the retrieval accuracy throughout the course of learning. Our gold standard is "Ground Truth" method that has (privileged) access (a, c) pairs, data that the other methods do not assume to have. The "Direct" method performs retrieval by comparing the representations A and Φc, representations that were not trained together, using the similarity function; the success of Published as a conference paper at ICLR 2025
darton Accuracy Barred
Mode!
(a) ((01,02)
I
(b) [() (2)) 01 022
(c) f(0i, (ba)
1 P
Figure 2: Testing the "Law" of the Unconscious Contrastive Learning with three parametrizations of the critic function. Wc assess whether the "Law" holds by comparing the success of the "Direct" method to oracle that trained (A, C)) examples. We also include the Monte Carlo method based on Lemma] to understand the assumptions belying our analysis. (Left) For the L2 critic, methods perform well, suggesting that all three assumptions are satisfied. (Center) For the dot product critic, the Direct method performs much worse than the Monte Carlo method, suggesting that Assumption 3 violated. Fig. conffrmms that Assumption is violated. (Center) The performance of the Direct and Monte Carlo methods when using the normalized dot product suggests that normalized dot product violates Assumption 2 but that this assumption might not be necessary for the "Law" to hold.
EXPERIMENTS
The primary aim of our experiments is to validate our theoretical results and understand how those results depend on the aforementioned assumptions. We also demonstrate how Lemma oppens up new avenues for using contrastive representations: for combining multiple pre-trained models, and for correctly handling ambiguity a
conditioned reinforcement learning problem.
We start by describing the experimental setup we use for most of the experiments. Appendix Fig. runs an additional experiment studying the influence of Assumption [ Code for all experiments is available on GitHub
6.1 EXPERIMENTS ON SYNTHETIC DATA
We first test our contrastive models on a synthetic dataset of modalities A, B, C. We generate B from a g e-dimensional Gaussian distribution parameterized by random Covariance matrix Σ. and C are generated from B through a random linear projection (MA E RR MB E R=cxxD) ) as well as additional uncorrelated Gaussian noise (C),CB).
E); A MAB EA.
C:= C:=MBB MB teB.
We generate two separate paired datasets of (A, B) and (B,C) to train our models. We evaluate the model's performance by measuring its retrieval accuracy: given a data point E A we assess whether the model can accurately identify the corresponding data point from a list of 32 candidate data points in C, using maximum similarity score. We compute similarity scores using the model's s critic function. We present I-sigma error-bar plots to show the mean and standard deviation of the validation accuracy over many trials, which is computed every epoch.
We train on datasets with 5000 pairs and evaluate accuracy on validation dataset of size 1000. For all synthetic experiments, we define accuracy as the Recall @ 1 metric: for each query item, we rank all candidate items by their similarity scores and check the correct match appears at rank 1. run the experiment for 20 trials with random seeds for μr Σ, MA, MR.
6.1.1 TESTING THE "LAW" OF THE UNCONSCIOUS CONTRASTIVE LEARNER
The aim of this section is to develop better understanding of the assumptions behind the "Law." We will see the "Law" need not hold when the assumptions are violated.
7

---

Published as a conference paper at ICLR 2025
this method tells us about when the "Law" holds. We also compare with the "Monte Carlo" method suggested by Lemma[] []] This method makes fewer assumptions, so its performance helps us figure out the role of different assumptions in the "Law."
We report results in Fig. In Fig. Za
we see that methods perform well, suggesting that the a gap between the Monte Carlo method and the Direct
assumptions satisfied. In Fig. 2b method, suggesting that Assumption 3 violated. While we do not provide formal proofs regarding the unnormalized dot product, we hypothesize that its representation distribution is highly skewed and fails to meet the sufficient conditions for our "Law". This hypothesis is supported by Fig. 7 which illustrates that the unnormalized representations (green) exhibit irregular behavior, violating key assumption of our framework. In addition, the gap between the Monte Carlo method and the Ground Truth methods here suggests that there may be optimization challenges in satisfying Assumption with the dot product critic. Finally, Fig. [2c shows that the Monte Carlo method performs poorly when using a normalized dot product. Intuitively, this makes sense, as this critic cannot represent log probabilities outside [1/e, el However, the Direct method still achieves excellent results in the setting. Fig. 7 suggests that the representations do satisfy Assumption but the reasoning above suggests they likely violate Assumption 2]
In summary, our results show that our theoretical analysis provides sufficient set of conditions for the "Law" to hold, and suggest remedy for settings where Assumption B violated. However, the results in Fig. 2c suggest that these conditions may not always be necessary. opening the door to future work find even less restrictive conditions under which the "Law" holds.
6.2 BRIDGING MODALITIES WITH PRETRAINED MODELS OR BLACK Box APIS
In many practical applications, users only have access to the similarity scores between pairs of modalities, but nonetheless want to reason about the similarities of unpaired modalities. For example, let's say a user wants to compute the similarity between A ++ C. but searching the internet they find one pretrained contrastive model for A ++ B and (e.g., in some other GitHub repository) another pre-trained contrastive model for B ++ C. These contrastive models may have different architectures and their representations may have different sizes. As another example, imagine that there are two MI. web services, one that returns the similarity score between modalities A and B (c.g., a radiology service that aligns X-rays with text) and a second that computes the similarity score between modalities B and C' (e.g., dictation service that aligns text with speech). The underlying models are proprietary, so the user cannot inspect their weights.
Validation Accuracy Unconrected Model
1.0
0.8
Accurscy
0.6-
(No
CA
10
75
20
30
Spsch
Figure 3: A principled way of combining pre- trained models. Given pre-trained models that compute the similarities A 4+ B and B +> C, we use Lemma to infer the similarity between A 4+ C.
The approach introduced in Sec. 5 handles both these problem settings. We first evaluate this approach on the synthetic benchmark described in Sec. 6.1 measuring performance by retrieval accuracy. We train contrastive critic O dB, and ΦB, + cc, using the l2 critic for both. We show results in Fig. 3 As expected, direct evaluation between representations A and &c performs no better than that the Monte Carlo method achieves accuracy
chance. However, supporting our theory, rate approaching that of an oracle that is trained on aligned (a, c) pairs.
6.2.1 APPLICATION: LARGE TRE-TRAINED LANGUAGE MODELS
We corroborate the above results with experiments on real-world data, confirming our theoretical results. We apply our methodology to three models CLIP, large pre-trained model Radford et al. (202T): CLAP, a large pre- traitre audio-langen languag model Elizalde al. (2023);
and LanguageBind, a massively multimodal pre-trained contrastive model connecting modalities
Including temperature in the critic expands the range of values, hut the range remains finite.
8

---

Published as a conference paper at ICLR 2025
(vision, audio, infrared, depth, etc.) via shared language modality Zhu al. (0223). Using these three models, we evaluate the performance of zero-shot Audio Visual alignment. First, we assess CLIP/CLAP encoders to demonstrate the effectivvnness of our Monte Carlo algorithm. Subsequently, we evaluate LanguageBind encoders to validate our proposed "Law".
We begin by showing how Monte Carlo Method (LogSumExp) over pre-trained CLIP and CLAP mod- els unlocks Audio-Visur inference with no additional training required. For direct evaluation, we measure the similarity score as the normalized dot product between the CLIP image encoder with the CLAP au- dio encoder, which expected to do no better than random chance. We then apply our LogSumExp algo- rithm the intermediate 'language' modality, using the CLIP and CLAP language encoders to connect the originally disparate Image and Audio encoders. We evaluate these models AudioSet Gemmeke et al. (2017) a dataset of short clips from YouTube with corresponding Audio descriptions. To perform Monte Carlo (LopSumEXp)) method over the interme- diate 'language' modality, we approximate the true distribution of intermediate language embeddings us- ing the AudioSet ontology, which defines universe of language descriptions that the clips fall into.
Direct Evaluation
LogSumExp
0.70
Accuracy (recall@
0.62
0.58
0.5
0.14
0.0
LanguageBind Figure 4: (Left) Direct evaluation of the CLIP Image Encoder audio-visual inference versus using our LogSum- Exp algorithm with those same encoders. (Right) Direct evaluation with LanguageBind encoders vs our LogSumExp algorithm with the those same en- coders. In Appendix Fig. we show that the 12% gap between the Directe Evaluation and LogSum- Exp on LanguageBind is caused by using too few Monte Carlo samples; this gap shrinks the number Monte Carlo samples is increased, in line with our theory.
CLIP and CLAP
CLAP Audio Encoder for
Given just the language ontology and pre-trained CLIP and CLAP models, we are able to achieve 62% Recall@ 10 using our Monte Carlo method. This beats the naive baseline of directly evaluating the similarity scores between the image embedding of CLIP and the audio embedding of CLAP, which achieves 14% Recall @ 10 accuracy. We also compare the performance our "Law" in its architecture. LanguageBind trains multiple encoders contrastively through an in- termediate language modality, achieving superior results on many multi-modal benchmarks. Direct evaluation with LanguageBind achieves 70% recall on the AudioSet benchmark, which further empirically validates our "Law". Our Monte Carlo approximation on the LanguageBinn encoders achieves 58% recall. However, by scaling the intermediate embeddings beyond just the AudioSet ontology, we can close this performance gap (see Appendix C.I
our LogSumExp algorithm against LanguageBind, which implicitly assumes
Our Monte Carlo approximation on CLIP/CLAP, while achieving lower accuracy than Language (62% vs 70% Recall@ 10), offers significant advantages. Notably, it enables the connection of previously disjoint models without additional training, demonstrating its flexibility and efficiency. Furthermore, our method requires data only from the intermediate modality, in contrast to prior approaches that demand data from all three modalities for full alignment Wang et al. (2023). This reduction in data requirements represents substantial practical benefit, potentially broadening the ap- plicability of multi-modal inference across diverse domains where comprehensive datasets are scarce.
6.2.2 TESTING REPRESENTATION UNIFORMITY IN COMPLEX DATA MODALITIES
We additionally validate that Assumption fares well in complex real-world settings. We compute the representations language descriptions for each model over the AudioSet ontology and perform two-sample Kolmogorov-Smimon test, comparing the distribution over representations to uniform hypersphere distribution. CLIP (p-value 0.0877) and CLAP (p-value = 0. 1788) both fail to show significant deviation from the uniform hyperspherical distribution.
6.3 APPLICATION:
CONDUAIGECODD REINFORCEMENT LEARNING
Our analysis not only helps explain the zero-shot capabilities of contrastive learning methods but also enables novel applications in reinforcement learning. We demonstrate this through language- conditioned navigation tasks, where learned representations can be highly non con-unfform and non- Gaussian, limiting direct evaluation effectiveness. Our Monte Carlo LogSumExp method significantly

---

Published as a conference paper at ICLR 2025
improves performance in these scenarios, boosting success rates by 20%-30% across different environments.
We design synthetic benchmark using continuous PointMaze environment to evaluate language understanding navigation. The environment consists of a grid where rows and columns are annotated with natural language descriptions (e.g.. "the first column," "the second-to-lass row"). The agent can move freely at any angle and must plan paths to reach locations specified through potentially ambiguous language descriptions We frame this reinforcement learning problem where an agent observes state 8t, takes action O₁, and transitions to state 8:4 The objective is to select actions that reach the language-described destination minimal steps.
To map this onto our multimodal framework from Sec. we treat modality A as state-action pairs (s,a), a), modality B as future states sy sampled several steps ahead, and modality C as language descriptions E of those future states. We learn encoders by aligning (s(s,a) + Φ (& and Φ < (c) The first alignment corresponds to contrastive reinforcement learning (Eysenbach et al. 2023 which learns policies by maximizing (8 a) ( between state-action representations and goal state representations.
Baseline, Our primary comparison is against the direct representation comparison approach widely
used in robotics and RL. (Lynch et al. [202] [Sontakke et al. 2024) Tziafas al. 2023), often utilizing CLIP embeddings (Radford et al. (022)) This baseline selects actions max, & (s, a) (c()).
These methods implicitly employ the law of the unconscious contrastive learner, and our experiments
test whether this assumption is appropriate when uncertainty quantification matters.
Our approach. Following Lemma and Sec. 5 instead of directly comparing state-action and language representations, we marginalize over future states:
lax
Σ
(ba(s,a) (olsr)toulss)) (c
f5mp(ay)
This directly maximizes the probability that future states satisfy the desired language description.
Our method demonstrates superior performance across three challenging environments (see Ap- pendix D for detailed results). key illustrative example occurs our fork maze environment (Figure 9). where agents must navigate to "the first column." The direct method's primary failure mode is its tendency to move toward the mean position of ambiguous descriptions rather than finding optimal paths. For instance, when starting from position (2, 6). direct evaluation takes suboptimal rightward path before eventually reaching the goal through a different alleyway. contrast, our LogSumExp method correctly identifies and follows the optimal leftward trajectory.
This performance disparity stems from the LogSumExp method's ability to maintain the full distribu- tion over possible goal states rather than reducing ambiguous language to averaged embeddings. By summing over all intermediate states, our approach can identify actions that maximize the probability of reaching valid goals, even under ambiguous language specifications. For comprehensive analysis of the learned representations and additional experimental results, we refer readers to Appendix D
7 CONCLUSION
This paper has two aims: to understand how to use contrastive representations to correctly reason about uncertainty, and to figure out when and why the commonly-used "direct comparison" heuristic works effectively. Our results provide a principled method for reasoning across unpaired modalities, along with analysis suggesting that the heuristic is principled, under some assumptions.
Limitations. The main limitation of our paper is that it does not provide the final word on when users should prefer the "direct comparison" approach over the Monte Carlo approach. In future work, we would like to design a simple test that users can apply to their data to understand whether satisfies the assumptions, and guide users in selecting algorithm for reasoning across modalities.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsca Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine. Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not as say: Grounding language in robotic affordances, 2022. URL https: / arwiv org/ aos 2204 01691
Theo Cachet, Christopher R Dance, and Olivier Sigaud. Bridging environments and language with rendering functions and vision-language models. In Forty-first International Conference on Machine Learning, 2024. UBBL/tttps://ppeneeriie net www.ooommnddnioo....
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffry Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR. 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9640-9699, 2021.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. TEEE, 2023.
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. 2023.
Benjamin Eysenbach, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. Inference via interpolation: Contrastive representations provably enable planning and inference. arXiv preprint arXiv:2403.0400 2024.
Andres Ferraro, Jachun Kim, Sergio Oramas, Andreas Ehmann, and Fabien Gouyon. Contrastive learning for cross-modal artist retrieval, 2023.
Jort F. Gemmeke, Daniel W. Ellis. Dylan Freedman, Aren Jansen, Wade Lawrence. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, 2017. doi: 10. 10.1109/CAASPP.2017..9575
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 3180-15190, 2023.
Scan-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33: 21271-21284, 2020.
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio, 2021.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 066) volume pp. 1735-1742. IEEE, 2006.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong. and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv: 1809,088/2, 2018.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 301.3788,, 2013.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /tt.::/807033772 2018
11

---

Published as a conference paper at ICLR 2025
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, and Sergey Levine Vint: A foundation model for visual navigation, 2023.
Sumedh Sontakke, Jesse Zhang, Séb Amold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboelip: One demonstration is enough to learn rohot policies. Advances in Neural Information Processing Systems, 36, 2024.
Georgios Tziafas, Yucheng Xu, Arushi Goel, Mohammadreza Kasaei, Zhibin Li, and Hamidreza Kasaei. Language-guiee guidedd robot grasping: Clip-based referring grasp synthesis in clutter. arXiv preprint arXiv:2311.05777 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, 9929-9939. PMLR, 2020.
Zehan Wang, Yang Zhao, Xize Cheng, Haifeng Huang. Jiageng Liu, Li Tang, Linjun Li, Yongqi Wang, Aoxiong Yin. Ziang Zhang, and Zhou Zhao. Connecting muilii-odda contrastive representations, 2023.
Le Xue, Mingfei Gao, Chen Xing, Roberto Niebles, and Silvio Savarese. Ulip: Learning unified representation of language, images, and point clouds for 3d understanding, 2023.
Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos
Jonathan Yedidia, William T Freeman, Yair Weiss, et al. Understanding belief propagation and its generaliza- tions. Exploring artificial intelligence the new millennium, 8(236-239):0018 239):0018-9944400 -94488 2003.
Mengyi Yuan, Minjie Chen, and Xiang Li. Muse: Multi view contrastive Icarning for heterophilic graphs, 2023.
Bin Zhu, Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.0183) 2023.
A AN IMPOSSIBILITY RESULT
Lemma 4, Given three random variables A, B, C, and oracle access to any probabilities involving (A, B) and (B, C)) separately, but not to any probabilities involving all three variables together, generally impossible to uniquely determine P(C)A) without additional assumptions.
Proof. We know that
P(CIA)= P(CIA, ((B]A).
However, both p(B]A) and p(C)A, depend the unknown joint distribution p(A, B, C). Knowing p(A, B) and p(B,CT C") is insufficient to uniquely determine p(A, B, C)), as there exist multiple joint distributions can share the same marginals p(A, B) and p(B, C) but differ in their dependence structure between A and C' given B.
For example, let A, B, and C' be binary random variables with given joint distributions p(A, B) and p(B, C) that are uniform over possible pairs.
Table 1: Joint distribution p(A, B)
Table 2: Joint distribution p(B,C)
p(A B)
=
B = 1
p(B,C)
C 0
C=1 1
A = 0
0.25
0.25
B 0
0.25
0.25
A =
0.25
0.25
==1
0.25
0.25
12

---

Published as a conference paper at ICLR 2025
We can construct two different joint distributions consistent with these marginals: where A and are independent given (yielding ((C = 0.5 for all A), and another where A and C perfectly correlated (so p(C|A) = when C' A). Despite both joint distributions matching the given p(A A, B) and pi B, C), they result in different A)C demonstrating that p(C|A) is not uniquely determined without additional information.
B LAW OF THE UNCONSCIOUS CONTRASTIVE LEARNING (GAUSSIAN SETTING)
In the main text, we focused on the setting where representations lay on the unit hypersphere. Lemma L extendde the analysis unnormalized representations, whose distribution a Gaussian, an assumption that is theoretically motivated based on prior work (Wang & Isola 2020 Eysenbach al. 2024). Here, we provide a proof of that Lemma.
be
Proof. We evaluate the integral under <(B) NO c I). We start with the expectation:
p(C|A)
(CC) EB(B) (e (IIoCC)) (10
(1)
=e- (Io(A)) ((C))))) EB( -1666B))1=0000 (0(A))
Because (BB) is Gaussian, we know p(o(B)) = (2m0) e - 100 Thus,
P(CIA)
p(C) e - =(0(44)(++1((()))))
/ e (L) ( do(B)
(2Tc)///2
JR
(3)
(2 e ((o(a1))++((((())))
/ +I++)))
(( (o(A)+
J
Letting a = 1+ 2n
y = ((A) + (CC), we complete the square to get
A(C(A)
+((0(4)/++1((())))
liut
# --
-aloo(
=l'(B(BB))
p(C)
k/
4m
E
(5)
JRk
e - -[(((((A)))+1101))))))))
llvl" 4u
k/2
(6)
(2
k/2
π
(160)))))) ((C)))I(C0)))))))))
(2mc))/22
(
e
(7)
K exp {
c+1
(2211) ((A) B(OO}
(
(8)
4c
2c 2c+1
Simplifying further, we finally arrive at
D(C1A) K expl-7 exp(->(1o(A)) - 56((A) (Ci))
p(C)
where 7
0+1 2 4c421 c+T'
C ADDITIONAL EXPERIMENTS
C.1 SCALING UP THE MONTE CARLO APPROXIMATION FOR LANGUAGEBIND AND IMAGEBIND
In Section 6.2. we evaluated our Monte Carlo approximation against direct computation using
I Bing AudioSet data, observing 12% performance gap. Given that Monte Carlo
13

---

Published as a conference paper at ICLR 2025
Acsureo Number Mame Carlo Sample 150 msatal inagedidd
Acouncty Namber Marde Carte Samplis maatll Largguagedd Nar Dara no
Hariae paro madia
4.2
Nenbiw Mosta
Hontow
(a) @ecall@| Accuracy for ImageBind
(b) Recall @ Accuracy for LanguageBind
Figure 5: The Accuracy of our LogSumExp (Monte Carlo) approximation scales with the number of intermediate embeddings. As the number of sampled embeddings (M) increases, the Monte Carlo method converges to direct evaluation performance with both ImageBind (left) and LanguageBinn (right). The shaded regions indicate 95% confidence intervals across multiple trials, and the dashed red lines represent the accuracy of direct computation. Recall is evaluated from set of 25 samples. These results validate our theoretical analysis and support the underlying assumptions of our approach.
method relies sampling intermediate embeddings, and the AudioSet ontology provides only 600-700 samples for this intermediate space, we investigate whether this gap stems from insufficient sampling rather theoretical limitations.
We conduct a systematic scaling analysis using audio, image, and text modality triples (A, B, C) from YouTube videos provided by AudioSet. We evaluate on two state-of-the-ar multimodal models: LanguageBind, which aligns modalities through shared language descriptions, and ImageBind, which aligns modalities through shared image spaces.
Figure demmonsraessss that as we increase the number of intermediate samples (M) from 1 to 500,000, our Monte Carlo approximation converges to direct evaluation performance for both models. This convergence across different multimodal models validates our hypothesis that the initial performance gap was primarily due to limited sampling rather than a limitation of our method.
C.2 SWAPPING MODALITIES WITHIN THE SAME SETUP
To further validate our theoretical findings, we conduct additional real-world experiments by varying the intermediate modality while maintaining the same experimental framework from Section 6.2.
C.2.1 EXPERIMENTAL SETUP
Recall that our original setup in Section 6.2.1 leverages CLIP IiaageLLanguage and CLAP (Audio-
Language) to align image and audio modalities through their shared language encoder. We design two additional experimental configurations:
1) Image-Language Alignment via Audio: We utilize ImageBind Girdhar et al. (023) as our Image- Audio model paired with CLAP as our Audio-L Language model. ImageBind provides high-quality audio-vine vndio-visual representations trained with InfoNCE loss. aligning with theoretical assumptions. We evaluate our Monte Carlo algorithm against baselines from ImageBind (Direct Evaluation using ImageBind's Vision and Language encoders) well CLIP.
Audio-Language
2) Audio- Audio-Language Alignment via Images:
We utilize ImageBind as our Image-Audio model paired with CLIP our Image-Language model. We evaluate our Monte Carlo algorithm against baselines from ImageBind (Direct Evaluation using ImageBind's Audio and Language encoders) as well as CLAP.
14

---

Published as a conference paper at ICLR 2025
Language Alignment 1
Audio-Language Alignment Recall@1
0.324
0:497
Recalig Accuracy
0.3
Recallas Accuracy
0.291
0.294
0.2
0.1
ImageSind
LogSuimExp (CLAPe Imagekind)
CLIP
ImageBind
LogSumExp
CLAP
(CLIP (CLiP-magealnn)
(a) Recall Accuracy for Aligning Vision and Lan- guage.
(b) Recall@ guage
Accuracy for Aligning Audio and Lan-
Figure 6: LogSumExp performs well with other intermediate modality types (audio and vision). The error bars indicate 95% confidence intervals across 100 trials. Recall@ @ is evaluated from a set of 25 samples. These results strengthen our real-world experiment in Section 6.2. and further validate our theoretical analysis.
C.2.2 RESULTS AND ANALYSIS
For both Vision-Languaae and Audio-Language tasks, we compare our LogSumExp method against ImageBind's direct evaluation, which implicitly implements the "Law". We find that LogSumExp accuracy closely matches direct evaluation:
In Vision-Language alignment, LogSumExp achieves a Recall @ of 31.4% + 1.7%, com-
pared ImageBind'ss 's direct evaluation performance of 32.4% 1.5%
For Audio-Language alignment, LogSumExp reaches 29.4% +: 3.3% versus ImageBind's 29.1% 1.7%
Baselines against Single-Mode Baselines (CLIP and CLAP in purple) are provided in Figure 6 The near-identical performance between LogSumExp and Direct Evaluation validates that similar phenom- ena to Section 6.2. [.2.1Jhol hold regardless of intermediate modality type. This provides additional empirical support for theoretical framework, suggesting that our Monte Carlo algorithm generalizes across different choices of intermediate representations.
C.3 REPRESENTATION DISTRIBUTION IN R
InfoNCE Embedding Distribution
-2
norm dist/model
model
Figure 7: Embedding Distributions under different critic functions
For a subset of synthetic data experiments, we run with a latent embedding size of 2, which allows us visualize the resulting representation distributions. See Figure for the plots.
15

---

Published as a conference paper at ICLR 2025
C.4 ABLATION ON CONDITIONAL INDEPENDENCE
Figure 8: Performance degrades when Assumption is violated. The x-axis corresponds to the
magnitude of
Assumption requires the conditional independence of A and C' given modality B. What happens when this assumption is violated? With the separating intermediate modality, becomes impossible to observe the correlation between A, C. When this happens, we do not have p(C A, B) p(C) anymore, so our analysis becomes approximate.
We construct synthetic dataset for this case, building off the existing synthetic dataset in Section 6 We now add Gaussian random variable x R² to modality pairs A and C that is unseen by B. Thus, A, C' are serially correlated independent of B.
~ N(w; E)
(9)
A = MAB + + kî
(10)
Mp +B +I
(11)
We denote I as the all-ones vector: When x = O, our conditional independence assumption is satisfied. As grows, our conditional independence of A, C given decreases.
See Figure g for the accuracy plots. The y-axis denotes the retrieval accuracy of the methods and the r-axis denotes the magnitude of our correlation variable k, described in the previous section. We see that the accuracy of our models unanimously goes down as our constant disturbance increases.
D AN APPLICATION TO CONTRASTIVE REINFORCEMENT LEARNING
We delve briefly into contrastive reinforcement learning. Contrastive representations lend themselves very well to self-supervised reinforcement learning due to their inherent encoding of probability distributions over future states. Interestingly, for the same reason, contrastive learning also parsimo- niously encodes uncertainty between image and text. Thus, if we are able to marry these two methods together under our framework, we will have method for language goal-conditioned reinforcement learning. This is the setting where we specify language goal of future state, and try to learn an algorithm that can reach that specified language state without any explicit rewards.
D.I THE ENVIRONMENT
We define a continuous grid environment for the player to move around in. The player can move around freely at any angle, and is trying to reach pre-specified goal state. We annotate the grid environment with natural language descriptions of grid positions. Examples include each row and column in the grid being given street and avenue name respectively. The language labels such that many of them are ambiguous, sometimes referring only to certain rows or blocks. This is to test the compositionality of the learned language embeddings and how they incorporate uncertainty.
16

---

Published as a conference paper at ICLR 2025
D.2 TRAINING METHODOLOGY
Building on prior work, we can frame reinforcement learning purely as a contrastive learning problem. We learn embeddings for (s, a) and (sy) pairs, where
current state, := action, 8y == future state
We assume dataset of oracle trajectories is collected, giving (8, a), 8f pairs to learn over. We also assume dataset of states with language annotations. Then using our unconscious method, can simultaneously learn an embedding between s and I == language description.
Learning the (s, a) embedding allows to choose the best action for the agent by taking an arg max over set of candidate actions. In practice, we find that this leads agents down deterministic path that could lead to cycles (an infinite loop). Thus, we sample actions probabilistically, weighted by the exponentiated dot product between the goal embedding and the (s, a) embedding.
This application is especially beautiful because contrastive representations allow us to capture inherent uncerlainty in language, and these representations can then be directly used downstream for the control task, due to the fact that both use contrastively-leara embeddings as the latent space.
Because the reinforcement learning algorithm only requires estimation of the marginal probability ratio, we can use pre-trained representations for both contrastive tasks.
D.3 CONNECTION TO EXISTING LANGUAGE- LANGUAGE-CONDITIONA REINFORCEMENT LEARNING METHODS
Our proposed "Law" bears significant resemblance to techniques used in prior works for language- conditioned tasks. The majority of existing research in language-conditioned reinforcement learning approaches the problem as a moodality-bridginn challenge, aiming to connect textual-visun align- ment with goal-conditioned reinforcement learning. Our "Law" provides principled probabilistic perspective that formalizes and justifies best practices for integrating these two sub-tasks.
Text2Control Cachet al. (2024) is one example of this approach. The algorithm decomposes the problem into two stages:
1. Text-to-goal generation
2. Goal-reaching
This work employs a Vision-Language Model (VL.)) to translate textual goal into goal 'configura- tion' image that can then be fed into a goal-reaching policy. However, this decomposition introduces critical flaw: may lead to the selection of 'impossible' configurations given the initial state. This is a severe limitation that can significantly impair the system's 's performance and reliability.
In separate line of work, Ahn al. (2022) introduces different approach. This method computes feasibility (" ('affordance") scores for future states conditioned on the current state. These feasibility scores are then combined with 'Instruction Relevance' scores determine the agent's policy.
Our LogSumExp method bears striking similarities to Ahn et al. (2022), as we implicitly compute:
Affordance: p(future state
current state)
Instruction Relevance: p(language description future state)
Our method thus provides unified probabilistic framework that encapsulates and formalizes the intuitions behind these existing approaches. By doing so, offers more rigorous foundation for reinforeement learning, potentially leading to more robust and generalizable algorithms. Furthermore, our approach inherently addresses the limitation of potentially selecting "impossible" configurations, as seen in Text2Control, by considering the affordance of future states.
D.4 RESULTS
Our Monte Carlo method outperforms direct evaluation on all tested maze environments. A subset of the results visualized in Figure 13
17

---

Published as a conference paper at ICLR 2025
Figure 9: Fork Maze Environment with multiple possible paths the "first column"
Figure 10: Direct evaluation chooses subopti- mal rightward action when given eight possible directions (unit vectors), failing find the short- est path to the "first column."
Figure 11: LogSumExp correctly identifies mov- ing left the optimal action for the same navi- gation task.
D.5 AN AMBIGUOUS LANGUAGE EXAMPLE
To demonstrate the advantages of our Monte Carlo LogSumExp approach in handling ambiguous language navigation, we present an illustrative example features fork-like pattern of walls across its left half, with each row forming separate alleyway (Figure 9
x maze environment. The maze
We first train encoders dA,B on state-action pairs and future states (8, a) (8y). and encoders
ÓB.C on states and language descriptions s ++ We then task an agent at position (2, 6) to navigate towards the "first column." While the shortest path is directly left, direct evaluation fails to find this optimal route. As shown in Figure 10 an agent using direct navigation takes significant detour, first moving right before eventually entering different alleyway. This behavior reveals fundamental limitation: direct evaluation cannot find the shortest path to a language-deserinn navigating towards the "mean" position of the ambiguous description. In contrast, Figure I shows that LogSumExp navigation correctly chooses to move left.
destination, instead
This performance difference stems from the state encoder's violation of Assumption 3. To understand LogSumExp's 's decision- decision-making process, [1 description "the first column" (left) and various actions (right) against state embeddings from each grid cell. The Q-value for each action can be understood as the sum of compositions between the left language graph and the respective action graphs on the right. Formally, we compute:
the critic functions for both the language
Σ dals,)) (dls))+ (e)
symp(ay)
The "left" action achieves the highest Q-value because it aligns the most probable "next" states with the nigh-activation "language" states, leading the agent to choose this optimal direction.
This example illustrates key advantage of our LogSumExp approach: its ability to handle ambiguous language descriptions by considering the full distribution over possible goal states. While direct evaluation reduces the language description "first column" to single averaged embedding -eading to suboptimal navigation-onn Monte Carlo LogSumExp method maintains the multimodal nature of the goal distribution. By summing over all possible intermediate states, it can identify that moving left maximizes the probability of reaching valid goal states, when those goals are ambiguously
18

---

Published as a conference paper at ICLR 2025
Language vatiions 8:33 8.3
000
Figure 12: Monte Carlo approximation visualization over the intermediate state modality, sampling uniformly over valid grid points. Left: Average critic function evaluations for each cell with the language description "the first column". Right: Average critic function evaluations for each cell with the state-action pair (current state, action) for up, down, left, and right actions.
specified in language. This capability particularly valuable in real- world applications where natural language commands often have multiple valid interpretations.
19

---

Published as a conference paper at ICLR 2025
Noce with Agont
SuccEo Me Campuroon
Succcn Woighed Hermd et Inegne
Naw WIT Acont
Nerre www.oooegiiiie (Nevr www.twereee Rith imag!
Succuss Heme
Figure 13: Success Rates of Direct Comparison vs LSE on different Grid Environments
20


Reviews:
[{"review_id": "0z7V2MYPLE", "reviewer": "Reviewer_rpJk", "summary": "A common belief in contrastive multimodal learning is that the embedding spaces of seen modality pairs (A&B and B&C) naturally align with those of unseen modality pairs (A&C). This paper introduces the \"Law of the Unconscious Contrastive Learner\", showing that under specific assumptions about the geometric and probabilistic properties of contrastive embeddings, it is possible to establish relationships between unseen modality pairs. The law relies on three key assumptions: the first two allow for evaluating the connection between two unpaired modalities (A&C) via a Bayesian approach that integrates over an intermediate modality (B), while the third enables the use of the intermediate representation marginal distribution to derive a closed-form solution. The law is then formalized into a practical algorithm using Monte Carlo approximations. It is validated through experiments on synthetic and real datasets, including CLIP and CLAP. Results show the alignment of unpaired modalities, though some assumptions may not be strictly necessary in practice.", "questions": "An easy way to strengthen the experimental findings could be to swap the modalities within the same setup. Theoretically, there is no strict assignment of text, image, or audio to roles A, B, or C, so these modalities could be switched to test the robustness of results.\n\n(There appears to be a formatting issue in the proof of Lemma 2 on page 5.)", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. The idea of aligning representations of the same concept or sense, regardless of modality, is valid and taps into a widely-held belief. Although this is commonly accepted, it has not been well grounded in theory. This paper makes strides by validating this belief under certain assumptions. It provides a foundation that may guide further theoretical development in contrastive embedding alignments.\n2. Although not checking line-by-line, the proposed theory assumes probabilistic contrastive learning, allowing connections between unpaired modalities (A&C) to be established across a shared intermediate modality (B) using Bayes Rule. Its algorithm using Monte Carlo approximations is backed by empirical evidence, which is interesting compared to the Oracle methods, where modality pairs (A&C) can be seen.\n3. The paper is well-presented, especially the first half theoretical sections. This clarity makes the framework and assumptions easy to follow. Results shown in Figure 2 are helpful for a good understanding.", "weaknesses": "1. The experimental setup in Section 6 lacks standard organization, making it difficult to immediately grasp the task, input, output, and evaluation metrics. A clearer presentation of each experiment would improve readability and understanding.\n2. There appears to be a gap between the Monte Carlo approximation method and the main theoretical framework. For example, Assumption 3 is noted as not strictly necessary, introducing ambiguity in the experiments and leaving some uncertainty about the necessity of all three proposed assumptions given the results.\n3. Of the three main experiments, both the first (6.1) and third (6.3) are conducted on in-house datasets, and only the second experiment applies the algorithm to an external dataset (AudioSet) by comparing CLIP and CLAP, but its scope, datasets, model types, and modality pairings are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6H33uo0a7U", "reviewer": "Reviewer_i8rh", "summary": "This paper proves under certain assumption that the “hope” of unseen modality pairs will be aligned when the embedding space of models between existing modality pairs are trained contrastively. Using Bayesian approach, the paper shows that directly comparing the representation of data from unpaired modalities can recover the same likelihood ratio. The analysis shows that contrastive representations can answer many of the same inferences as probabilistic graphical models. The result of the paper suggests that the contrastive representations can be used in settings with pre-trained contrastive models, and for handling language ambiguity. Experiments are done to verify the theoretical results over synthetic datasets and more realistic setting.", "questions": "- In equation from 4.2, $\\phi(A)^T\\phi(B) + \\phi(B)^T\\phi(C) \\geq \\phi(A)^T\\phi(C)$ is because the trained pairs are almost guaranteed to be closer than the unseen pair?\n- Missing subscript $C$ for $\\phi_C(C)$ in line 247?\n- Apologize for not being familiar enough with the theory of the work, why is von-Mises-Fisher the distribution in Lemma 2? Is it just because the normalization constant or wanting to match assumption 3?\n- The citation format in section 6.2.1 should be changed. E.g., AudioSet (Gemmeke et al., 2017).\n- Monte Carlo approximation can be expensive. What is the cost in computation and time efficiency?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The three assumptions are clearly stated, and proofs as well as empirical tests for the assumptions are provided. The derivation of the conclusion that the relationship between the $\\phi$ functions for both normalized and unnormalized representation closely resemble their critic functions.\n- Empirical evaluations using the Monte Carlo approximation validates some assumptions’ validity, and also brings the possibility of only use intermediate modality data to align all the modalities’ representations. Experiments are done in both synthetic setup (section 6.1, 6.3) and real-world data (Section 6.2).", "weaknesses": "Section 6.1’s setup might be a bit artificial. Since modality A and C are only a projection of B, it does not quite suffice the general definition of “modality” that most of us agrees on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHQx3Wyasq", "reviewer": "Reviewer_Xnbz", "summary": "Existing work often assumes alignment between different modality pairs without providing a theoretical basis. This paper, motivated by (1) isotropic representations and (2) a Bayesian probabilistic framework, introduces three assumptions that theoretically ensure alignment.\n\nIn a synthetic experiment, the authors first demonstrate that alignment fails when Assumption (iii)—that acquired representations have a marginal distribution following an isotropic Gaussian—is violated. They then show that retrieval accuracy remains high (indicating successful alignment) when Assumptions (i) and (ii), which involve Monte Carlo approximations, are satisfied, using an intermediate modality like language.\n\nFinally, in a real-world application, the authors apply contrastive representations from pre-trained contrastive models within a probabilistic framework to manage language ambiguity in reinforcement learning.", "questions": "See weakness above.\n\n1. Any practical implication based on the assumptions.\n2. Detailed explanations of the reason why the framework can help language ambiguity, is that possible applied to other frameworks. The motivation of taking different multi-modality data into consideration in the RL settings.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes theoretical assumption to guarantee the universally agreed alignment for multi-modalities.\n2. The experiments on synthetic data verifies the correctness of the assumptions. \n3. The real application experiment show promise of the assumption in reinforcement learning for addressing handling language ambiguity.", "weaknesses": "It is not clear why the proposed framework is better at language ambiguity situation, is it because that the framework is based on a probabilistic framework, i.e., the uncertainty. \n\nThe proposed assumptions seem to less contribute to the real-data application, i.e., language ambiguity in reinforcement learning framework, which limits its practical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z5MtN0XhQe", "reviewer": "Reviewer_RYrX", "summary": "This paper tries to provide a theoretical justification to the scenario of using the third modality to connect two modalities which are not explicitly trained during contrastive learning. A typical use case is to conduct image-audio retrieval tasks through CLIP and CLAP with language as the intermediate.\n\nAuthors prove that such heuristic is theoretically grounded, by making a conditional independence assumption (of the two modalities to be\nconnected), along with two ideas published in previous work.\n\nTwo numerical experiments with synthetic and real-world datasets are employed to validate the effectiveness of the proof.", "questions": "Line457. LanguageBind is claimed to assume the proposed Law\nimplicitly. However, when applying the law explicitly, why the\nperformance actually drops? I was expecting no impacts on LanguageBind\nif the Law is implicitly implemented already.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It is desired to have some theoretical justifications to the common\n  use case or practice of connecting two modalities that are not\n  explicitly trained during contrastive learning. There are published\n  work, but not much theoretical justifications contained.", "weaknesses": "- The method derived from the proof, LogSumExp, seems not working well for\n  LanguageBind, and it leads to performance drop (Figure4).\n\n- Overall the proposed proof seems mostly effective on synthetic\n  datasets. I am thus concerned about how strong are the assumptions,\n  especially assumption 1 -- are modalities A (e.g., audio) and C\n  (e.g., image) really independent conditioned on B (language)?\n\n- Writing needs to be improved; Sec. 6.3 experiment is difficult to\n  understand with the main text alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)