Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables
Authors: Yu Gui
Cong Ma
Zongming Ma
Venue: NeurIPS 2025 poster ()
Keywords: contrastive learning; multi-modal learning; machine learning theory;
Primary area: theory

Abstract: Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP~\citep{radford2021learning}. 
In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions. 
Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors. 
Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learning to learn low-dimensional and informative representations, bridging theoretical insights and practical performance.

Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables
Yu Gui
Cong Ma
Zongming Ma
Department Statistics and Data Science, University of Pennsylvania
2 Department of Statistics, University of Chicago 3 Department of Statistics and Data Science, Yale University
yugui wugui@wharton.co upenn edu, congm@uchicago edu, zongming ma@yale edu
Abstract
Multi-modal contrastive learning as self- self-suuer representation learning tech- nique has achieved great success in foundation model training, such as CL.IP [60]. In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions. Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors. Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learn- ing to learn www-imensional and informative representations, bridging theoretical insights and practical performance.
Introduction
The growing availability of data sources has enabled interdisciplinary research leverage multi- modal data, which measures each unit from different aspects with various types of information. For example, the availability of data types including images, text, and audio fostered the advancement of cross-modal foundation models in recent years [39, 45, 70, 60]. Another notable example is single-cell multi-omics technologies [71] which utilize multi-modal measurements of individual cells for more informative scientific discovery [4. 29, 21, 50). This emerging phenomenon raises key question: How can one efficiently integrate data from multi-modalities?
Contrastive togguage-image pre- vve-training (CLIP) was recently proposed in [60], which utilizes the self-supervised contrastive learning technique train vision-language model with unlabeled data and to obtain representations of data from multi- modalities. Mathematically. given observations from two modalities: continuous random vectors X E Rd and Y Rd the goal of multi-modal contrastive learning is to learn representation maps f E Hx : RRd + Rd and E Hy : Rdd - Rd that map data representations supported Rd, where d is the user-specified output dimension.
With a training set {(X,, {.))EN CLIP [60] minimizes the infoNCE contrastive loss:
o(f((x)..(O)
exp
min (xxr))xxxXyyXx
(N((Jg,TT)
log
N EJE[N] exp
a(f(X)))o(3))
N
exp (((((.))0(X))
log
N SE[N]
N- EjE( exp
((((X)))g(33)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input
Contrastive Loss
L
RNA
X-modailty
8(Y)
X-modality
Figure 1: Multi-modal contrastive learning applied to the bone marrow single-cell CITE-seq data.
where (:. is bivariate similarity measure, with larger values of (f(())) indicating higher similarity between embeddings f(X) and gY)). Here we take into account the optimization over temperature align with the practice of CLIP [60]. In Figure I, plot the pipeline of CLIP [60] using a bone marrow single-cell CITE-seq dataset [21] for illustration.
CLIP has achieved outstanding performance in zero-shot accuracy for downstream tasks and has inspired number of follow-up works across domains [17, 40, 81, 63]. The thcoretical analysis of CLIP was later initialized in [62], [49], and [9], where the properties of the learned representation and its effect on the downstream accuracy are of interest. However, the distribution of mulli-modal data is mainly restricted to factor models with shared latent variables, and the representation map commonly assumed to be linear, neither of which is close to the actual practice CLIP. In this paper, aim theoretically study the properties of multi-modal contrastive learning with data from general distributions and representation mappings in general function classes.
1.1 A motivating example: CLIP adapts to intrinsic dimension
We begin with describing several interesting phenomena arising from using CLIP on synthetic data. The phenomena motivate our later theoretic studies.
Consider the synthetic setting where X and Y are generated from the following distribution with k.' < min min d2): Y L.d.d
(d,,) & fid. I1 X;= (Ya, Yxx-, ETTT.
We set 2, d1 = dg = 20, and the output dimension of f(X) and g is set to be 3. We set the function class to be 5-layer ReLU neural networks with all middle-layer widths fixed at 50. We use a training set of size 12000 and a separate test set of size 2000. In the remaining part of this paper, adopt the inner product with population-level normalization i.e., c(f()),(())) = - the training set to estimate expected norms in each iteration. Figure 2, we plot the out-of sam-pe similarities between positive and negative pairs, the change of estimated intrinsic dimension along training (see Section for further details), and temperature 7 along training. We note the following phenomena from the results:
the similarity measure
(())) In experiments, we leave out fixed subset of size 2000 of
For positive pairs, (f(;)) g(Y)) tends to concentrate around one, while similarities be- tween negative pairs are capped by one;
2. Although the output dimension is 3, representations with intrinsic dimension 2 instead of are preferred by infoNCE loss;
3. The optimized temperature converges zero.
The observed phenomena, especially the intrinsic dimension selection and the convergence of tem- perature zero, motivate theoretical perspective in this work.
understand multi-modal contrastive learning via CLIP from the
In Appendix G.5, we show that it is comparable to cosine similarity in terms of both representation intrinsic dimensions and downstream task performances.

---

similarties:
simliartties: negative pairs
MLE-ID
famperature
60000
RX:
1.0
800
0.8
40000
800
0.8
2
400
0.4
20000
200
0.2
0
0
0.0
1.0
1.5
400
600
training epochs
training epochs
Figure 2: Histograms of out-of-sample similarities, change of intrinsic dimension, and convergence of temperature (linear setting: k" = 2, d
3).
1.2 Why existing understanding of multi-modal contrastive learning is insufficient?
Existing work explains the representations learned by contrastive learning from two different per- spectives. Here, we elaborate on why neither can fully explain the phenomena observed in Sec- tion 1.1.
Perspective from alignment and uniformity [79] decomposes the population infoNCE loss L into alignment and uniformity terms such that L(f. r) Lalign (flign(..g. (f.g. g. r) + Lunif g, 7). The first term Lalign favors alignment as it is minimized when f(X) g(Y) almost surely. In addition, the result in [79] implies that among all aligned representations, the second term Luiif favors unifor- mity as it is minimized when fX) is uniformly distributed over the entire output space, which is restricted be the unit hypersphere with a user-specified dimension [79]. A crucial limitation of this result is that one needs the correct specification of a true dimension of data. In other words, if the aligned representation has a smaller dimension compared the ambient output dimension (which arguably is always the case in practice) or the function class is not expressive enough to produce uniform representations on the entire output space, the theory established in [79] is not applicable.
Perspective from mutual information maximization Since the infoNCE loss be viewed as a variational upper bound of the negative mutual information [53, 59) (see also Lemma 8 below), another line of research understands the global minimizer of infoNCE loss as the mutual information maximizer [76, 24, 75, 73, 84, 41]. Although existing works connect infoNCE loss to its mutual information bound, is pointed out in [75], solely maximizing the mutual information may result in undesirable representations. For instance, in the example in Section 1.1, as long as f(X) ~ g(Y) almost surely, the mutual information between representations is infinite. Hence, this theory fails to explain why. in the motivating example, fXX and g are supported on a 2-dimensional manifold as opposed a curve or a number of standalone points, since the latter two can also maximize mutual information between representations at infinity. This indicates that infoNCE loss minimization is not simply mutual information maximization and requires more fine-grained analysis.
1.3 Preview of our results and contributions
We now provide preview of our main theoretical findings that allow us to explain the three phe- nomena observed in Section 1.1. A formal version will be presented in Section 3.
Theorem (Informal). Suppose there exist aligned representations that maximize the shared infor- mation between X and Y. Then, any "minimizer" (f.g.T) of the infoNCE loss satisfies
1. o(f(X) = constant almost surely, which caps similarities between negative pairs;
2. the intrinsic dimension of shared latent variables in the tured by (f()).g(Y)).
data exactly cap-
3. ==0 0+.
*Throughout this paper, "almost surely" refers to almost sure events with respect to the joint distribution of (X, Y). unless otherwise specified. Likewise, "measurable" refers to measurability with respect to the Lebesgue measure, unless stated otherwise.
3

---

In addition, we summarize our contributions in this paper as follows.
We investigate the theoretical properties of learned representations from multi-modal contrastive learning and take into account temperature optimization in theoretical analysis.
Based precise (variational) decomposition of the infoNCE loss in Lemma 8, theoretically show that temperature optimization enables multi-modal contrastive learning, which encourages multi-modal representations be sufficient and maximize the similarity measure, also adapt to the intrinsic dimension of data. To be more concrete, different from [79], our theory does not require the existence of aligned representations that are uniform over the entire output space, i.c., no prior knowledge about the true intrinsic dimension is required.
Moreover, the practical relevance of our theory is supported by empirical findings in real-world datasets, such as single-cell multi-omics and image-text datasets, which demonstrate that in many cases, there are a relatively small number of effective shared features that affect downstream tasks.
Organization of paper. The rest of this paper is organized as follows. The ideal properties of rep- resentations and the intrinsic dimension are formally defined Section Main theoretical results are shown in Section 3, which includes formal statement of our theorem in Section 3.2. Additional related works are summarized
Section Experimental results for both synthetic and CITE- seq single-cell datasets are presented in Section 4. Technical proofs, extensions (e.g., connection to suf- ficient dimension reduction in Appendix E. 1), and additional numerical experiments are deferred to the appendix.
2 Ideal representations and intrinsic dimension
In Section 2.1, define two key properties of the ideal representations (X) g (Y)) when learn- ing from the paired data (X, (1)-aligmment and maximal mutual information, and in Section 2.2, define the intrinsic dimension. Throughout the paper, define H Hx Hy.
2.1 Alignment and maximal mutual information
Alignment and similarity maximization. Inspired by the seminal work [79] in single-modality contrastive learning, we propose the following notion of alignment for learning representations from multiple modalities.
Definition 1. We define the set of representation maps that realize alignment and similarity maxi- mization as
A(H - ((100) -
,(Y) o(s(X),((Y) mm((fg) almost surely EX] Elg((Y)
S(X)
Here and after, mo mo(f.)) = css ILP off()),g()) for any (f.g) E H.
In our motivating example in Section 1.1, there exist representations f(X) (X (x,xx)// X2) X (Yi, /Y/ + YY g(Y) satisfying (f.g) A(H). In this case, o (F(X), =1 101=m (f.g) almost surely.
Maximal mutual information. A classical notion to measure statistical dependence is mutual information: [(X; Y) DKL (Px, Px @ Py). It is tempting ask for representations (f.g) such that the mutual information (((()):g(Y) is maximized. This is known as the infoMax princi- ple [75, 6]. However, we have argued in Section 1.2, this vanilla definition of mutual information not fine- -e-ggaainnd enough to compare aligned representations: whenever continuous representations f(X) = g{Y) almost surely, we have (X): 9YY)) too [33, 74].
To mitigate the deficiency, we adopt a fine-grained order for mutual information. For each integer M 2 1, we consider a discretized version (fM, gMr) of the representations (f.g)- As (fM, 9MM)
'See Section A.2 for precise definitions of the discretizations. Related results on approximating mutual information based on discretization and binning also be found in [32, 56, 12].

---

are supported on finitely many points, the mutual information 9x) (Y)) is always finite, albeit t- (X): (Y)) is possibly infinite.
With the discretization place, we can define the set of representations with maximal mutual infor- mation follows.
Definition 2 (Maximal mutual information). We define the following set of pairs (f.9) E H that sufficiently capture the dependence between X and Y:
W(H) ((.g) E H: inf (I(M((X;;M (M)) 2 0}, M4+0
where Ih (H) = suP(se))en ((
Roughly speaking, representations (f.g) have maximal mutual information if at every discretization level M, the mutual information [ff ( gx))) is comparable to the maximal discrete mutual information / (H) achievable by the function class. In our motivating example in Section 1.1, where the shared latent variables between and are two-dimensional, the learned representations maximize mutual information only if they capture all the latent features and have the intrinsic dimension of 2.
In the end, we define V(H) A(H) n W(H). Throughout the paper, we assume V(H) * 82, i.e., there exist aligned representations with maximal mutual information.
2.2 Intrinsic dimension
We now move on to define the intrinsic dimension of representation function f. To begin with, define the range of function f to be R(f) {f(x) x e Rd ]. and define the usual linear dimension as dim (R{(f) = (annspan(R(())) capture the nonlinearity in f, and the possible manifold structure in the range of f. With this in mind, we adopt the following notion of intrinsic dimension, which is closely related to the dimension of manifolds [34].
However, this vanilla dimension not able to
Definition 3 (Intrinsic dimension.). We define the intrinsic dimension of f E Hxx, denoted by ID(f). the smallest integer such that there exist measurable function h Rd Idd with dim(R(h) and an injective measurable function R(h) + 12d such that f(x) (ő (x) almost everywhere.
As an example, with a full-rank matrix A E Rxxdz and an injective map Ç R + Rd, the representation f(Z) ((AZ), which is usually known as the multi-index model [80, 38], has intrinsic dimension exactly k where we can choose h(2) = AZ and Φ =.
Proposition 1.
Suppose ((H) # Ø. Then all ideal representations (f.g) V(H) have the same intrinsic dimension k*, that is, for all (f.g) E V(H). we have ID(f) = ID(g) = k.
To further understand the definition of k* recall the example in Section 1.1. As the shared fea- ture between X and Y has dimension 2, any aligned representations supported on a curve cannot maximize the mutual information with certain function class. Also, any representations with in- trinsic dimension larger than 2 will have additional randomness conditioning on the 2-dimensional shared feature, and thus cannot align.
Global minimizers of InfoNCE and dimension adaptation
In this section, we analyze the global minimizers of the population infoNCE loss and prove that CLIP adapts the intrinsic dimension k of the ideal representations.
5

---

3.1 InfoNCE loss and its minimizers
Similar to prior work [79, 44], throughout the paper. we consider the population infoNCE loss
(((())((1))
a(f(X))0(11)
exp
exp
((f.g.T) -Bxy
log Ey exp (0(1())0(11))
Ex,
log
(2)
[(((())
E exp
where y _ Y, X, and (X, V) IL (X,Y). Based on [79] (Theorem 1) and [44] (The- orem 2.1), the population loss (2) is indeed the large-sample limit of the empirical loss (1) as lim (f.9.T) ((f.g77) = 0, for any fixed f, 9. and r > O.
Temperature optimization. We consider the regime where the temperature т 0 is also opti- mized in the training process, which aligns with the practice [60]. In previous theoretical works on CLIP, the temperature parameter was either treated fixed [79] or not taken into account [49, 22]. However, it has been revealed in both empirical and theoretical studies that different choices of T can lead to extremely different properties of the learned representations [79, 78, 18, 20). Thus, theo- retically understand properties of representations learned by CLIP in practice, we take temperature optimization into account and write 7 as an argument of the loss function.
Minimizers of the infoNCE loss. Challenges arise when defining the tuple (J,9.7) T) that mini- mizes the infoNCE loss in (2). Consider two representations 91) and (f2, 92). Both are aligned and continuous. It can be shown that lim,-so 1 C(fi, 91, 7) -o lim,-ol L(f2. 92. T). In other words, the infoNCE loss cannot differentiate among aligned continuous representations, due to the unboundedness of mutual information for aligned continuous representations.
To address this issue, we use the same discretization idea in Section 2.1. Precisely, for each slackness > O, we define the set of near minimizisss to be
Oc,,(H1) = (f.9) (7700(((1.00
H: H: 3r 2 e(n), lim sup (f(fM, 2T,, (7() - =2n M-++x
(3)
where s()) > 0 is nondecreasing" y with c(0) minimizers to be C.,00Cc,,,HH) A few remarks are in order.
Correspondingly, we define the set of
First, instead of looking at C(f.g. which could be -00, we compare its discretized version 9([M: T) against benchmark -27% (H). Both quantities finite.
Second, the lower bound E(n) the temperature measures the sensitivity of the solution (f with respect to the change of temperature т. Then, with any tolerance v) > O, the set Oz.,, (H) only contains representation maps that achieve small loss with temperature less than the threshold E()) and hence rules out the representations that can only be optimal in the singular case when the temperature is always zero, which is in line with the fact that the temperature can only decrease to zero at a certain rate in the actual training process.
3.2 CLIP automatically adapts to intrinsic dimensions
Recall the definition of k* in Proposition 1. We now present the main result, assuming that H includes all measurable functions of X and Y.
Theorem 2. Assume that V(H) * Ø. We have Onzo (H, + ø. In addition, for any (J.g)
O22o (H), we have
1. (similarity maximization) o(f(X).g()) - (J almost surely;
2. (intrinsic dimension adaptation) ID(J) ID(g) k*
3. (monotonicity in temperature) infoNCE loss C(f.g.7 T) is increasing in T;
4. (mutual information maximization) (f.g) W(H)).
'See Appendix C.2 for the exact definition of e(n).

---

top-0.05% accuracy
20
MLE-intrinsic dimension
histogram off(k)..(Y))
temperature
1.0
[X0
1.0
0.8
15
0.6
10
0.4
0.4
in-semple out-of-sample
0.0
0.0
10
20
30
20
30
0.0
1.0
1.5
200
400
600
output dimension
output dimension
training epoche
(a) Accuracy and intrinsic dimension with varying d. (b) Similarities and convergence of (d 20).
Figure 3: Results with synthetic data: linear setting.
Theorem 2 implies that (approximate) minimizers of the InfoNCE loss have an intrinsic dimension exactly equal k*. We note that the existence of aligned and uniformly distributed representations over the entire output space, as required in [79], provides sufficient condition for V(H) + Ø. In contrast, our result shows that even without requiring representations be aligned and uniform over the entire output space, CLIP can still adapt to the intrinsic dimension of multi-moda data. As corollary, in Appendix E.2, we further show that when the output dimension is correctly specified, the set of minimizers A,=0 (H) coincides with the set of aligned and uniform representations that maximize mutual information. We also connect our findings to sufficient dimension reduction in Appendix E.I.
We note that, in the concurrent work [52, 43], sufficiency (Corollary 1) is obtained when there exists pair of encoders that has infoNCE loss coinciding with the minimum over all similarity measures. In contrast, we consider the family of similarity measures taking the form SU V) = - = V) adopted in CLIP and take into account the optimization of temperature, which is relevant to the result in [53, 43] and [52], but also reveals and explains new phenomena in practice, such as the convergence of temperature and the adaptation to intrinsic dimension. More importantly, our paper puts more emphasis on the exact statistical properties of learned representations, such as sufficiency and low-dimensionality that are shown in Theorem 2, while [52, 43], through the sufficiency mea- sure, focus on the bounds for downstream accuracy and the learnability (via excess infoNCE loss) of architectures (e.g., Transformers) under certain structured models.
4 Numerical experiments
In this section, we further justify the theoretical findings with both synthetic and real-world datasets. Starting with synthetic dataset in Section 4.1, we further consider real datasets: CITE-sed seq dataset [68, 69] in Section 4.2, ImageNetV2 dataset [61] in Appendix 4.3, and YFCC dataset [72] in Appendix G.7. Throughout this section, fit the representation map with 5-layer ReLU neural network with middle-layer widths all fixed at 50 and the output space IRd, This function class is denoted by F where is the input dimension adjusted for each setting. We estimate the global intrinsic dimension of data using the MLE-based approach proposed in [35], which implemented in the skdim. id package."
4.1 Results with synthetic data
We start with two synthetic datasets with Ae < min dz}:
Linear setting: consider N i.i.d. draws from the same distribution shown Section 1.1.
Nonlinear setting: consider N i..dd draws from the following distribution: Y i.id (0, I, & i.id. NO0 [1, -+), and
2.
X₁ = (0.2Y;) sin(Y log ... 1,
'nttps://scikitt-ie otaataddddddeoooooorrraddddoooo http////aatt///sdii.ii. MLE, html.
7

---

top-0.05% accuracy
20
MLE-intrinsic dimension
histogram off()). 9tY,)
temperature
1.0
[X0
1.0
15
gtt)
0.6
10
0.4
0.4
in-semple sum-le-sample
0.0
0.0
10
20
30
20
30
0.0
1.0
1.5
200
400
600
output dimension
output dimension
training epoche
(a) Accuracy and intrinsic dimension with varying d.
(b) Similarities and convergence of (d 20).
Figure 4: Results with synthetic data: nonlinear setting.
Here set d1 d2 20 and k 5. The total N 14000 data points are partitioned into training set Dirain with 'Dtrainl = 10000, a test set Drest with [Drest| 2000, and a separate set with size 2000 for estimating the expected norm at each epoch. With representation maps f FNNN and " E Folid learned with Derain. we consider the downstream task as the top-a% matching of representations. More concretely, with D Dirsin or Drest, for any E D, denote No. (1; D) as all the indices such that (((X,)--(YY)) II the [a (2[Di1ssme -smallest among [If(X.) (((X.))g(YY))l)) ((%))) E D, and accordingly.
Acca (D) =
1 D
Σ IED
Then, we can define the in-sample and out-of-sample accuracy by Acc, Acca((rral (Dtrain) and Acca (Drest). ) respectively. Particularly, with synthetic data, choose a% 1/ Drestl = 0.05%, which refers to the top-1 matching accuracy for out-of out-of-sample matching.
Figures 3 and 4 report the results averaged respectively. We can see that in both settings, when the output dimension exceeds 5, both the in-sample and out-of-sample accuracy, and the MI.E-ased estimated intrinsic dimensions tend to saturate. Specifically, the estimated intrinsic dimensions approach the true value k. = 5, which validates that minimizing the multi-modal contrastive loss automatically adapts to the underlying intrinsic dimension of data when the d 2 k". In addition, in each setting, we fix d = 20 and present the histogram of similarities as well as the convergence of T in training in Figure 3b and Figure 4b, which validates the theoretical prediction that T converges to zero and that the similarity measure between positive pairs will concentrate constant that dominates the similarity measure between negative pairs.
50 repetitions in linear and nonlinear settings,
4.2 Results with CITE-seq dataset
The CITE-seq dataset contains simultaneous measurements of transcriptomes and cell-surface pro- teins from the same cell, and we get access to CITE-seq dataset via Seurat" [21] in R. We focus on the CITE-seq healthy human bone marrow cells (BMCs) data, consisting of 30672 measured scRNA-seq profiles from bone marrow [69], each with an additional panel of 25 antibodies. Fol- lowing the preprocessing procedure in Seurat, we obtain a two-modal dataset with 24-dimensional protein data and 200-dimensional RNA data. More details of data preprocessing are presented in Appendix and ablation studies with Transformer architecture are presented in Appendix G.6.
In each repetition the experiments, we randomly sample 20000 rows without replacement from the preprocessed dataset and randomly split the subset into training set Darain with |Dtrainl 10000, a test set D Drest with /Dtesel = 2000, and a separate set with size 8000 for estimating the expected norm at each epoch. Representation maps f.ğ E dd2.d are learned with D via contrastive loss and the quality of representations is evaluated on D terms of the classification accuracy two groups of cell types and the top-a% matching accuracy (a% 0.5%). We vary the output dimen- sion the neural network architecture from 1 to 29, and results averaged after 50 repetitions with varying d are presented in Figure 5, in which, both in-sample and out-of-sample accuracy tend to
"nttps://satijall org/seurat/.

---

top-0.5% accuracy
cell type accuracy
type accuracy
MLE-intrinsic dimension
15.0
1.0
In-sample ple out of sample
1.0
1.0
In-sample sample
[XX]
12.5
g(Y)
0.8
0.8
0.8
1.0
0.6
0.6
0.6
0.4
0.4
0.4
5.0
insemple
2.5
out-of-sampl
0.0
0.0
0.0
0.0
0
10
20
30
0
10
20
30
0
10
20
30
10
20
30
output dimension
outout dimension
output dimension
output dimension
Figure 5: Results with CITE-seq dataset.
saturate when the output dimension exceeds 10. In addition, in the right panel of Figure 5, the MLE- based estimated intrinsic dimension of in-sample representations does not exceed 10 regardless of the choice of output dimension. This demonstrates that the multi-modal contrastive loss effectively extracts intrinsically low-dimensional features from the CITE-seq dataset, which successfully cap- ture the information to differentiate two levels of cell types. Moreover, as is shown in Appendix G.4, with different choices of output dimension d. the temperature tends converge which offers evidence for the existence of nonempty (HH when H is a multi-layer ReLU network.
in training.
4.3 Results with ImageNetV2 dataset
We also consider the ImageNetV2 dataset? and focus on two modalities: images and text labels. Here we use the text "This is photo of a/an label" as the input of the text encoder for each image. In addition, each image in ImageNetV2 dataset can be classified by coarser classes with 67 levels [13], which will be adopted in downstream classification task. We first encoder (ViT-L14) and text encoder (a masked self-attention Transformer) to obtain preprocessed inputs: 1024- di2ensime image embeddings and 768-dimensional text embeddings, which are used as inputs to the 5-layer ReI.U neural networks.*
pretrained image
Similar to previous sections, with [Derain| 8000, (Drest| 1000, and a separate dataset with size 1000 to estimate the expected norms, we consider image classification and the top-a% matching (a% 0.5%) the downstream tasks. In addition, since images from the class share the same text input, to avoid the singularity when estimating the MLE-based intrinsic dimension, we add indepen- dent entrywise perturbations drawn from 00,00.01 to encoded text inputs before CLIP training for both Dirsin and Dtest. We can see that both accuracies tend to saturate when d is approaching 20 and the MLE-based estimation of the intrinsic dimension for the image embeddings upproximately 8 when d keeps increasing. Since the text embeddings have a cluster structure due to discrete labels, the MIEE-ased intrinsic dimensions are slightly smaller than those of image embeddings. Note that we use a 5-layer ReL.U network for CLIP training only for illustration purposes, and better results potentially be obtained with more sophisticated network architectures.
top-0.5% accuracy
classification accuracy
10 MLE-intrinsic dimension
1.0
1.0
in-sample out-of-sample
0.8
0.8
in sample out-of-sample
0.4
0.4
0.2
0.2
.... r(X)
9(Y)
20
40
20
40
20
40
output dimension
output dimension
output dimension
Figure 6: Results with ImageNetV2 dataset.
7 We load the datasct from https: / //tgttub..tttttte//g www.///odeettrraaaa//aagggeeeee More details of preprocessing deferred to Appendix G.

---

5
Related work
The integration of multi-modal data has been studied for decades in machine learning. The simplest practice is to concatenate features from all modalities, which is known as early fusion [5], but the direct concatenation can lead to redundancy and high dimensionality [48]. As a dimension reduc- tion technique, canonical correlation analysis (CCA) [23, 2, 26, 28, 25] is widely adopted to learn maximally correlated (linear) projections of modalities as the shared representation, which is further extended nonlinear projections by kernel CCA [1] and deep CCA [3]. With the initial purpose of integrating acoustic and visual speech signals [82, 57], muulti-moda learning has witnessed con- sistent progress in machine learning [7]. One line of research proposed fine-grained neural network architectures that can output joint representations of the multi- modal input [47, 51, 64]. Another line of research adopts probabilistic graphical models with latent variables to characterize multi-modal data, such as multi-modal deep belief networks [66, 30] and deep Boltzmann machines [67]. Differ- ent architectures for modalities are also explored in the field, where certain (constrained) measures of cross-modal similarities are proposed for representation learning [31, 83, 55].
Despite the wide applications of multi-modal learning, the theoretical understanding of its empirical success has drawn attention from the community only in recent years. Earlier works mainly interpret the advantage of modaiiiiosssssss from the ieformation-theoretiaall theoretical perspective [65, 14, 42). A recent work [27] compares the generalization error with different subsets of modalities theoretically, and the result is further discussed in [46].
Similar to our scope, more recent papers include [76], which focuses on the mutual information of multi-modal features the upper bound of the additive inverse of contrastive loss, and [44], which analyzes the solution of CLIP via the lens of independent component analysis (ICA) and depends on specific data generating process ensure the identifiability. In addition, concurrent papers [52, 43] introduce the notion of approximate sufficiency and show that this property is satisfied by learned representations by CLIP and also connect the sufficiency measure the accuracy of downstream tasks with a goal similar to [9].
6 Discussion
In this paper, we have characterized the statistical properties of learned representations in multi- modal contrastive learning, and specifically, have shown that the solution can adapt to the intrinsic dimension of data in the setting with V(H) * Ø. This property is also relevant to sufficient di- mension reduction as is demonstrated in Appendix E. The thcoretical result is also justified by both synthetic and real datasets, where the estimated intrinsic dimension as well as the downstream accuracy in various tasks tend to saturate regardless of the output dimension (once it exceeds the intrinsic dimension), which implies that the underlying information shared by modalities can indeed be captured by a low-dimensional structure. It is also worth noting that our theory suggests a two- stage fitting strategy when using CLIP. In the first stage, one selects large output dimension that the function class is expressive. In the second stage, one can use the intrinsic dimension discovered from the first stage and post-process the representation accelerate the inference speed.
lower dimension. This could potentially
Our findings also suggest several future directions. Although V(H) * relaxes the condition in [79], it would be of theoretical interest for future work to characterize the precise limit of optimized temperature in general settings beyond this regime, which may also guide the practical choice of т. In addition, it is interesting to study the finite-sample behavior of the infoNCE loss as well as its minimizers.
Acknowledgements
C.M. was partially supported by the National Science Foundation via grant DMS-2311 127 and DMS CAREER Award 2443867. Z.M. was partially supported by the National Institutes of Health via grant U01CA294514 and the National Science Foundation via grants DMS-2345215 and DMS- 2245575.
10

---

Broad impact
This paper provides the theoretical understanding of CLIP, especially its ability to adapt to the in- trinsic dimensions of datasets when the temperature parameter is optimized. The theoretical results are justified by both synthetic and real datasets. These findings will enhance the interpretability and guide the postprocessing of learned representations from multi-modal contrastive learning.
References
[1] Shotaro Akaho. A kernel method for canonical correlation analysis. arXiv preprint cs/0609071, 2006.
[2] Theodore Wilbur Anderson, Theodore Wilbur Anderson, Theodore Wilbur Anderson, and Theodore Wilbur Anderson. An introduction to multivariate stafistical analysis, volume 2. Wiley New York, 1958.
[3] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livesou. Deep canonical correlation analysis. In International conference on machine learning, pages 1247-1255. PMLR, 2013.
[4] Ricard Argelaguet, Damien Arnol, Danila Bredikhin, Yonatan Deloro, Britta Velten, John C Marioni, and Oliver Stegle. Mofa+: a statistical framework for comprehensive integration of multi-modal single-cell data. Genome biology, 21:1-17, 2020.
[5] Pradeep K Atrey, M Anwar Hossain, Abdulmotalcb El Saddik, and Mohan S Kankanhalli. Multimodal fusion for multimedia analysis: survey, Multimedia systems, 16:345-379, 2010.
[6] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by max- imizing mutual information across views. Advances in neural information processing systems, 32, 2019.
[7] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE transactions pattern analysis and machine intelligence, 41(2):423 443, 2018.
[8] YinFeng Chen, YuLing Jiao, Rui Qiu, and Zhou Yu. Deep nonlinear sufficient dimension reduction. The Annals of Statistics. 52(3): 2024.
[9] Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable repre- sentation learning and zero-shot transfer in clip. arXiv preprint arXiv:2310.00927, 2023.
[10] Dennis Cook. Graphics for regressions with a binary response. Journal of the American
Statistical Association, 91(45)):983-99) 9922, 1996.
[11] Dennis Cook and Bing Dimension reduction for conditional mean regression. The Annals of Statistics, 30(2):455-474, 2002.
[12] Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
[13] Noam Eshed. Novelty detection and analysis in convolutional neural networks. Master's thesis, Cornell University, 2020.
[14] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning bust representations via multi-view information bottleneck. arXiv preprinl arXiv:2002.07017, 2020.
[15] Kenji Fukumizu, Francis R Bach, and Michael Jordan. Dimensionality reduction for super- vised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research, 53Jan):73-99, 2004.
[16] Kenji Fukumizu, Francis R Bach, and Michaell Jordan. Kernel dimension reduction regres- sion. 2009.

---

[17] Federico A Galatolo, Mario GCA Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via clip-guided generative latent space search. arXiv preprint arXiv:2102.01645, 2021.
[18] Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learning. arXiv preprint arXiv:2304. (2210, 2023.
[19] Lawrence M Graves. The theory of functions of real variables. Courier Corporation, 2012.
[20] Yu Gui, Cong Ma, and Yiqiao Zhong. Unraveling projection heads in contrastive learning: Insights from expansion and shrinkage. arXiv preprint arXiv:2306. 03335, 2023.
[21] Yuhan Hao, Stephanie Hao, Erica Andersen-Nissen. William M Mauck, Shiwei Zheng, An- drew Butler, Maddie J Lee, Aaron J Wilk, Charlotte Darby, Michael Zager, et al. Integrated analysis of multimodal single-cell data. Cell, 184(13):3573-355) 2021.
[22] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Pro- cessing Systems, 34:5000-5011, 2021.
[23] Hotelling Harold. Relations between two sets of variables. Biometrika, 28(3)321-3377, 1936.
[24] R Devon Hjelm, Alex Fedorov, Samuel Mavoiie-arrchllon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1et..16080.077, 2018.
[25] Paul Horst. Generalized canonical correlations and their application to experimental data. Number 14. Journal of clinical psychology, 1961.
[26] Harold Hotelling, Relations between two sets of variates. In Breakthroughs in statistics: methodology and distribution, pages 162-190. Springer, 1992.
[27] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang, What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 109446 2021.
[28] Jon R Kettenring. Canonical analysis of several sets of variables. Biometrika, 58(3):433-451, 1971.
[29] Hani Jieun Kim, Yingxin Lin, Thomas A Geddes, Jean Yee Hwa Yang, and Pengyi Yang. Citefuse enables multi-modal analysis of cite-seq data, Bioinformatics, 36(14):4137-4143, 2020.
[30] Yelin Kim, Honglak Lee, and Emily Mower Provost. Deep learning for robust feature genera- tion audiovisual emotion recogmition. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 3687-3691. IEEE, 2013.
[31] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embed- dings with multimodal neural language models. arXiv preprint arXiv:1411.2553 2014.
[32] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review Statistical, Nonlinear, and Soft Matter Physics, 69(6))66133,, 2004.
[33] Simon Laughlin. A simple coding procedure enhances neuron's information capacity. Zeitschrift für Naturforschung c, 36(9-10):910 912, 1981.
[34] John M Lee and John M Lee. Smooth manifolds. Springer, 2003.
[35] Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension.
Advances in neural information processing systems, 17, 2004.
[36] Bing Li, Andreas Artemiou, and Lexin Li. Principal support vector machines for linear and nonlinear sufficient dimension reduction. 2011.
12

---

[37] Bing Li and Jun Song. Nonlinear sufficient dimension reduction for functional data. 2017.
[38] Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86(414):316-327, 1991.
[39] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 0355003557 2019.
[40] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjic Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:21 2021.
[41] Yazhe Li, Roman Pogodin, Danica Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems, 34: 15543-1555, 2021.
[42] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interactions: An information decomposition framework. Advances Neural In- formation Processing Systems, 36, 2024.
[43] Licong Lin and Song Mei. A statistical theory of contrastive learning via approximate sufficient statistics. arXiv preprint arXiv:2503. 17538, 2025.
[44] Yuhang Liu, Zhen Zhang, Dong Gong, Biwei Huang, Mingming Gong, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Revealing multimodal contrastive representation learning through latent partial causal models. arXiv preprint arXiv:2402.06223, 2024.
[45] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic vi- siolinguistic representations for vision-and- and-lanu tasks. Advances in neural information processing systems, 32, 2019.
[46] Zhou Lu. A theory of multimodal learning. Advances in Neural Information Processing Sys- lems, 36:57244-57255 2023.
[47] Youssef Mrouch, Etienne Marcheret, and Vaibhava Goel. Deep multimodal learning for audio- visual speech recognition. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 2130-2134. IEEE, 2015.
[48] Arsha Nagrani, Shan Yang, Anurag Amab, Aren Jansen, Cordelia Schmid, and Chen Sun. At- tention bottlenecks for multimodal fusion. Advances in neural information processing systems, 14:14200--22133 2021.
[49] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In Interna- sional Conference on Artificial Intelligence and Statistics, pages 4348-4380. PMLR, 2023.
[50] Sagnik Nandy and Zongming Ma. Multimodal data integration and cross-modal querying via orchestrated approximate message passing, arXiv preprint 2rXiv:2407.19030 2024.
[51] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Ng. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-I1) I), pages 689-696. 2011.
[52] Kazusato Oko, Licong Lin, Yuhang Cai, and Song Mei. A statistical theory of contrastive pre-training and multimodal generative ai. arXiv preprint arXiv:2501.04641 2025.
[53] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1rXiv:1807.03748, 2018
[54] William Fogg Osgood. Lehrbuch der funkionnenther,,,,,, volume 20. BG Teubner, 1907.
13

---

[55] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation bridge video and language. In Proceedings of the IEEE conference on com-- puler vision and pattern recognition, pages 4594 2016.
[56] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):1191-122 1253,, 2003.
[57] Sarah Partan and Peter Marler. Communication goes multimodal. Science, 283(5406):1272-
1273, 1999.
[58] Yury Polyanskiy and Yihong Wu. Lecture notes on information theory. Lecture Notes for ECE563 (UIUC) and, 6(2012-2016)7, 2014.
[59] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. International Conference on Machine Learning, pages 5171- 5180. PMLR, 2019.
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[61] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[62] Yunwei Ren and Yuanzhi Li. On the importance of contrastive loss in multimodal learning. arXiv preprint urXiv:2304.03711 2023.
[63] Peiyang Shi, Michael C Welle, Märten Björkman, and Danica Kragic. Towards understanding the modality gap in clip. In ICLR 2023 Workshop Multimodal Representation Learning: Perks and Pitfalls, 2023.
[64] Carina Silberer and Mirella Lapata. Learning grounded meaning representations with autoen- coders. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 721-732, 2014.
[65] Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. In COLT, number 114, pages 403-414, 2008
[66] Nitish Srivastava and Ruslan Salakhutdinov. Learning representations for multimodal data with deep belief nets. In International conference on machine learning workshop, volume 79, pages 978-1, 2012.
[67] Nitish Srivastava and Russ R Salakhutdinov. Multimodal learning with deep boltzmann ma- chines. Advances in neural information processing systems, 25, 2012.
[68] Marlon Stoeckius, Christoph Hafemeister, William Stephenson, Brian Houck- Pratip K Chattopadhyay. Harold Swerdlow, Rahul Satija, and Peter Smibert. Simultaneous epitope and transcriptome measurement single cells. Nature methods, 14(9):865-866 2017.
[69] Tim Stuart, Andrew Butler, Paul Hoffman, Christoph Hafemeister, Efthymia Papalexi, William M Mauck, Yuhan Hao, Marlon Stoeckius, Peter Smibert, and Rahul Satija. Com- prehensive integration of single-cell data. cell, 177(7):888-1102, 2019.
[70] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: 1908.0990 2019.
[71] Sarah Teichmann and Mirjana Efremova. Method of the year 2019: single-cell multimodal omics. Nat. Methods, 17(1):2020, 2020.
[72] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfee100m The new data in multimedia research. Com- munications of the ACM, 59(2):64-73, 2016.
14

---

[73] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Aision-ECCV 2020: 16th European Conference. Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XI 16, pages 776-794. Springer, 2020.
[74] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. Icee, 2015.
[75] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lu- cic. On mutual information maximization for representation learning. arXiv preprint 13625, 2019.
[76] Toshimitsu Uesaka, Taiji Suzuki, Yuhta Takida, Chich-Hsin Lai, Naoki Murata, and Yuki Mit- sufuji. Understanding multimodal contrastive learning through pointwise mutual information. arXiv preprint arXiv:2404. 19228, 2024.
[77] Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(7):3777-3820, 2014.
[78] Feng Wang and Huaping Liu. Understanding the behaviour contrastive loss. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2495-2504, 2021.
[79] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learn- ing, pages 9929-9939. PMLR, 2020.
[80] Yingeun Xia. A multiple-index model and dimension reduction. Journal of the American Statistical Association, 163(4 2008.
[81] Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(100)12113-121333 2023.
[82] Ben P Yuhas, Moise H Goldstein, and Terrence J Sejnowski. Integration of acoustic and visual speech signals using neural networks. IEEE Communications Magazine, 27(1 1):65-71, 1989.
[83] Dongqing Zhang and Wu-Jun Li. Large-scale supervised multimodal hashing with semantic correlation maximization. In Proceedings of the AAAI conference artificial intelligence, volume 28, 2014.
[84] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Bren- del. Contrastive learning inverts the data generating process, In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
15

---

A Preliminaries
We start with preliminaries, including notations and the discretization of Ba introduced in Sec- tion 2.1.
A.1
Notations
We start by summarizing important notations in Table 1.
Rd
d-dimensional Fudlidean space
Ba
{a R²: || S vΩ}
ID()
intrinsic dimension of f (Definition 3)
Cы = {cass: E [M]}
a disjoint partition of Bs discretization of Ba under CM
Ba.M fEMS E cars: i e [M]
fM. 9M
discretizations of (f.g) E H.
Has
{(MM 9M) (J,9) E H)
ma (f.9)
ess xuPxILF a (f(X), s(x)
A(H()
((f.)) H' Eif(()II
g(Y)
and mo(f.)) =
Elgff\i
T5r (H)
suP(s,9)M (X)ss(X), Iad (Y))
W(H")
{(f.g) E H': lim inf M (I - 15s 2 0}
V(H))
A(H') w(H"))
()()),(() Q
smoothed distributions defined (6)
())))))) = Q.M
QM QM
discretized distributions defined with (fM 9as)
E(1)
temperature threshold with tolerance 7 defined in (9)
(c. (7)
Set of minimizers up to tolerance with temperature threshold
(c,((H) (3)
Set minimizers up to tolerance with temperature threshold E(1)
Table Table of notations
A.2 Discretization of Ba
With the defined similarity measure a(s))) +), analyze the mutual information between f(X) and g(Y), it is equivalent to considering mutual information between f(X) = ((X)/B|((X)) and = 1, [[x)) ( Iī(Y)) < vn < co almost surely. Then, denote the range of normalized representations by Ba. which is compact.
ğ(Y) = / which satisfies [(X]I = EY)
For any fineness M M N, consider discretization C'r {CM,heejM of Ba such that the compact set B& is divided into A connected portions, and representatives from each portion CM2 form the set Ba.M { We also denote Cm the set of all discretizations CM of Ba such that
max diam(cM.)) + 0 <E[M]
M + +00.
Particularly, we consider the discretization Cas E Cas such that each portion CM. has the same Lebesgue measure AM, which satisfies limM-+ AM 0. In addition, assume the sequence of discretizations {CAr MEM is nested such that CA C Cas for any M' < M.
In the rest of the proof, we write M +o0 to denote M 3 M to0. Accordingly, for any function f. we define
fM (z) ZM.
if E (
16

---

and the function class Has = {(fa.9M)) : (f.9) E Hy, which is also nested in the sense that HM S Hi for all M I M. We also notice that HM can be viewed as the class of piecewise constant functions. For any S S d, the set H is defined similarly.
Abusing notations, for any random vector supported on Ba, we write UM - U(B..M) to denote the discretization of U, for which
P(U =M,) P(U E ..Ma) and P(Uar & Ba.M) = O.
If U is supported on Ba,M, we have Uas = almost surely and, particularly, [M(() (S(X)) almost surely.
A.2.1 A fine-grained order
For any pairs of random vectors U = 1U,, UU),Ü (ū,, .,)))) E (Ba)' and any functional G' of random vectors, we define the order 3M as follows.
G(U) 3M G(Ö) + lim sup
{G(UM) G(Nx)) so. So.
M-+0
and moreover, the inequality is strict, i.e. G(U) Km G(T), if and only
{G(U>)) - G(ÜM)) } < <0.
lim sup
M-+
We also write G(U) -M G(U) if limas
((UUn) -
exists and is zero.
A.2.2 Approximation by discretization
In this section, we study the properties of the aforementioned discretization. To start with, for discrete random vector U with the probability mass function pu, we define the entropy of U by HUU) -Bllog Pu (U)). For continuous random vector U with probability density function pu, we define the differential entropy U by HU)) = (U)]).
Approximation of entropy and relative entropy. To start with, consider the random vector U E Ba with probability density function and the discretized random variable UM E Ba such that p: M >M.) P(U E cM,s). As Sesr, p(u)du z p(ZM,))AM, we have the following result that connects the Shannon entropy of UM and the differential entropy of U.
Lemma 1. [12, Theorem 8.3.1] As M +o0 and diam (Car,) -> holds that
lim M
(H(UM) + log Am) (U))).
See proof in Section F.
In addition, for two random vectors U, V supported Bd, recall [77, Theorem 21] in the fol- lowing lemma.
Lemma 2. (77, Theorem 21] For any Cm E Cas with M + to0 and diam + 0, holds that for any M,
DKL (UM Vas) & DKL. (UM+41 VM+1) & DKL. (U || V),
and moreover,
lim +00
DKL (UM VM) DKL (U V).
Approximation of conditional entropy.
For later use, we also consider the approximation of the
following quantity
( (0),
where U, V are random vectors on Ba with joint density p(u, v) and Φ Bax x Ba + R is continuous. We have the following result.
17

---

Lemma 3. Assume Φ BB x Ba - R is continuous. For any Cm E CM with M - to0 and
diam(Car,) + O, if (OU, V) U) > 0. it holds that
lim inf H( VM) (UM) > 0. M +
See proof in Section F.
Approximation of (conditional) mutual information. Consider random vectors U, on Ba B, x Ba + R. We are interested in the approximation of
and continuous maps 6, v T[ (U, V)) and T( V). U) by its discretized version. To start with, we have the following result.
Lemma 4. Assume o and vp are continuous and (o V) (), < 1. For any CM E CM with M - 400 and diam (cas,) - O, holds that
lim 400000
[(oUUs, VMJ); (UM, Var)) I( V):0UU,V
See proof in Section
For the conditional mutual information, note that T( V); (U, V) U) T()(U, V); - Lemma 4 can be applied to both terms, which indicates that
lim T(S(UM, VM (4):://( (UU, Vas) (Us) I (U (U.
M -++0
Related results on estimating mutual information based on discretization and binning can found in [32, 56, 12].
Approximation of infoNCE loss. For any pair of representation maps (J,g) H. have the following result.
Lemma 5. For any (f.g) E H with
=|)()\ = and any > 0, it holds that
lim M #000
C(far.9 9M.. g(f.g.T).
See proof in Section F.
A.3
Intrinsic dimension and its properties
We revisit the definition of intrinsic dimension in Definition 3 for which we have the following property for ID(f).
Lemma Suppose f Rd: + Rdd satisfies ID()) = k. Then, for any measurable function Rd + Rd, holds that ID o f) < k.
A.3.1 Proof of Lemma 6
Suppose ID(f) k. Then, by Definition 3, there exist measurable function h R4 - Ed with
dim( ((()) k: and an injective function Φ Rdd + IRd such that f = h.
For any measurable function v) Rd + Rd, if is injective, then & = (+) o Φ) o h. Since el o o injective, definition, have ID(
k.
& not injective, for any x E RRd:, define
T(x) = {a': (w a oo h)(x') = (1) do h)()))
and accordingly, we define the equivalence relation ~y: x' if and only if T(x) T(x'), with which (x:) {a' x' r). Then, denote a/> the representative of [T.)). Based the notations above, define
h: + h(xl)).
18

---

We can verify that for any Rd,, it holds that
(vb o oo (z) (vb do h))x (v o (x).
In addition, for any 21, 22 E R( o h). it holds that (>1)) * (22)). Otherwise, suppose there exist #2 R(Φ o h). i.e., there exist , 412 E Rd satisfying -, (ib o (ar)) with i E (1,2), such that (0(11) (22). Then, by the definition of the equivalence relation and the definition of h. we obtain 1 thus I = 22 and
21 (Ø o (x1) = (6 (x))) 22:
which draws the contradiction. Hence, v n(dch)) i.e. & restricted on the range of Φ h is injective. Then, (1500) R(())) is also injective. Consequently, we have constructed an injective map (1ho@) R(T)
and measurable function h with dim(R(h) I dim(((h)) = such that (z) = 1( o の) (x) almost everywhere, which, by Definition 3, implies that ID( f) S ID(f) k.
B Properties of (HH)
In this section, we focus on the properties of the set VHH). To begin with, for any subset H' H'x x Hly S H. we generalize the definitions of A(H) and W(H) as follows.
Definition 4. We define the set of representation maps that realize alignment and similarity maxi- mization as
A(H') = (1:9) (F:0) :
f(X)
g(
almost surely and m. (f.g) = =1].
(X)((X] EY)
=
Here m. (f(fgg) == ess sup suP of of(X)) g{)) for any (f.g) E H.
Definition $ (Maximal mutual information). We define the following set of pairs (f.g) H that sufficiently capture the dependence between X and Y:
= {(f.g) ((f')=((fs)) E H': lim inf (VV)(()))v((V))))/)))))) 2 0}, M-+t0
where Th (H) = Sup(s.))EH I (XM(X):gM(Y))
Note that the definition of W(H') for any H' C H. the benchmark mutual information / (HH) is defined for the entire class H. Accordingly, we define V(H') = A(H') n W(H').
Then, we are ready to present the following lemma on properties of V(H)).
Lemma 7. Assume V(H) f Ø. There exists ks* E [d] such that for any (J*,g") E V(H),
(1) Intrinsic dimension adaptation. ID(F*) ID(g") = k Sufficiency. kf < d, for any (f.9) H with mIx( k", it holds that
S(X) 1L sY))
See proof in Section B. and Section B.2, respectively. We note that Proposition is directly implied by Lemma 7 (1).
B.1 Proof of Lemma 7 (1)
To start with, for any integer & E [d]. define H² {(f.g) H: ID(f) ID(g) = kb} and (T(+)) is defined accordingly by
V(H*) {(f.) e V(H): ID(,) ID(g) k}.
19

---

Then, we define
k = max ( V(H) # Ø).
By definitions of V(H) and )(H(",, we have V(H)) S V(H), and we will show that V(H) S V(H()..
Case 1. 7 = 1, recall the definition of = max(k E [d] : V(H*) * Ø3. For any (f.9) e V(H),
suppose ID(f) = h >> 1, it holds that (f.g) E H², thus k 2 k > 1. which draws the contradiction. Hence, for any (f.g) V(H), we have(f.g) H i.e. v(H(*) V(H).
Case 2. Consider the case with k > 1. Suppose there exist (f*,,"") E v(H*) and (f.g) E V(H*)
but & < k d. Since both mutual information and infoNCE loss are scale-invariant with respect to both f and g. without loss of generality, we assume [X] = Elgg(Y) = (x)] =
Ellg" (X)]l 1. Then, since max ID( ID())) = k < k S d. by Definition 3, there exist aligned F(X),G(Y) E Rf and injective measurable functions o, +b R² + Rd such that
f(X) o (F(X))
g(Y) &p (G(()).
Here we define HU) as the entropy of random vector U and UM id (U), where id is the identical map on Rd. Then, there exists v Rd x Rd + R such that
0< (f f(X)) f(X)).
Otherwise, there exists a measurable function ( such that f*(X) = c((()) almost surely. By Definition 3, it holds that ID(S) ID(S*) = =k < k. which draws the contradiction.
Then, as (f)) f(X)) we also have
-(())))))x(((())))))) Hf())),, by Lemma 3,
M (X)()( ((X))
Otherwise, for any V. it holds that f(X)) is a deterministic function of f(X). Specifi- cally, (X) is function of f(X) and by Definition 3, we obtain ie = ID(f*) - ID(f) = k. which draws the contradiction.
Hence, we can define
F(X)
G(Y) 02 ((O())
F(X)
Oa-K-I
and G(Y) =
(((()),
with which
-(F(())((()) =M (F((X))
-M - ((F( - (((())))))((())) F(X))
<M (F(()X))
FM -I(F());G(Y))
Since @ and v are injective, we further have
-T(F()):G(Y)) KM -I(F(X); =M (Y())))
which contradicts the fact that (f.g) E W(H), which completes the proof. Hence, the remaining part of the proof, we can define b-- = k.
B.2 Proof of Lemma 7 (2)
Suppose the statement is not true. Then, there exists $,, $2 Rd x Rd - R such that
0 < (1
(x))),
20

---

which, by Lemma 4, further indicates that
0 M (xx) (Y)) (x))).
(4)
To see this, if the statement not true, it holds that for any 1 क2 TR TR.
(f (XX)) IL ( (X)..
Specifically, we have f(X) IL 9(Y) (X), which draws the contradiction.
Since k* < d, there exist injective measurable functions o², 10" R + Rd and (Y))
R such that
(*XX) = (F)(
9'(8) = (G)
Then, we can define
F*(X)
GO(Y)
F(X)
Oa- (((((x),
and
G(Y)
Oa-x-
with which, we have
-I(F(X); IM (X); 3M (x
Here =M holds the first inequality if and only if ((Y) 1 (x((), ( (2() (V)) 1L ( (x)(x)
which draws the contradiction to the construction of Φ and T in (4). Since ø and a are injective, we further obtain
-I(F(X);G(Y)) KM (X(""();g which draws the contradiction to the fact that (f",g") E W(H). Thus, we obtain f(X) TL g(Y) (X)).
C Properties of infoNCE loss
Before presenting the technical proof of the main results on intrinsic dimension adaptation, we summarize some properties of the infoNCE loss.
C.1 decomposition of the infoNCE loss
A pillar of our proof is the following decomposition of the infoNCE loss that characterizes the gap between the infoNCE loss and mutual information. Similar decompositions have appeared in [53, 79, 52). Lemma 8. Let (f(X). ~ P)x)) ((()) < +o0, there exist joint distributions Q(()))(Y) and ((()).(Y)) such that (() can be decomposed as
-)(((():)(M)) + DKL (P)().9()) lI |1|//11001 (((x) + DRL. (Pr(O),cr)) |l ((()) (5)
The formal definitions of ((()) and Q((x)),())) presented in Appendix C.1.1. In short, they can be viewed as the kernel smoothing of ()))),(()) in the product space.
Conceptually, given the decomposition (5), minimizing the infoNCE loss is equivalent to simul- tancously maximizing the mutual information between f(X) and g(Y), and minimizing the KL- divergence between P)(x),(Y) and its smoothed versions. Here, the latter encourages the joint distribution of P)(x).g(Y) to spread across the support and implicitly enforces uniformity of the distribution P)(x)),(Y)+ which is in line with [79]. In our actual proof, the lemma will be used together with discretizations introduced in Section 2.1 and (3) that ensure the finiteness of mutual information.
This decomposition also relevant to the analysis in [53, 52], where the minimum of infoNCE loss over all possible similarity measures is shown to be bounded from below by -2T([(X) ((([(X)g(()) gY)). However, our paper, we focus on the inner product-based similarity that is in line with the imple- mentation of CLIP and does not rely any unknown information about the distributions of (X, (X,Y).
21

---

C.1.1 Proof of Lemma 8
Recall the definition of mutual information
[(x;) = DKL (PxY |1 Px @ Py),
where Px, is the joint distribution of (X,Y) and Px, Py are marginal distributions. For any map and g, we have the
T(X;Y) (((())
When of and one-to-one with measurable inverse maps, it holds that I(X I(f()):g()))
To characterize the dependence structure between S(X) and g the conditional distribution P)(X)1(() often of interest, but the estimation is usually not tractable practice. As an alternative, consider hypothesized conditional distribution Q((x))o(Y), with density function (((x))((Y)) with which we have
log (U
I(f()):(()) Ep
(U)
PS(X))(YY) (U (U (U [V)
Ep
log
(F(X))a()) (U (V) EBpar DKL (Pr(x))t)) ll (((0))))))))
Ep
log (U)
Similarly, if have hypothesized conditional distribution [)07)1(()) with density function (g()))((X).r we can symmetrize the foregoing argument to obtain
(U (U))((U
(U (VIV)
I(f(X):(()) - Ep
log
[DKL (PH(H) ll (
2
To (U)
+ Ep
log
+ Eprox
DKL (PorO)e 11 [amm)oo)))
2
PV)
2
We consider the following families of conditional distributions:
(u,o)/
H (1(X))(())) (F(x)())
Ep Epayy V//
Hylf (0(0)1(X))- u) (v) EPnx) colU,))
where (U, V) is a similarity measure between random vectors U and V. Then, in this case, for any (F(x)s((Y))- v) E Hig and (g(0)1((x).-(1 E Hg f. one can verify that
log (Fx0))( (U' V) (U)
o(f(X).g())
Ep
EEp (o((of()))) - Ex Ep
exp
DOYHAI)) (V \U)
o(f(().g(P))
Ep
log
Ep (o(f((). - Ey logEx
exp
(V)
In addition, if we define the joint distributions
Q1(0)ar 1((0))(()) @ Pa(t) and e = =ooYY)1e(x) @ PS()). (6)
22

---

we obtain the variational form of mutual information as follows:
T((()) - -
+ DKL. (Pr()) ((x)))()))) 1 DKi (Prexhecr) ll (xx))))))
1
2
2 ((f.g.T) +.. ((x))))))))))))))))))))))
Hence, minimizing L) equivalent to the following optimization problem:
- 1(F(X);(()) + A(P: (UPP:0000,,0)1, (.A.:000xxaaa.=--xxxxxx (110)
min f.g.T
subject
(0010))) E A dÂ (())))()) Hols-
C.2
Properties of (approximate) minimizers
Since temperature also optimized, to analyze the solution path with varying T, for any 0, we consider the set
(1)) (1.9) F 2 such that lim (Clfas 9as, T) (10) - =2n (7)
M-+0
Note that we adopt the natural order I on since with discretization, it always holds that 15 (H.) +00. Recall that
Oc.,( = - (f.9) H: 3 2 =(n) such that lim (C(fM,9M, 27,, - 2n
M-++0
where we define E()) as follows. Denote the set of nondecreasing univariate functions by and in addition, the subset
C" --(w(EE: n (ewwt)), Ø, lim w(1) 7-04
(8)
n20
The set C* is shown to be nonempty when V(H) + Ø in Lemma 9. We define w (-) I w2(+) if for
any 1) 2 0, it holds that w (n) S w2(n). Then, choose s(-) to be any item in C* such that
(w E C² =(-) 3 w()) = Ø.
Note that if there exists global minimizer (f. that minimizes C for any 2 0, we can simply define
<nn sup = (m)) # }
To start with, we have the following lemma.
Lemma 9, Assume V(H) + Ø. Then, holds that
n 920
(H,, # Ø0
In addition, lim inf E(n) = 0.
See proof in Section C.2.1.
Then, we are ready to present the main result on the properties of O,z0 (H). For a general similarity measure a oU, V) and any (f.g) On20Oc.n (H), we have the following results.
23

---

Lemma 10. Assume (f,9) E (H, and m (f. <
co. Then, the following
properties hold.
(1) Maximal mutual information. (f.g) + W(H).
(2) Maximal similarity. of((),, (())) = ma(f.g) almost surely.
(3) Monotonicity in ((P; (x ( is increasing in
C.2.1
Proof of Lemma
Nonemptiness. To see the nonemptiness, it suffices to show that for any (f",9") E V(H),
lim --0++
lim sup (C(fMr-9M, (T))
M-++0
}.
= lim
sup (H)) (I( + 22(P: + S. M-+000
--+0+
Without loss of generality, assume (X)] = Pllg'(()) 1. By definition, since gf" V(H), we have
lim inf M-+00
(X):M (Y)) - (H)) = lim sup (
M M-++00
which implies that, for any 2 0,
lim sup (C(fm: ghr. T) + + S 2 lim sup A(P: ((P:: (tt.sc/eeeeiii...sssssaaaaaa.aa (1 M-++00
M-0000
As 1f" E A(H), i.e. (X) = (Y) almost surely, we further have (1) = (g-(x)())
and
(x) = [P)
which is also true after discretization, i.c.. (X) = (Y) almost surely and (x) (w, = (1) (u)a In addition, for any M M,
DKL (P 11 (1)
/| PSu PIz ( 1087, v)dude RdXRd
J1 (7 v) log
( (x)
JEM Psa (x) (i)e dudu
((M( [. (1)
( Pixxx (ū) (w.0)/ dū du SE (T) Px) (w)e2/7
Jac (u)( log
du.
Then, by Lemma have
lim (PL(PP2,xX)ot1 ll (1) (KL(P/-XX)991)) ll (1=0)))00)))) M++00
for which, it holds that
--0+ [p log Sau (0
lim DKI(Pr (x0.9 (1 (1(0)) -+
lim
(1-(x)())e11 du
Pr( log log P)-(x)(u) du O. ()-(x)(1)
24

---

Hence, by the symmetry of (K (x in f and 9, we obtain
lim 7-90+
sum (x) U 0ra
0.
M-++00
Then, the constant function w(n) = 0 is in the set C defined in (8), indicating that C* + Ø, thus 2..0 (H) Ø.
Zero limit infimum.
Suppose lim inf, E(1) I Then, for any (f.g) E H. it holds that
lim sup (C(JM, 9M, I + 2T (H)) S 0.
M-+00
In addition, have shown in Lemma 8, it holds that
L(SM, SMMM99M:7> T) + 27 (H) 2 -XI[[(((X); (22 2 0,
which further implies that
lim inf (C(fM + (H)) I lim sup (C(MM.MM: + (H)) S 0. M-++0
M *-+
Hence, lim (CfM::9 T) + 21 (H)) = O. Moreover,
L + 2T 2 A (1 ( 0.
further obtain
lim to
(1 0.
In this case, the joint distribution of (f(X). takes the form
exp a(u.v)
(FCx)(U)
(X) u1)
Eraxy exp a(f(().g(M))
thus, the mutual information satisfies
/1
(rX v)levloogg log P((X) (u, v)
1(f());((Y))
PS(x) (u)Po(r)()) dudo
$ || (u, exp
I
dude
((f(X))0(0))
E
PPxx exp
I
(o(f(X).
- log p exp
I
Ω
<0.
I
Hence, lim sumsuu (X)x (Y)) < 00 for any (f.g) E H. Then, we obtain
sup sup sup (X): s Ω M-++0 (F-g)H
lim 15r (55()) lim
<0.
(10)
M-++00
In addition, since V(H) * Ø, there exist (f*,g") gg") such that ("X))/ 9" ( (Y)
almost surely, thus
sup 15 (H) 2 lim sup (X) = +00,
M-++0
M+++2
which draws the contradiction to (10).
25

---

C.2.2 Proof of Lemma 10 (1) and (2)
Assume (f,g) E C.,0 (H). There exists a decreasing sequence {7, Lng such that 7 - O as j + +oc. Accordingly, there is an associated sequence Tj s(7)), which satisfies lim inf t- Tj 0 as is shown in Lemma 9. By the decomposition in Lemma 8 and the def- inition of (2,n(H)), for any j E N, it holds that
sup M-++00
1h (H) 1[f (X) IM (Y)) AIP
(im()) 7/.
Since lim +-00 Tj Without loss of generality, we assume {TybeeN is convergent such that lim =0.
O, there is a convergent subsequence (Ty heN with limy-400 Tye O.
In addition, we have, for any M M and E N.
(H)) - (( 2 O,
A(P; (XP:Q) (aaaa
(ta O.
which implics that
max
sup (Tār(H)) - ((f(((()) lim lim M++x
M-+0
S limsup (() - T(JM( + A(P:: ( s nn- M++00
Then, by Moore-Osgood theorem [54, 19], we can exchange the order of limit operators and obtain
{
lim SuPM-+00
APP; (X (1) = lim i- ([)M(X);)MMY)) -1,
Denote m my mo (f.g) esssup off(X), g()gg(()))))))))))())) (Y)), and &m = ((u,v) E R(f) x R(g) a(u,v) m). (T) {u E R(g): olu, v) m), (²) = fu E R(f): a(u, m). As 0, ensure lim sup ~ A(PM, Q. (M) = 0, we have
1 1o(u,))=m
(4, = PΛ( (w)
(11)
E By(ry (V:o(uu))=))
1 (u,))==
(1) (v)
E Bx(x)|1 (W:o(t))))))))))
which indicates that
Ps(x) // e..
= Am-
Here 1 the indicator function with respect the set A and if A {a}, we define [1x]][a] Pa(x)(a). In this case, a(f(X), almost surely, which completes the proof.
C.2.3 Proof of Lemma 10 (3)
Recall the definition
A A P.
1 - DKi. (P)(), ll ((x))))))))) + DKI. (Pri))ssC) @ ((x)))00)))
suffices to show the monotonicity of the first term in r. Since infoNCE loss is scale-invariant with respect to both ff and without loss of generality,
assume [X] gY)) = 1.
When the distribution Pr(x)a(Y degenerates on the support (a(f(X).g(Y)) m²(f.g)). P)))) is not absolutely continuous with respect to Q)()))() and 1((x)) thus
26

---

A(P M... = +oo for any J > 0 and A(P: ((X:0r()) 1(M)))))))) O if and only if
the distribution P)(X).(() is not degenerate, by the joint distribution in (11), we further have
DKL (Px()),(()) ((()))(()))
= // P)(X), v) log ((( dudv
(Xx)),00) (,,
Bd xB*
/f P((X),(()),u)) log PX( (u, dude
R XB
|| J Pf(X) (() log
en//T dudu.
Jau (or dude.
Then, since the first term free T, we have
0
DEL. (PS(X),()) ll (1x))))))
0r
0
{
-1J -J (1( log
e//
Ja Po(r) (g (w) dudu
0r
0
{ + II PrCX),(M)) +
log / Po(r) (7) ( dō
dudu
0r
1 ( o olu, Pgr) (ū) (u)e
dude
Ja Po(Y) (u.)dd)))
Since ou, S m, the derivative is nonnegative, thus
i Or
DKL (PF()),(() (1
The equality holds if and only (L product distribution PΛ(x) @ Pa(r) In this case, for any T 0, the infoNCE loss L(f.g.T) T) = 0, which contradicts the fact that (frg) E En200O.,,71) thus, we have
m mo(f.g) almost surely with respect to the
DEL (PF()) ((x)) > 0. 0r
By the symmetry of A(P: ((P:Q ) in f and we conclude that (PP: Q100) (/1 increasing in r.
D Proof of Theorem 2
We present the proof of Theorem in this section. If (H)) # Ø, we have V(H) V(H(**) as result of Lemma and Oyz00 (H) # Ø as a result of Lemma 9, respectively.
In addition, Theorem 2 (1)(3)(4) directly implied by Lemma 10, thus it suffices prove Theo- rem 2 (2).
Based on the previous notations, we will prove the result with the following steps.
Step 1: for any (f.g) E E.. it holds that min 2 k";
.
Step 2: for any (f.g) E Onzo0 (H), (ax(())))) < k*
Combining the two steps, we conclude that for any (f,g) (.200c.,(HH) holds that ID( ID(g) = k".
27

---

D.1 Proof of Step 1
ID(() = k I d, there exists a random vector Z B& such that f(X) x(Z) almost surely for some injective measurable function TR Rdd In addition, have ID(g) < d. To see this, suppose [g) = d. Consider (u, v) R(f) x R(g) such that (u, v) m (f. (f. (...))ll(.. [ and there exists an open set B, satisfying v B, S R(g). Then, consider the linear function <(z) = (u, z). which is continuous in & E B, There exists ì E B. C R(g)) such that (())
(u, > ((v) ma (f. (Blg( which draws the contradiction.
Then, there exist F G(Y) E R* and injective measurable functions Φ, v R IR such that f(x) o (F(X))
s(Y) 3 (G(Y)).
Accordingly, for any (f",g") E V(H) = V(H(") there exist 11, D2 Rd x Rd + such that < (f(() f(x)): (2(g(()) f(X)).
To see this, otherwise, for any 1 42 R F + R. it holds that
(
(X)) f(x)) 0,
which further indicates that
T (f()) TT (, f(X).
Thus, specifically, ((X) 1L gY f(X), which holds if and only if there exists a measurable
function h such that (X) (ho (X) almost surely, indicating that ID(f*) s ID(f) = k k", which draws the contradiction. By Lemma 4, we further have
0 ~M (()1 (
Hence, we can define
F(X)
G(Y)
F(X)=
Oa-k-1 Φ (J(),,
and ū(Y)
0 0x- 0
By the Data Processing Inequality [58, Theorem 2.17], we have
-I((()); IM -(((())((())
3M (IF(()())))())
In the first inequality, =M holds if and only if G(Y) 1L (1(()) i.
L1(f(X), f*x)) 1L 42(g(Y), 9 (())) f(X),
drawing the contradiction the construction of 4 thus the first inequality is strict, i.e.
- -I(F(X);E()) ~M - (((()))))))
Since & and &e are injective, we further have - OI))))))() Im -I()()))(()) =M -I(f(X); which draws the contradiction to the fact that (f.9) W(H)), thus ID(f) k.
D.2 Proof of Step 2
As we already show that ID(f), Ig( 2 ke, let ID(f) = b lf = d, we naturally have k k., thus we consider the case where k* d and there exist injective measurable functions
6², 66 : Rd and F* F*XX), (Y) E R such that
('(X) 6" (X))
g¹(Y) = 36 (Y))
By Lemma 7, for any (f',9)) E V(H), it holds that ID(J*), ID(g") S k and
f(x) IL g( (X).
(12)
28

---

Hence, we can define
fx( = -If(X) (X))
Then, for any >> O, consider the infoNCE loss
(I((X))9(()) Exx C(f.g.T) (EI
(J(X), (.)))))))))
log
exp
(
logEx Ex
(F(X).9(Y)
+ Eg
exp
In addition, with the defined representation maps (fa.g), we further have
((+.ggTT) - 2 E (X),,,), ((
exp (X (1) )
(X) g())
+ +Ex
1o
log By
E (Y))
)
+ By log Ex exp (I(().
(X)IBBB
where, by Jensen's inequality, [|S((X] = [B]((X) (*XIIII S EIIS())II I [X)) almost surely.
We note that, according to Lemma 10, holds that
(f(X),9(Y)) 9) ()) almost surely.
In addition, by the conditional independence (12). further obtain
E ( (E[ (*()), (EK(((()) I - (
almost surely and, in the meantime,
(F(X),9(V))
Ex
log Ey
exp
=EIf((X) (E))(())
(f(X), 9(\))
|rool}
=Ey.(x)
E
log s(V)
exp
(
Since log Ey exp(x (U) is convex in æ, by Jensen's 's inequality, we obtain
(f(x),((1))
Ex
log By
exp
[B]]S(X) (E)lg())
(EL((X) I (EL((X) (X))),
> E Ex((x)
log OCF) exp
(B
(E) (()))) }
(x((),,(())
Ex
log Ryy
exp
In addition, since exp(x U) is convex in x, we also obtain
((( [())((() )
(E[J(X) (x(X) ((gg))(()))) +E]S(() (Y
Ex exp
Exx(x) exp
Combining the pieces above, if we define
EX)(
= E
ST.
[[X]
29

---

it holds that
C(S. (,T) S (f),
where the equality holds if and only if f(X) can be expressed as a measurable function of (X) almost surely. Thus, if f(X) cannot be expressed as functions f (X)), we have
g() > ((
(13)
Then, according Lemma 10 (3), since 7 7 2 0, it holds that
((J.g7T) 2 A(),g,7,
which, by Lemma further implies that
sip (C(fM, (M. + (H)) 2 lim sup (C(fM:99s + 2T) ++
sup M-++0
According to (13) and Lemma 5, we further obtain
inf (S(Ja::M, L((J.M, 2 inf C(JM. (UM-9MM77) - limssu [((++ 999M77) M-+00
M
M-+000
lim +00000
C(fM,9M
lim
(((f=) > 0.
M
Hence, it holds that
lim sum +13 2 limsup
(C(fM, 9M. + 215r (H))
M-+000
M +00000
> lim sup (C((f)) IMs T) 215s (H))
M +
+ liminf (C(fM,9ss -((f-))a.
M
lim sup (C((f))M. T) 21 (H)).
M 400
Particularly, for any 7 O, with = =(1). it holds that
lim (C(fM.9M, (H() > lim (((f.)).9MM..E) (1)) M M++30
M-+
for all r7 0.
Then, by the definition of Oc., (H), we can conclude that (f.g) & Oc,, which draws the contradiction. Hence, there exist measurable functions h and e such that
f(X) (h (X) g(Y) (E))")( almost surely.
By Lemma 6, it holds that ID(f), ID(g) < k. and combining with the result Step we obtain ID(S) ID(g) = k", which concludes the proof.
E
Extensions and additional theoretical results
E.1
Connection with sufficient dimension reduction
Sufficient dimension reduction (SDR) [38, 10, 11] is an important topic in statistics and machine learning, in which, the goal to find low-dimensional representation f(X) such that s(X) is sufficient for the conditional distribution of X in the sense that Y JL X (X). In the following proposition, will show that the representations learned by CLIP are also sufficient if V(H) # To simplify notation, where E 4 and Y E Rd are provided in Appendix 1. Here we denote id Rd + Rdd as the identical map and Xi idar(X), Then, have the following result.
focus the case where X,Y Rdd Extensions to
general setting,
30

---

Proposition 2. Assume V(H) * 2 and there exists (f,g) E H such that (1 (X)) gM (Y)) 1(MM YM)) O. For any (f gg)) E V(H the
lim o- dimension reductions f*(X) and g (Y) are sufficient, i.e. Y IL (X) and Y 1 gY).
We note that compared with Theorem 2. Proposition 2 has additional condition that there exists (f:g) E H such that lim in--r- (T(Sas ([[[a(()):: ga ga)) T(XM; Ym)) = One specific example that satisfies this condition is the scenario where we set (x) = (X) h ( X)) almost surely such that IL X (h1 (X), hy (X)), which is commonly considered the literature of (nonlinear) SDR [15, 16, 36, 8] and the tuple of nonlinear functions can be further extended to o-fieel [37, 8].
E.1.1 Proof of Proposition 2
We prove by showing that Y IL X | (X)). By the assumption that there exists (f,9) E H such that XX; =M L(f(X):ī({)) and the definition of V(H**) ), it holds that (Y; =M
IXf*(X): (")) for all gf") E V(H*)) Then, by property of mutual information ([58) Theo- rem 2.3), we have
I(g (Y)) 3M I(Y: 3M I(;; -M (Y)())))"
which implies that
I(Y - XX;
(14)
In addition, combining (14) with the Kolmogorov identity of mutual information ([58) Theorem 2.5 (2)) that
I(Y;)) -M I(Y; ((X) -M I(Y (')) I(Y; f*()));
further obtain I(Y; [ (x))) =M 0. According to the property of the conditional mutual information ([58] Theorem 2.5 (1)), XY;X (*())) =M 0 if and only if (XM, (X), Ym) forms Markov chain, i.e. YM IL XM (M) for all M M, which, by continuity of X, Y.
and f (X), further indicates that Y IL X [X)(X).
E.1.2 Extension of Proposition 2
We then turn to general case where X E Ba, and Y E Bdg where BA, and Bd, are bounded sets in Rd and R2,, respectively. If V(H) + Ø, we fix pair (f". :g")= V(H**) Consider discretizations DM (DM.s and EM = (iMM)) of Ba, and BAg, respectively, where Das, is the preimage of CM2 under f and ΣM, is the preimage cM, under g". Denote representatives in DM and ΣMA by IM and yM, respectively. In addition, assume DM and EM are nested discretizations and as + +oc, holds that
lim max diam(DM.) O. M-+00 sE[Ms]
lim max diam( (M.s) =
M M-++00 ME[M]
Similar notations for Ba before, we write XM and YM to denote discretized random vectors where
P(XM .M.) = P(X DM.)) and P(YM = vM.)) IPY E EM.)).
Then, we define a new partial order 3xx as follows. For any random vectors E Rd Y E Rdh U supported on Ba, and functionals F, G of random vectors, we define the order ZM associated with (f" (f".9") such that
F(X.)) SM G(U) = lim (F(XM, Ym) - G(UM)} $ 0. M
and moreover, the inequality is strict, i.c. F(X,Y) Smm G(U), if and only if
F(XM, Y.) 2Mm G(UM)) <.
lim sup M++00
We also write F(X, Y) AM G(U) if limm (F(XM, Ym) G(UM)) = O. Then, we state the extension of Proposition follows.
31

---

Proposition 3. Suppose V(H) & Ø. Assume there exists (f.g) E H such that I(X;Y) AM g(f());(()) with =M associated with (f" E v(HH*). Then, the dimension reductions (X) and g (Y) are sufficient, i.e.
Y X
and X
Proof of Proposition 3. Similar with the proof of Proposition 2, for any (f",g") E ), holds that
T(' (V)( 3M T(V: 3M X):
In addition, by assumption, there exists (f.g) E such that
T(Y; X) AM I(f(X):(())) 3m (V):)))
Combining results above, we obtain
X(; -M lgg fX):
In addition, by the Kolmogorov identity of mutual information ([58) Theorem 2.5 (2)), we obtain
X(Y;x) AM XX;; AM ((X) +
(*()),,
which implies that X (*(X)) =M 0. According to the property of the conditional mutual information ([58) Theorem 2.5 (I)), I(Y; f*()) =M 0 if and only if (XM,füx (X), YM) forms Markov chain, i.c. YM IL XM (x)(X) for all M E M, which, by continuity of X, Y and (X), further indicates that Y JLL X [X)(X).
E.2 Alignment and uniformity with correctly specified dimension
To begin with, recall that for any (f.g) E A(H)), it holds that Sá-1, Then, we define the set of uniformly distributed representations by
Up(H') ((.9) H'
Unif(S
Note that for any representation map f E H' with ID(J) = k; if the function class is sufficiently expressive, the linear dimension of R(f)) can exceed k, then recalling the definition of A(H() and (,HH))), we define
Dkk = max {p E [d] A(H*) n nup(H") # #0.
Specifically, write d' D(k", H). Then, define subset of H by 71" H2x Hy with
Hxx fs Exx
Hx. [(X)/E]](()I - E
Hy (1s0--19 :1 Hy, g(Y)/B)b(() E -*1
We further denote U(H+*) ((r.)) H ((X))/EII(())II)(())))) (o()) ~ Unif(("--1))) Then, by specifying the output dimension d d², we have the following result.
Theorem 3. With the function class H H² and output dimension d = d" D(k", H), it holds that
n Oc., V(H") (.H)).
120
Particularly, for any (f*,9") V(H*) the tuple (S*, """ , ) is minimizer of C(J TT
32

---

E.2.1 Proof of Theorem
By [79, Theorem (2)), if define f(x) f(x)) and g(Y) / then (f.g) is the global minimizer of in Ht for any given 0. Then, based on the definition of (H,,,, it holds that
A(H') ((())) n o,,, 820
In addition, with the choice of H", the uniformly distributed representation on Sd" maximizes the entropy. Hence, for any (f.g) E A(H")) nu(H)) we also have (J,g) W(H"), thus A(H*) n U(H²)) nu(H) W(H+* V(H") NU(H") which completes the proof.
F
Proof of preliminary results in Section A.2
F.1 Proof of Lemma
We adapt the proof in [12, Theorem 8.3.1]. The entropy of UM can be written as
H(UM) = Σ
pi log
iE[M]
(L plu)du log ( Jeme p(u)du)
Σ KE[M]
Σ
(p(zars) (og(EMM))) AM log AM + o(1).
KE[M]
Hence, it holds that
lim M M-+00
(H(Um) log Am) HT))
F22
Proof of Lemma 3
We note that for which, already see in Lemma that
(O(UM, V&s) UM) H(6UUM. VM),UM) - H(UM),
lim M M
(H(Um) + log Av) i)())
Denoting W (U.V). V(U (O(U,V), WM.ij = (EM3) and Carag (WMaj,, 2M3). In addition, we write
4i = PUU E CM.. V E CM.)),
4 = P(Ç E B(CM.) x cM.))).
and A ( 2M as the Lebesgue measure of ((CM,,, cM.)), AM as the Lebesgue measure of CM Define q (w, u) the joint density of 4 V) (d(U. U).
33

---

The joint entropy takes the form
H HOLUM, VMa), UM)
Σ
log Pij
E
(WM,s 2M, A (FM, 2M1,)) 2M
[.JE|M]
x log
(M. ( ZMJ)) o(1)
Σ
(WMij.
=M log 900 (wwMis RM.))
Σ
(WM,s 2M, (M. log
(A(()M +0(1
JJEMM]
E
(Wsss 2M, A (#M,s 2M1,) A log (9 (WM.. RM..))
Σ
q (WM,j, (RM,i ,M, log A (2
log Am + o(1).
Then, by the results in Lemma 1, we obtain
( VMd,, UnA) - H(UM)
RM.,) )M,)) log ( (wM, 2M,))
Σ
MJEM
(a)
+
[p(zMs)AM logp(=M.))
iM[M]
(b)
Σ
a (WWM.f.MM.)) (MM, log A (EM.) o(1)
[)E(M)
where, as M 400 and diam diam(cM..) 0, we have
(a) + - V), (b) -H(U).
In addition, as A (eM.,, <
lim sup
Σ (WMM.) (ZM,s, (M A log A (EM,EMM, <
M +o (JE[M]
Hence, we obtain
(H(((UM,,VV)) UM) HH(M))
lim inf
M M
(). - H(U)
-limsup
Σ q
log A (2M, 2M.))
M
(JE(M
-) = H(O(U,
34

---

F.3 Proof of Lemma 4
Denoting R ((U.), W (V.,V), and TMas o(2M.) 2M)) was,i ((EMss:2M,,))) we note that
()(U,)))(UU /f PR.W (2, w) Pn.w (r, w) pR(F)pw(w)) dzdw.
Denote V) V) v), and 13 P(A(U,V) E p(CMA x car)). zij V()(U.V) E (CCM, x cM,)), k = P(O(U.V) E D(CM, x cads)). In addition, denote (TMa), (MMM))) as the Lebesgue measure ((CM, cMa), Air (TM,)) wM.s)) as the Lebesgue measure of (CM., x (M.)), and A M (r..aj, mMas)) as the Lehesgue measure of vh(MM, x cM.j)- Then, for the discretized mutual information, we have
M, Var); (UM, VM))
(15)
+ Psj A +
(TMii WMas)
Pij Pija MJEIM
Σ
+
1 (Taa)) WM.)) log
o().
2 P.j
Pi
1
M
(TM,)) A (r
Since A.V (Tass, wM.)) = ( (WM.)) + o(1), (15) can be further written as T(ő(UM, Vad): Var))
= Σ
Pt
A (MM,s, (M.ss) log
Poj
+ o(1)
PizPis
Σ
(pi) log
A (rM.s)) (wM,6s)
[JE[M]
(a)
pij log (p(p)t) A (rM:s))) +o(1).
(b)
As M + too and | diam(cM,,) 0, it holds that
(a) + -- -H, W),
(b) -Ho(PR PR @ Pw).
where Hc(P,Q) Q) denotes the cross-entropy of Q relative to P. Hence, as M + +00 and maxie[M[] (iam(Ca,,) + 0,
(a) (b) + (KL PR Pw) I(R:W).
F4
Proof of Lemma 5
Since infoNCE loss is scale-invariant with respect both f and g. without loss of generality, we assume f(X]| = gY))l = 1. For any fineness M E M C N. recall the infoNCE loss
[((M.9M,7) - 2 (UM(X)9MM()))
(XM(())
Ex
log Ev
cxp
T
(FM(
Ey
log Ex
exp
T
Similar to the proof of [79, Theorem 1], we have
M M E (V))
lim
35

---

histogram of g
ave. of
R55 E122
ethu ESROTT
p [rX]]
|
2.0
400
400
16
1.5
300
300
1.0
200
200
0.5
100
0
0.0
0.0
0.5
1.0
1.5
200
400
600
0
2
of iterations
Figure 7: Histogram of similarities and norms of representations: (d, = (3, 2).
Epoch: 500
Epoch: 10
Epoch: 50
Epoch: 100
120
50 0033
0.00
1.00
1.00
0.00
1.00 1.20
g(Y)
-1.02
-1.00 1.00
Figure 8: Scatterplots of out-of-sample representations: (d, (3, 2).
and, by the boundedness of exp (f(()), E [0,e²] and the dominated convergence theorem,
(fN((X)
lim
Ex
log
log
exp
+000
(((X),
lim
Ex
log log
exp
M++00
T
(1(X))0(77))]
Ex
log By
exp
T
Hence, it holds that
lim [(M,,MM -(S,g,)) M
G Additional experimental results
In this section, we present additional results to Section 4 and we follow the same setting for each dataset in Section 4.
G.1 Norm concentration
Following the discussion in Section 1.1, we consider the same setting with (dd = (3, 2) (Fig- ure 7) and (d, k*) (20. 5) (Figure 9), respectively. In addition, for the setting with d = 3. we also present the scatterplot for oft-of-sample representations along the training process in Figure 8.
From Figure 7 and Figure 9, it is clear that (f.fgg) ( Rs 1, which indicates that (S.g) E A(H). This empirical finding is nontrivial in the sense that in our training, there is no constraint on [| and ( but, when the function class H is sufficiently expressive, the representations after population-level normalization tend to concentrate the unit hypersphere automatically.
1 and
36

---

histogram of g(()))
8V6.
|
1.2
200
4
1.0
150
0.8
100
2
0.6
50
50
0.4
0.0
0.5
1.0
1.5
200
400
600
Figure 9: Histogram of similarities and norms of representations: (d. (20, 5).
lemperature
MLE-ID
Epoch: 50
Epoch: 300
1.0
10)
gtY)
0.6
((Y)
0.00
8:00
0.6
0.1
TTX)
Pocoo
00
0.00
Boy
0.2
0.0
200
400
600
200
400
600
training epochs
tra training epochs
Figure 10: Results with two-layer ReLU: m (f.g) O.
G.2 Example with V(H) = Ø: alignment versus simply similarity maximization
To illustrate why the assumption V(H) & Ø is important, in this section, we use empirical examples to show that CLIP minimizers may not be semantically meaningful if ((H) = Ø. We consider the same setting in Section 1.1 but instead adopt the function class with 2-layer ReLU neural networks with the width of the middle layer as 50, in which case A(H) Ø. It illustrated in Figure 10 that CLIP can lead to minimizers (f: g) with mf(f.9) = O, i.e., orthogonal representations. However, even in this undesired setting, the intrinsic dimension k- 2 is correctly specified.
G.3 Experiment setup
We start with details of the experiment setup for real datasets. The same architecture is used in experiments: a 5-layer ReLU neural network with a width of middle-layer set to be 50 and varying input and output dimensions. The neural network is trained for 800 epochs with learning rate 10-3 and weight decay 10 10-4, and a slightly faster rate is used for temperature T: 10 3 in synthetic experiments and 2 x 10 for real data.
CITE-seq dataset. We follow the preprocessing in https://satijalal http:iijjaaaaaooo/ssuseeeee org/seurat/ wewllolss/eeighted_eeeeeeeee wawiresstneighbor... _aabbor_anallsss,,.aaa where we normalize ADT data with cen- tering to produce 24-dimension input, and normalize the extremely high-dimensional RNA data and extract the first 200 principal components the inputs for CLIP.
For the downstream classification tasks, we consider two set of labels. CITE-seq provides annota- tions of cells at two levels of granularity [21]. For BMCs data, at level (1), 5 major cell populations include CD4 T cells, CD8 T cells, B cells, classical monocytes (CM), and natural killer (NK) cells. Cell populations are further divided into 27 finer subpopulations at level (2).
ImageNetV2 and YFCC dataset. For the image-text datasets, we first use pretrained image and text encoders to transform images and texts to numerical inputs. Concretely, we adopt a pretrained ViT-14L from openai/clip-vit /clip-ittllarrree top-vit-large-ptcchl wwrgg--pacch4" (without projection) as the image encoder and a masked self-attention Transformer as the text encoder from the same pretrained model. Then,
large- 1ar.p.hhh....
37

---

with the pretrained encoders, we obtain 1024 dimenssona image inputs and f08-dimensional text inputs.
G.4
Convergence of temperature
We vary the output dimension and present the convergence of temperature for the following three datasets.
temperalure
lemperallte
Temperalure r
ozdre
1.0
19
19
23
15
0.2
0.2
0.2
0.0
0.0
0.0
600
(a) CITE- CITE-seq dataset.
(b) YFCC dataset.
ImageNetV2 dataset.
Figure 11: Convergence of temperature.
From Figure 11, we can see that with each real dataset and each choice of the output dimension, we have the temperature converging to zero, which is in line with our theory, and partially justifies that there are representations that can simultaneously maximize similarity and mutual information.
G.5 Comparison with cosine similarity
In this section, we present the experiment results with cosine similarity. Results for the synthetic (linear and nonlinear) and CITE-seq datasets are shown in Figure 12a, Figure 12b, and Figure 13, respectively.
lop-0.05% accuracy
MLE-intrinsic
top-0. accuracy
MLE-intrinsic dimension
20
1.0
-+- (X0
1.0
[X)
0.8
15
a(Y)
0.8
15
a(\)
0.6
10
0.6
10
0.4
0.4
0.2
in-sample
0.2
In-aampla
cul-of-sample
out-ot-sample
0.0
0.0
10
10
10
30
output
oulput dimensian
oulput dimonsion
aulput dimension
(a) Linear setting.
(b) Nonlincar sctting.
Figure 12: Results with synthetic dataset.
top-0.5% accuracy
cell type accuracy (1)
cell type accuracy (2)
MLE-innrinsic dimension
15.0
1.0
In-sample
1.0
1.0
sa-mmle
gM
0.8
0.8
0.8
10.0
0.6
0.6
0.6
7.5
0.4
0.4
0.4
5.0
0.2
0.2
in-eample out-of-sample
0.2
in-sample
2.5
out-of-sample
0.0
0.0
0.0
0.0
0
20
30
30
30
10
20
30
output dimension
output dimension
output dimenslon
output dimension
Figure 13: Results with CITE-seq dataset.
We can see that with two kinds similarity measures, the change of estimated intrinsic dimensions with varying output dimensions is nearly the same. With the similarity measure a(.,)) adopted in
38

---

top-0.5% accuracy
classification accuracy
MLE-intrinsic dim ension
10
1.0
1.0
in-sample
out-of-sample
0.8
0.8
In-sample out-of- sample
0.4
0.4
0.2
0.2
++ (OX)
g(Y)
0.0
20
40
20
40
20 output dimension
40
output dimension
output dimension
Figure 14: Results with YFCC dataset.
the paper, the top-oe% accuracy is even higher for synthetic data, and in the meantime, the top-a% accuracy for CITE-seq dataset is higher with cosine similarity, which partially due to the lower signal-to-noise ratio in the dataset and cosine similarity normalizes representations more strictly. In addition, for the downstream accuracy with respect two levels of cell types, the results with the two similarity measures are comparable.
G.6 Ablation study with Transformer encoder
Although our theory treats the function class as fixed and is agnostic the specification of H. we have additionally implemented small-scale Transformer with 2 layers, 2 heads, a feedforward dimension of 128, and embedding dimension of max(dz; dyJ, topped with final linear layer CITE-seq dataset (same setting with Figure 5), the results
with user-specified output dimension, of which are presented in the following table. From Table 2, we can see that both level-1 and level-2 accuracies averaged over 5 repetitions tend to saturate as the intrinsic dimension exceeds 15, which similar to the results presented in Figure 5. These results indicate that our theory is agnostic to network architecture long as the underlying architecture is sufficiently "expressive" so that the ideal representation maps can be -ell-approximated.
Table 2: Intrinsic dimension and cell type accuracy across output dimensions.
Output dimension
13
19
25
31
37
ID of f(X) ID of gYY)
1.00
5.67
9.79
12.19
14.15
15.67
15.97
1.00
5.15
8.63
10.39
11.75
12.64
12.96
Accuracy (level-1)
60.00%
93.00%
94.00%
94.00%
95.00%
95.00%
95.00%
Accuracy (level-2)
22.00%
78.00%
79.00%
80.00%
81.00%
79.00%
81.00%
G.7 Results with YFCC dataset
YFCC (or YFCC100M) is a multimedia dataset consisting of images and videos as media objects with metadata including title, tags, etc [72]. We adopt subset used by OpenAl and focus two modalities: images and text data with descriptions of images. In addition, each image in YFCC dataset assigned 9-level class label (farmid), which will be adopted a downstream classification task. We follow the same procedure as that in YFCC experiments. Similar to the CITE-seq experiments, randomly sample 10000 rows without replacement from the preprocessed dataset and randomly split the subset into a training set D Derain with [Deraiul 8000, test set D Drest with |Dtest = 1000, and a separate dataset with size 1000 to estimate the expected norms. With learned representation maps f E Fdh.d 99 Fdl.dd we consider image classification and the top-a% matching (a% = 0.5) as the downstream tasks on Drest- We vary the output dimension d from 1 to 29, and the results averaged after 50 repetitions are presented Figure 14. We can see that both accuracies tend to saturate when d is around 20, and the MLE-based estimation of the intrinsic dimension for the image embeddings is approximately 9 when d keeps increasing. Similarly, we use a llaer ReLU network for CLIP training only for illustration purposes, and better results can potentially be obtained with more sophisticated network architectures.
10 http:///tggaaattoooc/teeee//lttt
39

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency. research ethics, and societal impact. Do not re- move the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper. and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- ation. While "[Yes] is generally preferable to "[No] is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way, acknowledge that the answer is often
used"). In
nuanced, so please just your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list".
Keep the checklist subsection headings, questions/answers and guidelines below.
.
Do not modify the questions and only use the provided macros for your answers.
1.
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, which are justified by theoretical and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Ys]]
40

---

Justification:
We discuss the limitations and connections with the literature in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitttions" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
The assumptions are clearly discussed and discussed in Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Both experiment pipelines and dataset access are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
41

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome to describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Datasets are available online and reproducible codes are submitted as supple- mentary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co. heabb////guidee//eeeuubbbbbeeeee Sooe/eueeeeeeeeeeeeeeeeee ss asionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
42

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All details are presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: Simulations and experiments are small-scale and can be operated local laptops.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
43

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: ptttps://neuripee wwtpeerrrs.....pppp.eeeeee cctppblic//eteeeeccceeccccc
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics https: http://ennriisc..oee http://..........o.ccccccceeeea.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have a section for broad impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No risk.
Guidelines:
The answer NA means that the paper poses such risks.
44

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi paperswithcode thcode. com/ datasets has
curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: No assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A.
Guidelines:
45

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No LLM usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: htttps://neuripet wec/oonffeencee////0/ for what should or should be described.
46


Reviews:
[{"review_id": "pT5tWgpBsK", "reviewer": "Reviewer_5L7J", "summary": "This paper presents a novel theoretical analysis of multi-modal contrastive learning, particularly inspired by the success of models like CLIP. The authors investigate the properties of the learned representations when the contrastive loss function (infoNCE) is minimized jointly over the representation functions and the temperature parameter. The central claim is that this process not only maximizes the mutual information between modalities but also automatically adapts the learned representations to the intrinsic dimension of the shared latent variables, even when the user-specifies a higher-dimensional output space. The theoretical findings are supported by a comprehensive set of experiments on both synthetic and real-world datasets (CITE-seq, ImageNetV2, YFCC), demonstrating that the learned representations capture low-dimensional structures that are effective for downstream tasks.", "questions": "None", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The theoretical underpinnings of large-scale, self-supervised multi-modal models are a crucial and timely area of research. The paper astutely identifies a key phenomenon observed in practice—the learning of low-dimensional representations—and correctly points out the inability of existing theoretical frameworks (based on uniformity or simple mutual information maximization) to fully explain it.\n\n2. By analyzing the infoNCE loss with an optimized temperature, it provides a compelling explanation for three key phenomena:\ni. Similarity Concentration: Positive pairs achieve maximal similarity, while negative pairs are capped.\nii. Intrinsic Dimension Adaptation: The learned representations converge to the true intrinsic dimension of the shared information. This is the core contribution and a significant step beyond prior work which often required the true dimension to be known beforehand.\niii. Temperature Convergence: The optimization naturally drives the temperature parameter (τ) towards zero , which aligns with empirical observations.\n\n3. The authors build their theory on a solid foundation. The introduction of the set of \"ideal representations\" V(H)—defined as those that achieve both alignment and maximal mutual information—is elegant. The use of a discretization argument to rigorously handle the mutual information of continuous representations is technically sound and allows them to sidestep the issue of infinite mutual information for aligned representations.\n\n4. Comprehensive and Convincing Empirical Validation: The experiments are exceptionally well-designed to support the theoretical claims.\ni. Synthetic Data: By using synthetic data with a known ground-truth intrinsic dimension (k*), the authors clearly demonstrate that both downstream accuracy and the estimated intrinsic dimension saturate at k*, regardless of the user-specified output dimension (provided d≥k*). This provides direct, unambiguous evidence for their central claim.\nii. Real-World Data: The application to diverse, real-world datasets like CITE-seq , ImageNetV2 , and YFCC  shows that these phenomena are not just artifacts of a synthetic setup but are present in practical applications.\niii. Ablation Studies: The appendix provides crucial supporting evidence. The experiment in Appendix G.2, which shows that minimizers are not meaningful when the central assumption (V(H) is not ∅) is violated, is particularly insightful and strengthens the paper by clearly defining the scope and limitations of the theory.\n\nWeaknesses:\n1. The entire theoretical framework rests on the non-emptiness of V(H)—the assumption that a representation simultaneously achieving perfect alignment and maximal mutual information exists within the chosen function class H. This is a strong assumption. While the paper does an excellent job justifying it by connecting the convergence of temperature to zero in experiments to this condition, the link could be made more explicit. A deeper discussion on what properties of a function class (e.g., a sufficiently deep/wide neural network) and dataset would lead to this assumption holding would further strengthen the paper's practical implications.\n\n2. The definition of the infoNCE loss minimizer, using the set of near-minimizers O L,η(H) and the temperature threshold ϵ(η), is technically necessary to handle the asymptotic nature of the problem (τ→0). However, it is quite dense and may be difficult for readers to parse. While intuition is provided, a more streamlined or high-level explanation in the main text of Section 3.1 could improve readability and make the core results more accessible.\n\nOverall: This is a high-quality paper that makes a significant and novel contribution to our understanding of multi-modal contrastive learning. It successfully bridges theory and practice by providing a rigorous explanation for the empirically observed phenomenon of intrinsic dimension adaptation. The claims are well-supported by elegant theory and validated by an extensive set of well-conducted experiments. While the core assumption is strong, the authors are transparent about it and provide compelling evidence for its relevance. This work will be of great interest to researchers in self-supervised learning, representation learning, and multi-modal AI.", "was_revised": true, "final_justification": "Based on the discussion with the authors, the updates will strengthen the overall contribution of the paper.", "rebuttal": "We thank the reviewer for the positive feedback on our work, especially the rigor of our theoretical analysis, and the comprehensiveness of our empirical validation. Below, we address the weaknesses you identified.\n\n- [W1: assumption on $\\mathcal{V}(\\mathcal{H})$] Thank you for pointing this out! \n    - Regarding the assumption on $\\mathcal{V}(\\mathcal{H})$, we agree that it is an idealized statistical setting, in which we could sharply understand the performance of multi-modal contrastive learning, and the established theory can also explain the observed phenomena, especially the convergence of temperature and the revealed intrinsic dimension, in some real-world datasets (e.g. CITE-seq and ImageNet, additional evidence is presented in Appendix G.4 and G.6).\n    - We agree with the reviewer that although our theory treats the function class $\\mathcal{H}$ as fixed and is agnostic to the specification of $\\mathcal{H}$, we may expect $\\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. The choice in experiments (i.e. the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\\max(100, d_x, d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset (same setting with Figure 5), the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5. These results indicate that our theory is agnostic to network architecture as long as the underlying architecture is sufficiently “expressive” so that the ideal representation maps can be well-approximated. \n| output dimension    | 1    | 7    | 13   | 19   | 25   | 31   | 37   |\n|---------------------|------|------|------|------|------|------|------|\n| ID of f(X)          | 1.00 | 5.67 | 9.79 | 12.19| 14.15| 15.67| 15.97|\n| ID of g(Y)          | 1.00 | 5.15 | 8.63 | 10.39| 11.75| 12.64| 12.96|\n| cell type acc (1)   | 0.60 | 0.93 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 |\n| cell type acc (2)   | 0.22 | 0.78 | 0.79 | 0.80 | 0.81 | 0.79 | 0.81 |\nWe will add a detailed discussion on this condition and its connection with architectures and datasets.\n\n- [W2: readability of Section 3.1] Thank you for your helpful suggestions on exposition! To improve clarity and readability, we will revise Section 3.1 in the updated version to include a more intuitive and high-level explanation before presenting the formal definitions. This will highlight the core intuition behind our asymptotic analysis in the regime as $\\tau$ goes to zero and better motivate the role of the approximate minimizers that we defined."}, {"review_id": "txSVzSueyE", "reviewer": "Reviewer_qdkz", "summary": "This paper provides a theoretical analysis of CLIP-style multi-modal contrastive learning objectives. The authors show that, when temperature is optimized (like in CLIP), the InfoNCE loss not only aligns representations across modalities but also automatically adapts to the true intrinsic dimension of the shared latent variables. They formalize ideal representations via alignment and maximal mutual information (Definitions 1–2) and intrinsic dimension (Definition 3). Consequently in Theorem 2 show that any approxmate minimizers of the InfoNCE achieves similarity maximization, exact intrinsic-dimension adaptation, monotonicity in temperature, and maximal mutual information, even without requiring representations to be uniformly distributed over the entire output space. Empirical results on synthetic datasets and on a CITE-seq single-cell dataset confirm that downstream accuracy and estimated intrinsic dimensions saturate once the model’s output dimension exceeds the true dimension", "questions": "* Q1. Can the authors characterize finite-sample generalization for InfoNCE minimization? In particular, how many samples are needed to guarantee, with high probability, similarity maximization and intrinsic-dimension adaptation?\n* Q2. Empirical work reveals a persistent modality gap in CLIP-like models (few are cited in weakness section); can the V(H)≠∅ assumption be relaxed to account for approximate alignment?\n* Q3. Theorem 2 is proven under inner-product similarity and abstract hypothesis classes—how do these guarantees translate when using cosine similarity and modern architectures?\n* Q4. Many applications employ noisy pairs or one-to-many pairings, and multi-view data; can the analysis be extended to these settings, and under what conditions does intrinsic-dimension adaptation still hold in the presence of label noise or weak alignments?\n\nI would be happy to raise my rating if the authors satisfactorily address my concerns.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n* S1. The paper extends prior theoretical work on contrastive learning by explicitly incorporating temperature τ optimization into the analysis, explaining convergence as τ→0 and its role in intrinsic-dimension adaptation.\n* S2. Formal definitions of ideal representations and intrinsic dimension (Definitions 1–3) create a unified framework linking alignment, mutual information maximization, and latent representation complexity.\n* S3. Theorem 2 provides rigorous, population-level guarantees, including similarity maximization, exact intrinsic-dimension adaptation, temperature monotonicity, and maximal mutual information\n\n## Weaknesses\n* W1. The analysis assumes access to population-level objectives and infinite data; finite-sample effects and generalization bounds are not addressed\n* W2. The requirement V(H)≠∅ assumes the existence of perfectly aligned, maximally informative representations across modalities, yet empirical studies reveal a persistent modality gap in CLIP-like model, i.e. a systematic geometric separation between image and text embeddings that violates this assumption and undermines practical applicability [1, 2, 3]\n* W3. Experimental evaluation is limited to a 5-layer ReLU network and one biological dataset; broader benchmarks (e.g., large-scale image-text like COCO or CC3M) would strengthen the claims.\n* W4. The impact of different similarity measures (inner product vs. cosine) and architectural choices (e.g., transformers vs. MLPs) is only briefly mentioned.\n* W5. The paper does not explore how this theory extends to multi-class or noisy pairing scenarios common in real-world contrastive pretraining.\n\n[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning\n[2] It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap\n[3] Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion", "was_revised": true, "final_justification": "I found the new experiments on the modality gap especially insightful. I believe these additions would strengthen the paper.  I’m happy to update my score to borderline accept", "rebuttal": "We thank the reviewer for the valuable feedback and constructive suggestions, and for appreciating our analysis of temperature optimization and the rigor of our theory that links similarity maximization and mutual information distillation. We address weaknesses and questions below.\n\n- [finite-sample analysis: W1 & Q1] To understand the effect of sample size, from the empirical viewpoint, we conduct an ablation study in the same setting as Figure 3 (with a fixed output dimension 20) with varying sample size $N_{\\rm train}$ from 2000 to 14000. We present the estimated IDs, in-sample top-1 accuracy, and out-of-sample top-1 accuracy in the following table, from which we can see that as $N_{\\rm train} \\geq 10000$, these quantities begin to saturate, which validates that there is indeed a lower bound of $N_{\\rm train}$ to ensure good properties. To answer this question, it is also relevant to the convergence analysis of infoNCE loss in [Wang and Isola (2020), Theorem 1 (3)]. When we consider compact sets of maps and temperature (e.g., with a boundedness condition), we expect to derive the uniform law of large numbers for the finite-sample loss. In the updated version, we will add a detailed discussion on the convergence analysis below Lemma 8.\n| sample size       | 2000 | 4000 | 6000 | 8000 | 10000 | 12000 | 14000 |\n|-------------------|------|------|------|------|-------|--------|--------|\n| ID of f(X)        | 8.50 | 6.80 | 5.62 | 5.18 | 4.76  | 4.53   | 4.45   |\n| ID of g(Y)        | 8.54 | 6.88 | 5.81 | 5.46 | 5.07  | 4.87   | 4.71   |\n| in-sample ACC     | 0.38 | 0.75 | 0.91 | 0.95 | 0.96  | 0.96   | 0.96   |\n| out-of-sample ACC | 0.08 | 0.44 | 0.74 | 0.81 | 0.87  | 0.90   | 0.91   |\n\n- [modality gap and $\\mathcal{V}(\\mathcal{H})$: W2 & Q2] Thank you for pointing this out and for very helpful references! We will definitely add these references and related discussions on the modality gap to our updated version. Regarding the assumption on $\\mathcal{V}(\\mathcal{H})$, we agree that it is an idealized statistical setting, in which we could sharply understand the performance of multi-modal contrastive learning, and the established theory can also explain the observed phenomena, especially the convergence of temperature and the revealed intrinsic dimension, in some real-world datasets (e.g. CITE-seq and ImageNet, additional evidence is presented in Appendix G.4 and G.6). Also, the modality gap is indeed a crucial phenomenon in contrastive learning. To first see this empirically, we conduct an ablation study for the setting in Figure 2, where we add entrywise iid centered Gaussian noise to the first three components of X and Y, respectively (variance is set to be $\\sigma^2$ and $N_{\\rm train}=10000$). Empirical studies on the modality gap defined by $|| n^{-1} \\sum_{i \\leq n} f(X_i) - g(Y_i) ||$ with $n$ test samples [Liang et al., 2022] in the table show the following: \n\n    - (1) When $\\sigma \\leq 0.2$, both the estimated ID, the similarity maximization phenomenon, and top-K accuracies are comparable to those with $\\sigma=0$, indicating that our theory can be robust when the condition is approximately met. This will imply the future work on the robustness analysis of our framework, and we will add a detailed discussion on this in an updated version.\n    - (2) The computed modality gap increases with $\\sigma$, and its magnitude becomes more pronounced when $\\sigma > 0.4$. When $\\sigma = 0$, the tiny modality gap may be an artifact of the model training process, as it depends on the number of training epochs and step size scheduling. Hence, our setting with non-empty $\\mathcal{V}(\\mathcal{H})$ can be viewed as a baseline scenario, and the modality gap can, in this sense, quantify the difficulty in multi-modal learning, which also motivates future theoretical work on the robustness analysis. \n    | $\\sigma$                                      | 0     | 0.2   | 0.4   | 0.6   | 0.8   | 1.0   |\n    |----------------------------------------------|-------|-------|-------|-------|-------|-------|\n    | modality gap                                  | 0.056 | 0.078 | 0.075 | 0.105 | 0.159 | 0.178 |\n    | modality gap with self-normalized representations | 0.030 | 0.047 | 0.044 | 0.065 | 0.130 | 0.134 |\n\n\n\n\n\n- [alternative similarities and architectures: W3 & W4 & Q3] \n    - [similarities] To begin with, as shown in Figure 11 and 12, the empirical performance with cosine similarity and the similarity adopted in our paper are comparable. In addition, we would like to note that the key variational decomposition in Lemma 8, as well as the results in Lemma 10, are agnostic to the similarity measure and also apply to cosine similarity. Concretely, with a general similarity (including cosine similarity, or, equivalently, the function class maps to the unit hypersphere), as we have stated in Lemma 10 in Appendix C.2, we can also show that\n        - (1) with any $(f,g) \\in \\mathcal{V}(\\mathcal{H})$, $L(f,g,\\tau)$ is non-decreasing in $\\tau$ and  $-L(f,g,\\tau)/2$ converges to the maximum mutual information as $\\tau \\rightarrow 0$;\n        - (2) Reversely, by Lemma 10, for any $(f,g)$ as the approximate minimizer, similar to the proof in Appendix C 2.2, we can show that $(f,g)$ must maximize both mutual information and similarity, and $L(f,g,\\tau)$ is non-decreasing in $\\tau$. In addition, for intrinsic dimension adaptation with other similarities, specific technical tools may be needed. As a concrete example, with the cosine similarity, tools from potential minimization and geodesic projections on a hypersphere could be promising to obtain results similar to those in Appendix D. \n        - To sum up, the current similarity measure is a choice that has both comparable empirical performance with cosine similarity and elegant theoretical guarantees. We will add a more detailed discussion on variants of similarity measures in the revision.\n\n    - [architectures] Although our theory treats the function class $\\mathcal{H}$ as fixed and is agnostic to the specification of $\\mathcal{H}$, we expect $\\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. We would like to clarify that the choice in experiments (i.e. the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\\max(100, d_x, d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset (same setting with Figure 5), the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5. These results indicate that our theory is agnostic to network architecture as long as the underlying architecture is sufficiently “expressive” so that the ideal representation maps can be well-approximated.\n     | output dimension    | 1    | 7    | 13   | 19   | 25   | 31   | 37   |\n     |---------------------|------|------|------|------|------|------|------|\n     | ID of f(X)          | 1.00 | 5.67 | 9.79 | 12.19| 14.15| 15.67| 15.97|\n     | ID of g(Y)          | 1.00 | 5.15 | 8.63 | 10.39| 11.75| 12.64| 12.96|\n     | cell type acc (1)   | 0.60 | 0.93 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 |\n     | cell type acc (2)   | 0.22 | 0.78 | 0.79 | 0.80 | 0.81 | 0.79 | 0.81 |\n\n\n\n- [imperfect pairing: W5 & Q4] We agree that extending our results to settings with noisy pairs, one-to-many pairings, or multi-view data is an interesting and important future direction. \n    - For the noisy pairing, the simple simulation in response to the modality gap may offer us some useful insight: (1) Within a moderate range of perturbations of matched features, contrastive learning tend to be robust in revealing the ideal shared features; (2) When the perturbation is large, the modality gap in learned representations cannot be ignored, in which case, either new algorithms or advanced technical tools ought to be designed. Moreover, noisy pairing may also be related to multi-modal data with shared and modality-specific features mentioned by Reviewer dKqv. \n    - For the one-to-many scenario (e.g., for the pair $(X,Y)=({\\rm image}, {\\rm caption})$, suppose there are additional/augmented pairs taking the form $(X,Y_{\\rm aug})$ in the training set [Waseda et al., 2025]), if our goal is still to identify shared features and to ensure that our theory applies, one shortcut solution is to predefine an integration map $T_y$ (e.g., early fusion for the text modality), and define $T_y(Y,Y_{\\rm aug})$ as the new inputs. Then, if the augmentation maps are invertible (e.g., permutations), if $V(H) \\neq \\varnothing$ is true for each (image, caption) pair, then it can still hold for $(X, T_y(Y,Y_{\\rm aug}))$ in this simple setting. If more general scenarios with one-to-many and many-to-one pairs are of interest, more sophisticated statistical models should be introduced and a fine-grained set $\\mathcal{V}(\\mathcal{H})$ should be considered. It will be a compelling future direction to theoretically derive the theoretical guarantee in this setting to better understand the advantage of one-to-many data/augmentations in multi-modal learning. We will add detailed discussions on this aspect in the revision.\n\n**References**\n- Liang, Victor Weixin, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" \n- Waseda, Futa Kai, Antonio Tejero-de-Pablos, and Isao Echizen. \"Leveraging One-To-Many Relationships in Multimodal Adversarial Defense for Robust Image-Text Retrieval.\""}, {"review_id": "fKypHTVLXv", "reviewer": "Reviewer_dKqv", "summary": "This paper develops a theory of CLIP loss to analyze the efficiency of data alignment in the multi-modalities through optimizing the temperature. The main theory (theorem 2) claims that the CLIP automatically adapts to intrinsic dimensions near global minimizers, which contains four merits including the similarity maximization, the intrinsic dimension adaptation, monotonicity in temperature, and mutual information maximization. Empirical results on both synthetic and CITE-seq single-cell datasets confirm that matching accuracy and estimated intrinsic dimensions plateau once the output dimension exceeds latent dimension, validating the theory.", "questions": "Q1: Why the theory you care about intrinsic-dimension and temperature-tuned InfoNCE is important, and how does understanding it lead to the improvements (e.g. in embedding size or temperature scheduling) in real-world contrastive training?\n\n\n\nQ2: Theorem 2 crucially assumes that H contains all measurable functions, yet in experiments you restrict H to finite-width 5-layer ReLU networks. How do these capacity limitations affect the intrinsic-dimension adaptation guarantee, and can you clarify the robustness of the MLE-based dimension estimator under finite samples and nonlinear mappings?\n\n\nQ3: One important difference between the CLIP and the infoNCE loss is that the CLIP usually aligns two very different modalities.  How do the inherent differences between modalities (e.g. image vs. text)—impact your InfoNCE minimizer analysis and the resulting intrinsic-dimension adaptation? Could modality‐specific properties invalidate or alter the guarantees of Theorem 2?\n\n\n\nQ4: The experiments are generally good to me, including synthetic data, a small CITE-seq protein/RNA pairing, and ImageNetV2—can the authors demonstrate intrinsic-dimension adaptation on truly heterogeneous modalities (e.g. image–text CLIP datasets)?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "For strengths across the dimensions of originality, quality, clarity, and significance:\n\n\nQuality:\n\n\nOverall, this paper contains a well setup framework to support the intrinsic-dimension adaptation in CLIP loss. Even previous papers discussed the similarity maximization, monotonicity in temperature, or mutual information maximization, this paper provides a serious proof of those conclusions via the variational decomposition of InfoNCE loss.  The authors validate their theory with experiments on both synthetic and real single-cell datasets, demonstrating consistency between empirical findings and theoretical predictions. \n\n\nClarity: \n\n\nThe writing is generally clear and well-structured, with helpful motivating examples and detailed appendices. But still, the structure could be further improved.\n\n\nSignificance:\n\n\nAs this paper mentioned, the purpose here is to understand “How can one efficiently integrate data from multi-modalities?” While the paper sets out to explain how to integrate truly diverse modalities, its core contribution is a formal proof that any near-minimizer of the InfoNCE loss must automatically select the correct latent dimension and drive the temperature to zero—phenomena that earlier works had observed empirically but may not rigorously established—rather than offering new training algorithms or modality-specific insights. The practical impact on real-world multi-modal training awaits further demonstration.\n\n\nOriginality:\n\n\nThe paper’s main originality lies in its variational decomposition of the InfoNCE loss that explicitly integrates temperature optimization and a novel discretization framework to define near-minimizers. This unified approach lets the authors rigorously prove that contrastive learning automatically adapts to the true latent dimension and drives the temperature to zero—results previously only observed empirically and not shown for general (nonlinear, deep‐net) encoders.\n\n\nWeakness:\n\n\nW1: Theorem 2 gives a careful, rigorous proof that InfoNCE minimizers must maximize alignment, adapt to the true latent dimension, and drive temperature to zero, each of these behaviors has already been empirically noted in earlier contrastive-learning work.  The paper’s main novelty is the rigor of its discretization-based argument rather than fundamentally new phenomena.\n\n\nW2:  All theoretical results hinge on properties of the InfoNCE loss itself and make no use of any structure specific to language, vision, single-cell data, or other modalities. As a result, it’s unclear why this should be called a “multimodal” analysis rather than simply a general InfoNCE study.\n\nW3: The structure could be further improved, i. e., the Lemma 8’ s variational decomposition—are referenced repeatedly across sections even before the definition 1.  Meanwhile Theorem 1 is stated informally without proof, making it feel more like an unsubstantiated observation than a formal result.", "was_revised": true, "final_justification": "This is a good paper in multi-model.", "rebuttal": "We thank the reviewer for the valuable feedback, insightful questions, constructive suggestions for improvement, and for appreciating the novelty and quality of our theoretical analysis that incorporates temperature optimization. \nWe address weaknesses and questions as follows.\n\n- [W1: novelty] We acknowledge that some of the studied empirical phenomena have been observed in prior works. Our contribution is to rigorously formalize and unify these observations through a statistical lens, showing how they arise naturally from contrastive learning via mutual information maximization and intrinsic dimension adaptation. Prior works reported isolated effects without fully clarifying the underlying mechanisms or their connections. Our theory provides the first principled justification and explanation of these links, bridging empirical findings with fundamental properties of contrastive methods. We welcome additional references on empirical findings to further strengthen the discussion in our revision.\n\n- [W2: role of multi-modality] We appreciate this insightful comment. Our framework is motivated by multimodal settings because shared latent features and their intrinsic dimension are most naturally and meaningfully defined when aligning truly different modalities (e.g., RNA–protein, image–text). This is where temperature optimization and dimension adaptation are crucial, as different modalities may have heterogeneous noise levels, making it important to adaptively select the embedding dimension.\nThat said, the framework is general enough to recover single-modality contrastive learning as a special case: under a fixed augmentation strategy, the augmented data acts as a second modality, and the intrinsic dimension is determined by the augmentation-induced shared information. This demonstrates that our results apply broadly but are particularly impactful in multi-modal integration, where shared structure is less obvious and the benefits of adaptive temperature/dimension are amplified. We will revise the paper to make this motivation explicit and highlight modality-specific implications of our theory.\n\n- [W3: paper structure] Thank you for pointing this out! We agree that referencing Lemma 8’s variational decomposition prior to formally introducing Definition 1 may interrupt the logic flow. In the revised version, we will adjust the structure to introduce key definitions earlier and streamline cross-references where possible. Regarding Theorem 1, our intention was to present the core insight informally up front to convey intuition before delving into technical details. That said, we understand the concern, and in an updated version, we will point to the formal theorem to avoid confusion.\n\n- [Q1: real-world practice] Thank you for raising this insightful question. Our theoretical and experimental findings suggest that when contrastive learning is combined with temperature optimization, the learned features naturally adapt to the intrinsic dimension (ID) of the dataset. In practice, we recommend the following guideline:\n(1) Optimize the temperature parameter during training (as done in CLIP) to ensure that feature learning aligns with the data’s ID.\n(2) Start with a large output dimension and pretrain the model for a few hundred epochs to estimate the ID of the shared features. Then, iteratively (e.g., via binary search) reduce the output dimension until it is close to the estimated ID (as illustrated in Panel 4, Figure 5). This approach maintains accuracy while reducing inference cost in downstream tasks.\n\n- [Q2: capacity of function class and robustness of ID estimator] Thank you for this insightful question! \n    - [MLE-based ID estimator] The estimator was proposed in [Levina and Bickle, 2004]. This estimator is not restricted to data with linear structures, indicating that it can handle data supported on nonlinear manifolds (unlike eigenvalue-based methods such as global PCA). In addition, the asymptotic properties, including bias and variance, are also presented in [Levina and Bickle, 2004, Section 3.1]. Concretely, the estimator is unbiased and the variance decreases as the number of effective “neighbors” increases, which guarantees the finite-sample performance when the sample size is sufficiently large.\n    - [architecture] Although our theory treats the function class $\\mathcal{H}$ as fixed and is agnostic to the specification of $\\mathcal{H}$, we may expect $\\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. We would like to clarify that the choice in experiments (i.e., the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\\max(100, d_x,d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset, the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5.\n\n| Output dimension  | 1    | 7    | 13   | 19   | 25   | 31   | 37   |\n|------------------|------|------|------|------|------|------|------|\n| ID of f(X)       | 1.00 | 5.67 | 9.79 | 12.19| 14.15| 15.67| 15.97|\n| ID of g(Y)       | 1.00 | 5.15 | 8.63 | 10.39| 11.75| 12.64| 12.96|\n| cell type acc (1)| 0.60 | 0.93 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 |\n| cell type acc (2)| 0.22 | 0.78 | 0.79 | 0.80 | 0.81 | 0.79 | 0.81 |\n- [Q3: modality-specific features] Our theory focuses on shared representation adaptation under temperature optimization. Modality-specific components require specialized losses or post-processing (see [Liang et al., 2023; Wang et al., 2024]). Their presence does not contradict our results: when shared dimension is small, adapting to it is crucial, and our method facilitates subsequent extraction of unique modality features. We will clarify this point. \n\n- [Q4: heterogeneous modalities] We agree that testing on more heterogeneous datasets is valuable. In addition to CITE-seq, we report results on image–text datasets such as ImageNetv2 and YFCC  (Appendix G.4, G.6, G.7). We will expand experiments and quantify “intrinsic dimensions” across datasets in the revision.\n\n**References**\n- Levina, Elizaveta, and Peter Bickel. \"Maximum likelihood estimation of intrinsic dimension.\" Advances in neural information processing systems 17 (2004).\n- Liang, Paul Pu, Zihao Deng, Martin Q. Ma, James Y. Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. \"Factorized contrastive learning: Going beyond multi-view redundancy.\" Advances in Neural Information Processing Systems 36 (2023): 32971-32998.\n- Wang, Chenyu, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, and Caroline Uhler. \"An information criterion for controlled disentanglement of multimodal data.\" arXiv preprint arXiv:2410.23996 (2024)."}, {"review_id": "3SUqw97rHk", "reviewer": "Reviewer_qwpm", "summary": "This paper presents a thorough theoretical analysis of multi-modal contrastive learning, focusing on the properties of the learned representations beyond linear representations and specific data distributions. The authors provide a detailed investigation into how the intrinsic dimensions of shared latent variables influence the learning process and the resulting representations.", "questions": "- Although the author has conducted experimental verification on multiple datasets, the types and scales of these datasets are relatively limited. Could more complex models, such as those for video or audio, be tried for training?\n- However, in practical applications, how can we choose appropriate model parameters (such as output dimension, temperature parameter, etc.) based on the theoretical results? Can the author provide some practical guidelines based on the theoretical analysis to help researchers and developers better apply multimodal contrastive learning methods? For example, in practical tasks, how can we adjust the output dimension according to the intrinsic dimension of the data to achieve optimal performance?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- This paper is well written and easy to follow.\n- The paper offers significant theoretical contributions by analyzing the global minimizers of the InfoNCE loss and demonstrating how multi-modal contrastive learning adapts to the intrinsic dimension of data. The proof techniques are well-presented and provide a deeper understanding of the underlying mechanisms.\n- The experimental results on both synthetic and real-world datasets effectively support the theoretical findings.\n- The findings have the potential to impact the design and understanding of multi-modal models. I think this paper can give insights to people.\n\nWeakness\n\n- In writing, clearer notation should be provided. Many symbols are not explained in detail, which reduces readability.\n- The experimental part used a 5-layer ReLU network, but larger models should be tried to further illustrate the conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback, insightful questions, and constructive suggestions for improvement, and for appreciating our contribution to the understanding of multi-modal learning. \nWe address weaknesses and questions as follows.\n\n- [W1: notation] Thank you for the helpful feedback! In the revision, we will (1) add a notation table clarifying all symbols and variables; (2) ensure that important notations are adequately explained. \n\n- [W2 and Q1: architectures and datasets] We appreciate the reviewer’s questions on alternative architectures and datasets, and we will address them as follows.\n    - [architectures] Although our theory treats the function class $\\mathcal{H}$ as fixed and is agnostic to the specification of $\\mathcal{H}$, we may expect $\\mathcal{H}$ to have “expressive power” such that the set of “ideal” representation maps is not empty. We would like to clarify that the choice in experiments (i.e. the class of 5-layer ReLU MLPs) is only for illustrative purposes to justify our theory, and it turns out to be adequately powerful for datasets such as CITE-seq. In this sense, it is very meaningful to implement more powerful architectures in simulations and applications. We have additionally tried a small-scale Transformer (2 layers, 2 heads, feedforward dimension 64, and embedding dimension $\\max(100, d_x, d_y)$), topped with a final linear layer with user-specified output dimension, on CITE-seq dataset (same setting with Figure 5), the results of which are presented in the following table. From the table, we can see that both level-1 and level-2 accuracies tend to saturate as the intrinsic dimension exceeds 15, which is similar to the results presented in Figure 5.\n| output dimension    | 1    | 7    | 13   | 19   | 25   | 31   | 37   |\n|---------------------|------|------|------|------|------|------|------|\n| ID of f(X)          | 1.00 | 5.67 | 9.79 | 12.19| 14.15| 15.67| 15.97|\n| ID of g(Y)          | 1.00 | 5.15 | 8.63 | 10.39| 11.75| 12.64| 12.96|\n| cell type acc (1)   | 0.60 | 0.93 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 |\n| cell type acc (2)   | 0.22 | 0.78 | 0.79 | 0.80 | 0.81 | 0.79 | 0.81 |\n\n    - [datasets] It is meaningful to evaluate our theory on more real-world, especially heterogeneous, datasets. In addition to the CITE-seq dataset presented in the main paper, we would like to note that results on image-text datasets such as ImageNetv2 and YFCC are presented in Appendix G.4, G.6, and G.7. For these datasets, we also observe the convergence of optimized temperature to zero as is shown in Appendix G.4. In our updated version, we will present more experimental results, and it would also be interesting to quantify and compare the “intrinsic dimensions” across datasets.\n\n\n- [Q2: practical guidance] Thank you for raising this insightful question. Our theoretical and experimental findings suggest that when contrastive learning is combined with temperature optimization, the learned features naturally adapt to the intrinsic dimension (ID) of the dataset. In practice, we recommend the following guideline:\n(1) Optimize the temperature parameter during training (as done in CLIP) to ensure that feature learning aligns with the data’s ID.\n(2) Start with a large output dimension and pretrain the model for a few hundred epochs to estimate the ID of the shared features. Then, iteratively (e.g., via binary search) reduce the output dimension until it is close to the estimated ID (as illustrated in Panel 4, Figure 5). This approach maintains accuracy while reducing inference cost in downstream tasks."}]

Author rebuttal:

Meta-review: The paper provides a theoretical analysis of multi-modal contrastive learning, focusing on the InfoNCE loss when jointly optimizing the representation functions and temperature. The central claim is that this setup not only maximizes cross-modal mutual information but also adapts to the intrinsic dimension of the shared latent variables, even when the output dimension is larger.

All reviewers agree that this work should be accepted. While the limitations of "alignment and uniformity" and "MI maximization" have been known, prior theoretical analyses have been insufficient. This paper makes a meaningful theoretical contribution on this topic, and given the scarcity of such contributions, the work is valuable. Some reviewers noted the lack of direct empirical improvements based on the findings, but also agreed that empirical gains are not necessary for this type of contribution.

Therefore, acceptance is recommended.
Decision: Accept (poster)