Title: Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables
Venue: NeurIPS 2025
Accepted: yes
Keywords: contrastive learning; multi-modal learning; machine learning theory;
Abstract: Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP~\citep{radford2021learning}. 
In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions. 
Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors. 
Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learning to learn low-dimensional and informative representations, bridging theoretical insights and practical performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a thorough theoretical analysis of multi-modal contrastive learning, focusing on the properties of the learned representations beyond linear representations and specific data distributions. The authors provide a detailed investigation into how the intrinsic dimensions of shared latent variables influence the learning process and the resulting representations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- This paper is well written and easy to follow.\n- The paper offers significant theoretical contributions by analyzing the global minimizers of the InfoNCE loss and demonstrating how multi-modal contrastive learning adapts to the intrinsic dimension of data. The proof techniques are well-presented and provide a deeper understanding of the underlying mechanisms.\n- The experimental results on both synthetic and real-world datasets effectively support the theoretical findings.\n- The findings have the potential to impact the design and understanding of multi-modal models. I think this paper can give insights to people.\n\nWeakness\n\n- In writing, clearer notation should be provided. Many symbols are not explained in detail, which reduces readability.\n- The experimental part used a 5-layer ReLU network, but larger models should be tried to further illustrate the conclusions.\n\nQUESTIONS:\n- Although the author has conducted experimental verification on multiple datasets, the types and scales of these datasets are relatively limited. Could more complex models, such as those for video or audio, be tried for training?\n- However, in practical applications, how can we choose appropriate model parameters (such as output dimension, temperature parameter, etc.) based on the theoretical results? Can the author provide some practical guidelines based on the theoretical analysis to help researchers and developers better apply multimodal contrastive learning methods? For example, in practical tasks, how can we adjust the output dimension according to the intrinsic dimension of the data to achieve optimal performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a theory of CLIP loss to analyze the efficiency of data alignment in the multi-modalities through optimizing the temperature. The main theory (theorem 2) claims that the CLIP automatically adapts to intrinsic dimensions near global minimizers, which contains four merits including the similarity maximization, the intrinsic dimension adaptation, monotonicity in temperature, and mutual information maximization. Empirical results on both synthetic and CITE-seq single-cell datasets confirm that matching accuracy and estimated intrinsic dimensions plateau once the output dimension exceeds latent dimension, validating the theory.\n\nSTRENGTHS AND WEAKNESSES:\nFor strengths across the dimensions of originality, quality, clarity, and significance:\n\n\nQuality:\n\n\nOverall, this paper contains a well setup framework to support the intrinsic-dimension adaptation in CLIP loss. Even previous papers discussed the similarity maximization, monotonicity in temperature, or mutual information maximization, this paper provides a serious proof of those conclusions via the variational decomposition of InfoNCE loss.  The authors validate their theory with experiments on both synthetic and real single-cell datasets, demonstrating consistency between empirical findings and theoretical predictions. \n\n\nClarity: \n\n\nThe writing is generally clear and well-structured, with helpful motivating examples and detailed appendices. But still, the structure could be further improved.\n\n\nSignificance:\n\n\nAs this paper mentioned, the purpose here is to understand “How can one efficiently integrate data from multi-modalities?” While the paper sets out to explain how to integrate truly diverse modalities, its core contribution is a formal proof that any near-minimizer of the InfoNCE loss must automatically select the correct latent dimension and drive the temperature to zero—phenomena that earlier works had observed empirically but may not rigorously established—rather than offering new training algorithms or modality-specific insights. The practical impact on real-world multi-modal training awaits further demonstration.\n\n\nOriginality:\n\n\nThe paper’s main originality lies in its variational decomposition of the InfoNCE loss that explicitly integrates temperature optimization and a novel discretization framework to define near-minimizers. This unified approach lets the authors rigorously prove that contrastive learning automatically adapts to the true latent dimension and drives the temperature to zero—results previously only observed empirically and not shown for general (nonlinear, deep‐net) encoders.\n\n\nWeakness:\n\n\nW1: Theorem 2 gives a careful, rigorous proof that InfoNCE minimizers must maximize alignment, adapt to the true latent dimension, and drive temperature to zero, each of these behaviors has already been empirically noted in earlier contrastive-learning work.  The paper’s main novelty is the rigor of its discretization-based argument rather than fundamentally new phenomena.\n\n\nW2:  All theoretical results hinge on properties of the InfoNCE loss itself and make no use of any structure specific to language, vision, single-cell data, or other modalities. As a result, it’s unclear why this should be called a “multimodal” analysis rather than simply a general InfoNCE study.\n\nW3: The structure could be further improved, i. e., the Lemma 8’ s variational decomposition—are referenced repeatedly across sections even before the definition 1.  Meanwhile Theorem 1 is stated informally without proof, making it feel more like an unsubstantiated observation than a formal result.\n\nQUESTIONS:\nQ1: Why the theory you care about intrinsic-dimension and temperature-tuned InfoNCE is important, and how does understanding it lead to the improvements (e.g. in embedding size or temperature scheduling) in real-world contrastive training?\n\n\n\nQ2: Theorem 2 crucially assumes that H contains all measurable functions, yet in experiments you restrict H to finite-width 5-layer ReLU networks. How do these capacity limitations affect the intrinsic-dimension adaptation guarantee, and can you clarify the robustness of the MLE-based dimension estimator under finite samples and nonlinear mappings?\n\n\nQ3: One important difference between the CLIP and the infoNCE loss is that the CLIP usually aligns two very different modalities.  How do the inherent differences between modalities (e.g. image vs. text)—impact your InfoNCE minimizer analysis and the resulting intrinsic-dimension adaptation? Could modality‐specific properties invalidate or alter the guarantees of Theorem 2?\n\n\n\nQ4: The experiments are generally good to me, including synthetic data, a small CITE-seq protein/RNA pairing, and ImageNetV2—can the authors demonstrate intrinsic-dimension adaptation on truly heterogeneous modalities (e.g. image–text CLIP datasets)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of CLIP-style multi-modal contrastive learning objectives. The authors show that, when temperature is optimized (like in CLIP), the InfoNCE loss not only aligns representations across modalities but also automatically adapts to the true intrinsic dimension of the shared latent variables. They formalize ideal representations via alignment and maximal mutual information (Definitions 1–2) and intrinsic dimension (Definition 3). Consequently in Theorem 2 show that any approxmate minimizers of the InfoNCE achieves similarity maximization, exact intrinsic-dimension adaptation, monotonicity in temperature, and maximal mutual information, even without requiring representations to be uniformly distributed over the entire output space. Empirical results on synthetic datasets and on a CITE-seq single-cell dataset confirm that downstream accuracy and estimated intrinsic dimensions saturate once the model’s output dimension exceeds the true dimension\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* S1. The paper extends prior theoretical work on contrastive learning by explicitly incorporating temperature τ optimization into the analysis, explaining convergence as τ→0 and its role in intrinsic-dimension adaptation.\n* S2. Formal definitions of ideal representations and intrinsic dimension (Definitions 1–3) create a unified framework linking alignment, mutual information maximization, and latent representation complexity.\n* S3. Theorem 2 provides rigorous, population-level guarantees, including similarity maximization, exact intrinsic-dimension adaptation, temperature monotonicity, and maximal mutual information\n\n## Weaknesses\n* W1. The analysis assumes access to population-level objectives and infinite data; finite-sample effects and generalization bounds are not addressed\n* W2. The requirement V(H)≠∅ assumes the existence of perfectly aligned, maximally informative representations across modalities, yet empirical studies reveal a persistent modality gap in CLIP-like model, i.e. a systematic geometric separation between image and text embeddings that violates this assumption and undermines practical applicability [1, 2, 3]\n* W3. Experimental evaluation is limited to a 5-layer ReLU network and one biological dataset; broader benchmarks (e.g., large-scale image-text like COCO or CC3M) would strengthen the claims.\n* W4. The impact of different similarity measures (inner product vs. cosine) and architectural choices (e.g., transformers vs. MLPs) is only briefly mentioned.\n* W5. The paper does not explore how this theory extends to multi-class or noisy pairing scenarios common in real-world contrastive pretraining.\n\n[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning\n[2] It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap\n[3] Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion\n\nQUESTIONS:\n* Q1. Can the authors characterize finite-sample generalization for InfoNCE minimization? In particular, how many samples are needed to guarantee, with high probability, similarity maximization and intrinsic-dimension adaptation?\n* Q2. Empirical work reveals a persistent modality gap in CLIP-like models (few are cited in weakness section); can the V(H)≠∅ assumption be relaxed to account for approximate alignment?\n* Q3. Theorem 2 is proven under inner-product similarity and abstract hypothesis classes—how do these guarantees translate when using cosine similarity and modern architectures?\n* Q4. Many applications employ noisy pairs or one-to-many pairings, and multi-view data; can the analysis be extended to these settings, and under what conditions does intrinsic-dimension adaptation still hold in the presence of label noise or weak alignments?\n\nI would be happy to raise my rating if the authors satisfactorily address my concerns.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel theoretical analysis of multi-modal contrastive learning, particularly inspired by the success of models like CLIP. The authors investigate the properties of the learned representations when the contrastive loss function (infoNCE) is minimized jointly over the representation functions and the temperature parameter. The central claim is that this process not only maximizes the mutual information between modalities but also automatically adapts the learned representations to the intrinsic dimension of the shared latent variables, even when the user-specifies a higher-dimensional output space. The theoretical findings are supported by a comprehensive set of experiments on both synthetic and real-world datasets (CITE-seq, ImageNetV2, YFCC), demonstrating that the learned representations capture low-dimensional structures that are effective for downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The theoretical underpinnings of large-scale, self-supervised multi-modal models are a crucial and timely area of research. The paper astutely identifies a key phenomenon observed in practice—the learning of low-dimensional representations—and correctly points out the inability of existing theoretical frameworks (based on uniformity or simple mutual information maximization) to fully explain it.\n\n2. By analyzing the infoNCE loss with an optimized temperature, it provides a compelling explanation for three key phenomena:\ni. Similarity Concentration: Positive pairs achieve maximal similarity, while negative pairs are capped.\nii. Intrinsic Dimension Adaptation: The learned representations converge to the true intrinsic dimension of the shared information. This is the core contribution and a significant step beyond prior work which often required the true dimension to be known beforehand.\niii. Temperature Convergence: The optimization naturally drives the temperature parameter (τ) towards zero , which aligns with empirical observations.\n\n3. The authors build their theory on a solid foundation. The introduction of the set of \"ideal representations\" V(H)—defined as those that achieve both alignment and maximal mutual information—is elegant. The use of a discretization argument to rigorously handle the mutual information of continuous representations is technically sound and allows them to sidestep the issue of infinite mutual information for aligned representations.\n\n4. Comprehensive and Convincing Empirical Validation: The experiments are exceptionally well-designed to support the theoretical claims.\ni. Synthetic Data: By using synthetic data with a known ground-truth intrinsic dimension (k*), the authors clearly demonstrate that both downstream accuracy and the estimated intrinsic dimension saturate at k*, regardless of the user-specified output dimension (provided d≥k*). This provides direct, unambiguous evidence for their central claim.\nii. Real-World Data: The application to diverse, real-world datasets like CITE-seq , ImageNetV2 , and YFCC  shows that these phenomena are not just artifacts of a synthetic setup but are present in practical applications.\niii. Ablation Studies: The appendix provides crucial supporting evidence. The experiment in Appendix G.2, which shows that minimizers are not meaningful when the central assumption (V(H) is not ∅) is violated, is particularly insightful and strengthens the paper by clearly defining the scope and limitations of the theory.\n\nWeaknesses:\n1. The entire theoretical framework rests on the non-emptiness of V(H)—the assumption that a representation simultaneously achieving perfect alignment and maximal mutual information exists within the chosen function class H. This is a strong assumption. While the paper does an excellent job justifying it by connecting the convergence of temperature to zero in experiments to this condition, the link could be made more explicit. A deeper discussion on what properties of a function class (e.g., a sufficiently deep/wide neural network) and dataset would lead to this assumption holding would further strengthen the paper's practical implications.\n\n2. The definition of the infoNCE loss minimizer, using the set of near-minimizers O L,η(H) and the temperature threshold ϵ(η), is technically necessary to handle the asymptotic nature of the problem (τ→0). However, it is quite dense and may be difficult for readers to parse. While intuition is provided, a more streamlined or high-level explanation in the main text of Section 3.1 could improve readability and make the core results more accessible.\n\nOverall: This is a high-quality paper that makes a significant and novel contribution to our understanding of multi-modal contrastive learning. It successfully bridges theory and practice by providing a rigorous explanation for the empirically observed phenomenon of intrinsic dimension adaptation. The claims are well-supported by elegant theory and validated by an extensive set of well-conducted experiments. While the core assumption is strong, the authors are transparent about it and provide compelling evidence for its relevance. This work will be of great interest to researchers in self-supervised learning, representation learning, and multi-modal AI.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}