Title: Understanding Contrastive Learning via Gaussian Mixture Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: contrastive learning; self-supervised learning; gaussian mixture models; linear dimensionality reduction
Abstract: Contrastive learning involves learning representations via a loss function that encourages each (unlabeled) sample to be far from other samples, but close to its own *augmentation*. In this paper, we aim to understand  why this simple idea performs remarkably well, by theoretically analyzing it for a simple, natural problem setting: dimensionality reduction in Gaussian Mixture Models (GMMs). Note that the standard GMM setup lacks the concept of augmentations. We study an intuitive extension: we define the pair of data sample and its augmentation as a coupled random draw from the GMM such that the marginal over the "noisy" augmentation is *biased* towards the component of the data sample. For this setup, we show that vanilla contrastive loss, e.g., InfoNCE, is able to find the *optimal* lower-dimensional subspace even when the Gaussian components are non-isotropic. In particular, we show that InfoNCE can match the performance of a fully supervised algorithm, e.g., LDA, (where each data point is labeled with the mixture component it comes from) even when the augmentations are "noisy". We further extend our setup to the multi-modal case, and develop a GMM-like setting to study the contrastive CLIP loss. We corroborate our theoretical with real-data experiments on CIFAR100; representations learned by InfoNCE loss match the performance of LDA on clustering metrics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies contrastive learning through the lens of linear dimensionality‑reduction for Gaussian‑mixture models (GMMs). By introducing an \"*augmentation‑enabled distribution*\" that biases each augmentation toward the same mixture component as its anchor, the authors prove that (i) InfoNCE and a simplified SimSiam objective recover the Fisher–optimal subspace for shared‑covariance GMMs, thereby matching the performance of fully‑supervised LDA, and (ii) a CLIP‑style multi‑modal InfoNCE learns a subset of the Fisher subspaces for each modality.\n\nExperiments on synthetic mixtures and CIFAR‑100 corroborate the theory, showing that InfoNCE outperforms PCA and approaches (sometimes exceeds) LDA on clustering metrics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- [theory] Sharp, closed‑form characterization of when InfoNCE is equivalent to LDA. Introduces a clean, tunable notion of “noisy” augmentations ($\\delta$‑biased draws).\n\n- [theory] Through comparison to spectral methods (SVD) and supervised LDA, the work offers valuable insights of when and why contrastive losses can match or exceed traditional methods in the linear setting.\n\n- [empirical validation] Systematic synthetic study isolates effect of (i) augmentation noise, (ii) anisotropy, (iii) projection rank. Real‑data CIFAR‑100 experiment shows gains over PCA and parity with LDA.\n\n**Weaknesses**\n\n- Linear Mapping Assumption: The analysis is restricted to linear projectors, whereas modern contrastive methods use deep nonlinear encoders. It remains unclear how these results extend to non‑linear settings.\n\n- Simplified SimSiam Analysis: The treatment of SimSiam employs a greatly simplified loss (no StopGrad or prediction head dynamics), raising questions about applicability to actual implementations.\n\n- Result for $0 < \\delta < 1$ is only conjectured (stated but unproven, l. 256‑260).\n\nQUESTIONS:\n- [minor] Nonlinear Extensions: Do the authors have insights or conjectures on how their linear‑theory might generalize to deep nonlinear encoders? Could kernel methods bridge this gap?\n\n- [minor] Finite‑Sample Bounds: Can the authors extend their analysis to finite datasets? What sample complexity is required for the empirical InfoNCE solution to approach the Fisher subspace?\n\n- [minor] Role of Temperature and Batch Size: InfoNCE performance in practice depends on hyperparameters like temperature and negative sampling. How do these factors interact with the theoretical model?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper setup a framework to analyze the contrastive learning through the perspective of GMMs, which projects the unlabeled augmentation pairs in InfoNCE into the Fisher-optimal subspace. This paper discusses the ability of infoNCE to recover the optimal shared-covariance projections, which states that minimizing the CLIP InfoNCE loss gives subspaces for each modality that lie within their respective Fisher optimal subspaces. Then they validate their theoretical findings with the synthetic data and CIFAR100 data.\n\nSTRENGTHS AND WEAKNESSES:\nFor strengths across the dimensions of originality, quality, clarity, and significance:\n\n\nQuality:\n\n\n\nTheory is good in proving that InfoNCE and SimSiam objectives recover the Fisher subspace under a shared‐covariance GMM, which discusses how the contrastive and non‐contrastive losses align with supervised LDA. Besides, this paper extends the analysis to a CLIP-style multi-modal GMM and shows that contrastive loss learns discriminative subspaces in each modality without explicit class information.\n\n\n\nClarity:\n\n\nIntroduces a clean, analytically tractable model (Augmentation-enabled GMM) that captures “noisy” augmentations and paired multi-modal data. Clearly describe the difference between the collapse scenario with the well separated scenario.\n\n\nSignificance:\n\n\n\nThere are a few papers trying to illustrate how contrastive learning works so well. However, this paper provides an interesting perspective to show that, InfoNCE and related losses recover the Fisher-optimal subspace matching supervised LDA. They extend this to multi-modal settings, which might potentially guide the future design of self-supervised and vision–language models.\n\n\n\nOriginality:\n\n\n\nThe main point in this paper, including casts contrastive learning as linear dimensionality reduction on GMM, defines “noisy” augmentations with AeD, and proves that minimizing InfoNCE recovers the Fisher-optimal subspace. These results are original to me.\n\n\n\n\nWeakness:\n\n\n\nW1: In this paper, the feature extraction through the matrix A looks like a linear mapping. The work does provide a clear analysis, but I want to know how these Fisher-subspace results extend to real encoder architectures or with more non-linear setup.\n\n\nW2:  Validation is restricted to small synthetic mixtures and a toy CIFAR-100 clustering setup (with grayscale, downsampled features and known K for K-Means). There’s no demonstration on large-scale vision or vision–language benchmarks to support the theories are useful under the CLIP situations.\n\n\n\nW3: Natural images or languages lie on highly curved, non-linear manifolds.  Their complex, non‐linear, multi‐modal manifolds could not be well captured by any finite mixture of Gaussians with shared covariance. All theoretical guarantees assume access to the true data distribution (infinite samples). It remains unclear how many real examples or negatives are needed for InfoNCE to work in practice without convergence-rate analysis.\n\nQUESTIONS:\nQ1: Your contributions show that InfoNCE recovers the Fisher‐optimal subspace under GMMs and extends to multi-modal CLIP losses—but could you clarify what concrete practical benefits or downstream applications that we can expect from these theoretical guarantees? This part seems ambiguous to me.\n\n\nQ2: In Theorem 4.1, you show that InfoNCE (and SimSiam) recover the full Fisher subspace only when the augmentation bias delta = 1. Could you clarify how sensitive the model is with these hyperparameters in practice when we try to ensure near-optimal projection recovery?\n\n\nQ3: For the multi-modal CLIP-GMM result (Theorem 5.2), you prove that CLIP InfoNCE learns only a subset of each modality’s Fisher subspace—could you suggest how we identify or recover the “missing” directions, and what downstream impact this subset selection has on retrieval or classification performance?\n\n\nQ4: The current scope of experiments is somehow narrow.  Only linear mappings, with no experiments on any multi-modal CLIP-style data or validation on deep, non-linear embeddings or real CLIP retrieval benchmarks. Could you comment on whether and how your CLIP-GMM theory holds up when applied to actual CLIP embeddings (e.g. ResNet+Transformer) on image–text retrieval tasks?\n\n\nQ5: Table 1 shows LDA achieving the highest ARI while InfoNCE leads in AMI—could you explain which properties of the learned projections drive this divergence, and what this implies for choosing one method over the other?\n\n\n\nMinor:\n\n1. Missing citations in line 63. will be generated by separate GMMs. “For instance, in the CLIP []”\n\n2. It contains more space in Definition 3.1.1.  \n\n3. In Figure 1, why (a) is well-separated while the (b) is the mode collapse? From the figure, the (b) looks more separated. \n\n4. Why in Line 187 the SVD subspace is italics while in line 189  is not?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new theoretical perspective on self-supervised learning via contrastive learning, aiming to explain why the simple strategy of contrasting or relating pairs is so effective. The authors formulate the representation learning problem as a form of dimensionality reduction for Gaussian Mixture Models (GMMs) with a shared covariance matrix, referred to as sharedGMMs. Their central finding is that contrastive methods learn the Fisher-optimal subspace—surpassing traditional unsupervised methods like SVD and approaching the performance of supervised techniques such as Linear Discriminant Analysis (LDA). To allow for analytical tractability, the analysis is restricted to linear mappings, and the optimality of representations learned via contrastive learning is studied under the assumption of perfect augmentations for the InfoNCE loss. The authors also distinguish between two categories of contrastive learning objectives: uni-modal and multi-modal GMMs, the latter modeled after CLIP-like architectures. In both scenarios, they demonstrate that contrastive learning with InfoNCE recovers a subset of the Fisher-optimal subspace. For empirical validation, the paper evaluates various linear dimensionality reduction methods on both synthetic sharedGMM datasets and real-world data (CIFAR-100), showing strong alignment between theoretical predictions and experimental results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n•\tThe proposed analytical approach to studying contrastive learning through the InfoNCE loss, even in the simplified setting of linear mappings, is both insightful and well-motivated. The theoretical connection to supervised LDA is particularly compelling.\n\n•\tThe analysis of optimal projection learning for multi-modal data is valuable. Demonstrating that multi-modal InfoNCE learns a subset of the Fisher-optimal subspace contributes meaningfully to the understanding of contrastive objectives in complex settings.\n\n•\tThe introduction of the Augmentation-enabled Distribution (AeD), which includes a hyperparameter to control the correlation between a sample and its augmentation, is novel and provides a more principled framework for modeling augmentations compared to traditional distance-based approaches.\n\n•\tThe manuscript is fairy well written and well-structured, making the theoretical contributions easy to follow.\n\n**Weaknesses**\n\n•\tTheorem 4.1 is restricted to the ideal case of perfect augmentation ($\\delta = 1$). However, such a scenario rarely holds in practice. The analysis does not account for more realistic conditions, such as limited numbers of positive or negative samples, or imperfect augmentations.\n\n•\tThere is no discussion of the number of positive and negative samples required for effective contrastive learning. This is a crucial factor, especially when sample balance and augmentation noise can significantly impact learning outcomes.\n\n•\tThe experimental analysis is quite limited. The use of CIFAR-100 as the primary dataset may not be well aligned with the assumptions of the theoretical framework. As shown in Table 1, the reported performance across all methods is quite low. A more suitable approach might involve first learning a nonlinear embedding for the image data that better satisfies the sharedGMM assumption, followed by applying linear mappings. This could also be extended to non-image datasets to further validate the theory.\n\n•\tWhile the paper touches on the role of augmentation and the distinction between perfect and noisy augmentations, a more quantitative analysis would be valuable. Specifically, measuring the effect of augmentation noise on the ratio of inter-component to intra-component variance would strengthen the empirical insights.\n\n•\tThe experimental details, particularly for the synthetic studies, are not quite clear and need to be clarified. More transparency in how the synthetic data is generated and evaluated would improve reproducibility and understanding.\n\n•\tAlthough the paper provides theoretical analysis for the multi-modal contrastive learning case, it lacks corresponding experimental validation. The authors could consider using a subset of the datasets from the CLIP paper to demonstrate their findings in practice.\n\nQUESTIONS:\n•  The authors state, “Although we do not provide a proof for $S_{Info}=S_{F}$ when $0<\\delta<1$ we conjecture that it is true, ...” (lines 256-7). Doesn’t this result depend on the noise level introduced by the augmentation process? Is it reasonable to assume that the learned subspace remains optimal under imperfect augmentations?\n\n•  The authors say that *“both contrastive and non-contrastive objectives learn the same subspace”* (lines 261–262). However, Theorem 4.1 suggests that $S_{Siam}$ recovers the Fisher subspace even without perfect augmentations ($\\delta < 1$), while the results for InfoNCE are tied specifically to the assumption of perfect augmentations. This appears inconsistent with the general conclusion (without making use of negative samples). Could the authors clarify this discrepancy?\n\n•  In practice, augmentation techniques do not rely on independently sampled transformations. Instead, each augmentation is typically generated conditionally, based on the original sample (e.g., type-preserving augmentations). Would it not be more realistic to model the augmentation process as a conditional distribution $p(\\hat{x}∣x)$? How would this change the theoretical conclusions, especially under statistical dependencies introduced by type-based augmentations?\n\n•  Theorem 4.1 assumes $r \\geq K$, but in all experiments, the projection dimension $r$ appears to be smaller than the number of clusters. Could the authors explain this mismatch between theory and experimental setup?\n\n•  The synthetic experiments are not clearly described. What is the data distribution? What is the ambient space dimension? Why does the \"optimal\" method perform worse? Is the data generated from a sharedGMM? Why is $r=K−1$ used in the numerical experiments?What is the rationale behind the design of the cluster means?\n\n•  The authors write that *“InfoNCE loss learns to scale within the subspace leading to better clustering performance”* (lines 336–337). Could the authors elaborate on what is meant by \"scaling within the subspace\" and how this contributes to improved clustering?\n\n•  In Definition 2.2, the equation for AeD, should $K \\neq K'$ be satisfied? \n\n•  The paper defines a \"good projection\" as one that performs well in classification tasks. Is this definition sufficient for evaluating the quality of representations more broadly, especially in unsupervised or transfer learning scenarios?\n\n•  Section G.1 in the supplementary material appears to be incomplete. \n\n•  In line 63, the citation is missing.\n\nLIMITATIONS:\nI think the main limitations of the proposed analytical framework for studying contrastive learning are as follows:\n\n•\tIt relies on the assumption of shared covariance in the GMM model, which is a strong condition and may not hold in many practical scenarios.\n\n•\tIt lacks an explicit analysis or quantification of the effects of augmentation noise and the overlap between components in the GMM, both of which are critical factors in real-world applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors investigate a Gausian Mixture Model variant of contrastive learning. They show a noisy label contrastive setup can converge to the same subspace as discovered through LDA (which needs class labels). They perform quantitative experiments on small scale datasets to demonstrate the theory and show how it varies with key parameters like noise fraction.

STRENGTHS AND WEAKNESSES:
Strengths:
-	Good attention to detail in the mathematics and notation
-	Interesting theorem connecting contrastive learning with LDA and Fisher subspaces
-	Reasonable analysis of results and experiments
Weaknesses:
-	The work is good but comes off a bit dense at times, perhaps include a overview figure in the work that captures the main plot, this will make it easier for people to quickly glean the important aspects of your work.
-	Minor typos
-	Smaller scale datasets for evaluation

QUESTIONS:
-Line 63 Missing clip cite
- Can you help me understand the impact of the shared covariance on the model. Is this an assumption that all clusters have the same shape? If so this seems a little restrictive and I wonder whether your work could be applied to the general GMM case. 
- (For the authors curiosity) One paper that seems relevant to your work is “I-Con: A Unifying Framework for Representation Learning” from the looks of it, your theorem might represent a new row or column in their periodic table of representation learning algorithms.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02