Understanding Contrastive Learning via Gaussian Mixture Models
Authors: Parikshit Bansal
Ali Kavis
sujay sanghavi
Venue: NeurIPS 2025 poster ()
Keywords: contrastive learning; self-supervised learning; gaussian mixture models; linear dimensionality reduction
Primary area: general_machine_learning

Abstract: Contrastive learning involves learning representations via a loss function that encourages each (unlabeled) sample to be far from other samples, but close to its own *augmentation*. In this paper, we aim to understand  why this simple idea performs remarkably well, by theoretically analyzing it for a simple, natural problem setting: dimensionality reduction in Gaussian Mixture Models (GMMs). Note that the standard GMM setup lacks the concept of augmentations. We study an intuitive extension: we define the pair of data sample and its augmentation as a coupled random draw from the GMM such that the marginal over the "noisy" augmentation is *biased* towards the component of the data sample. For this setup, we show that vanilla contrastive loss, e.g., InfoNCE, is able to find the *optimal* lower-dimensional subspace even when the Gaussian components are non-isotropic. In particular, we show that InfoNCE can match the performance of a fully supervised algorithm, e.g., LDA, (where each data point is labeled with the mixture component it comes from) even when the augmentations are "noisy". We further extend our setup to the multi-modal case, and develop a GMM-like setting to study the contrastive CLIP loss. We corroborate our theoretical with real-data experiments on CIFAR100; representations learned by InfoNCE loss match the performance of LDA on clustering metrics.

Understanding Contrastive Learning via Gaussian Mixture Models
Parikshit Bansal
Ali Kavis UT Austin kaviseaustin utexas edu
Sujay Sanghavi
UT Austin Saansal@utexas edu
UT Austin sanghavi @mail utexas edu
Abstract
Contrastive learning involves learning representations via loss function that en- courages each (unlabeled) sample to be far from other samples, but close to its own augmentation. In this paper, we aim understand why this simple idea performs remarkably well, by theoretically analyzing it for simple, natural problem setting: dimensionality reduction in Gaussian Mixture Models (GMMs). Note that the standard GMM setup lacks the concept of augmentations. We study an intuitive extension: we define the pair of data sample and its augmentation as a coupled random draw from the GMM such that the marginal over the "noisy" augmentation is biased towards the component the data sample. For this setup, we show that vanilla contrastive loss, e.g.. InfoNCE, is able to find the optimal lower-dimensional subspace even when the Gaussian components are non-isotropic. In particular, we show that InfoNCE can match the performance of fully supervised algorithm, c.g., LDA, (where each data point is labeled with the mixture component it comes from) even when the augmentations are "noisy". We further extend setup to the multi-modal case, and develop a GMM-like setting to study the contrastive CLIPP loss. We corroborate our theory with experiments CIFAR 100; representations learned by InfoNCE loss match the performance of LDA on clustering metrics.
Introduction
Contrastive learning (CL) is now a gold-standard paradigm for learning representations with popular examples such as vision models like SimCLR [6] and MoCo [8] text models like Dense Passage Retrieval (DPR) [18] and vision-language models like CLIP
20 Once learned, these representations both demonstrate commendable zero-shot performance, and could easily be adapted to achieve SoTA
performance on various tasks like classification 6 and retrieval [13.
Contrastive learning is a self-supervised strategy that aims to learn coherent representations given unlabeled dataset. The core idea is to relate similar (positive) data samples each other while contrasting unrelated (negative) points. To formalize, consider a pair of points (æ, æ),, where æ is sample from the dataset and à is "similar". positive example. Contrastive methods learn representations so that the embedding of the sample x is close to that of its partner ă, while being far away from other points. Sometimes, these pairs are naturally present. For instance, in CLIP, each training data comes as a pair of an image and the corresponding text caption. In other scenarios, they can be manually obtained via data augmentation that preserves the underlying semantics, e.g., we can define an image and its resizee//roopeed/rttaee version pair.
It is fundamentally important to understand why this simple idea of relating/contrastingg pairs works so well. In this paper, we aim to answer this question with a theoretical study of contrastive learning by casting is as dimensionality reduction for Gaussian Mixture Models (GMMs). We motivate our study through the lens representation learning which aims at finding a mapping between the high-dimensional input space, e.g., images of million pixels, and lower dimensional
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

output domain of representations, e.g., 1024-dimension feature vectors. Similarly, contrastive learning aims at discovering robust mappings between the data and the representations by exploiting additional information in the form of augmentations. In our framework of GMMs, we consider linear mappings/projections (for analytical tractability) and characterize the optimality of the mapping functions learned by contrastive methods by analyzing the associated projection subspaces.
To highlight the success of contrastive methods in learning linear projectors, we conduct our study in comparison to fully supervised methods and standard spectral methods, i.e., based on singular value decomposition (SVD). We provide an example where we provide three-way comparison in finding anticipated projection subspaces (see Section B.I. Note that SVD-based methods are not designed to leverage augmentations. Therefore, we use SVD-based methods merely as reference point to argue that when (noisy) augmentations are present, contrastive learning can provably go beyond what would otherwise he possible. However, a more relevant and interesting comparison is with respect to supervised methods, i.e., LDA in this case. In that sense, we provide principled, theoretical insight into how contrastive learning works and how it performs with respect to measure of optimality.
Within our GMM framework, a key proposal of ours is formalizing the "notion of augmentations", i.e. point-pairs, going beyond the idealized definition considered in [3] In the context of GMMs, a natural choice of augmentation would be drawing a random sample from the same component as the original point belongs to. Instead, we assume that for every point æ, its augmentation is a random draw from the GMM following distribution (see Definition 2.2) that is biased towards the component of the original point æ. To keep the setting realistic, we are oblivious to which GMM component either of them comes from.
Given the model of noisy augmentations, we analyze contrastive learning objectives in two categories; single and multi-modal GMMs. In the single modal setting, the points in a pair are drawn from the same GMM distribution. We analyze InfoNCE and SimSiam losses, which are suited to leverage the additional information provided by the augmentations. In the multi-modal context, the pair (æ, )) will be generated by separate GMMs. For instance, in the CLIP [] images are associated with text descriptions such that we have pairs of the form (zv, DT), where Zv (images) and ør (texts) can have different dimensions. The objective now is learn two different projections, one for each modality to project them onto the same space. Contrastive methods aim to make the embedding of av close to that of its pair æT, and far from random other äT. Since the samples come in natural pairs, we do not need augmentations. We provide further details in Section 5
Contributions: Our objective is to theoretically understand the vital role of augmentations in self-supervised representation learning; we do so in the context of GMMs. To summarize,
1. We formalize a generalized notion of noisy augmentation/point pairs, enabling us to concretely characterize contrastive learning methods in the context of GMMs.
2. We quantify measure of optimality for the projections learned in the context of GMMs. We show that InfoNCE can find optimal projections for GMMs with general shared covariances.
3. For the multi-modal setting, we show that the CLIP loss learns linear projections that are a subset of the FFisher-Joptima subspaces for each modality, filtering out noise directions.
1.1 Related work
Contrastive learning and InfoNCE. Contrastive learning is at the heart of state-of-the-art repre- sentation learning methods like SimCLR (6) MoCo K etc. Prior theoretical work on contrastive learning argue that the contrastive objective leads to learning of "general purpose" representations that facilitates better downstream performance and sample complexity. [2] gives the first theoretical analyses of the contrastive loss, showing provable guarantees downstream task with reduced sam- [22] relaxes the augmentation assumptions made in [2] by showing that contrastive learning is equivalent to spectral clustering some appropriately defined data graph. 22 extend this line of work by arguing for the inclusion of the inductive biases of the neural network architecture into the theoretical analyses. [16, (7] extend the setup of [2] to show that the eigenfunctions of the positive pair graph (as defined in [12) are the basis of the set of view-invariant functions. They further show that these representations are equivalent to Kernel PCA with "positive-pair kernel".
ple complexity.
Self-supervised learning. BYOL 10, SwAV 5 SimSiam 7 are some of the most popular methods for self-supervieed representation learning. They are also based on Siamese architecture
2

---

used in contrastive methods, but notably, they do not make use of negatives to prevent "collapsing" solutions. Contrastive methods like SimCLR [6] and CLIP [20] aim to learn representations so that embedding of point is close to its partner while far away from that of any other point. On the contrary, non-contrastive self-supervised learning methods only enforce the former condition and do not explicitly deal with the separation between the embeddings of samples. These methods are empirically motivated by the notion of breaking the symmetry between the encoders in Siamese networks (25). [23 15 [6] propose various techniques for breaking this symmetry and study their training dynamics to understand their role in preventing collapse. Compared to the relevant work on self-supervised learning, we instead focus on the fixed point analysis of the loss functions.
Dimensionality reduction for GMMs. Spectral methods are popularly used for dimensionality reduction in GMMs by analyzing the principal directions of the data covariance. Spectral clustering essentially combine spectral analysis with standard clustering algorithms (e.g. K-Means) in the low dimensional space. Since their error bounds grow as square root of the ambient dimension, the two-step approach leads lower clustering error than in the original space.
algorithms 21 24 17 4
Problem setup
We consider the task of learning "good" representation function for points drawn from data distribution. Informally, such a function should map the data onto lower-dimensional subspace while preserving the class information. For tractability, a - A æ; where A E Raxr, In this setup, we will show that self-supervised methods can learn the optimal mapping for this task. We compare and contrast results for self-supervised methods to well-known results for supervised methods (e.g.. LDA) and spectral methods.
restrict f to the class of linear mappings;
As the basis of our study, we consider data distributions following Gaussian mixture model (GMM). Intuitively, each mixture component in GMM represents class in data and we assume that components have shared covariance.
Definition 2.1. A Shared Covariance Gaussian Mixture Model (SharedGMM) parameterized by {wk, Mkg [}{{(K] is defined the probability distribution F Σne(k] WtN WaNN Σ) where Σx wx = 1, μa are the means and Σ E 5 is the covariance matrix shared by the components.
To complete our data model, we formalize the definition of augmentation. Given point æ. its augmentation E is an independent sample from the mixture distribution with a bias towards the underlying component of æ. The bias implies that is sampled from a component 2, then its augmentation, on average. is more likely to be sampled from the same component 2.
Definition 2.2. For a SharedGMM F parameterized by {wk. Mk: E) we define its Augmentation-enablee Distribution (AeD) F. parameterized by {wks pk: E, E,8/kE(A)
F- = E wk (N E) x (M Σ)) + (1 - (1-5) {) WwN(un Σ) x Ex wwwNWw Σ))
To elucidate, nngmentationeeaable distribution (AeD) returns pair of points (x, .) F in two-step process. We first flip a coin with bias 8. Based on the coin flip, we either sample twice from the same component sample from possibly different components (chosen with probability wk). The bias of the coin 5 controls the correlation between æ and á: 5 = means æ and & are independent samples, while 5 means z and i are independent draws from the same component. The definition ensures that the marginal of x and è are equal to F for any 5.
Spectral versus self seff-supervised methods. Our results corroborate the known advantages of self- supervised methods over spectral methods. Clearly, spectral methods learn linear mappings directly on the data samples
D, whereas, self-supervised methods operate the augmented pairs (x, .) ~ D pair- Consequently, their optimization problems differ in their target objectives:
Esprctral (A;D) = -11 x1 & Lecolfeup CaaeupppA. (A; Dpair)
(A"
R(#
attractive term
regularizer
While spectral objectives find the best-fit subspace explaining maximum data variance, self-supervised objectives are based on the principle of bringing embeddings of similar point closer in conjunction with loss-specific regularizer. The regularizer typically induces separation between distinct points.
3

---

Linear dimensionality reduction (LDR), Framing our task as LDR allows us to establish a mea- sure of comparison between mappings learned by spectral (Aspectral) and self-supervised methods (Aselfspp) For SharedGMM, we denote it's "intra-componen" variance (i.e. variance within component) by Σ and "iitter-ommpnent" variance (i.e. the separation between components) by Ex W& 9 A GMM is said to be well-separated if it has low intra-component variance and high inter-component variance. LDR aims to map a GMM to low dimensional subspace where is well-separated. We evaluate the performance of these mappings by Fisher discriminant [9].
Definition 2.3 (Fisher Discriminant), Suppose a SharedGMM (Def 2.1) parameterized by {wk, M4g kE((]] Then, the fisher discriminant J (AA for mapping A is defined as:
J(A)= TTr "TA"A"AAAAA
A (En
"WHa a))
Remark 2.4. While ((A) is defined in terms of A, its value is solely a function of Col( A), i.e. the column space of A. We hence use the mapping matrix A and projection subspace Col(A) interchangeably in our discussion.
Note that A'EA and (E WaMEE A denote the intra-component and the comer-component variances, respectively, after projection via A. Thus, the Fisher discriminant measures the ratio of inter-component to mntra-componett variances post-projection. A "favorable" mapping A will have high value of the Fisher discriminant J (A),, hence, the Fisher discriminant J serves as the quantitative metric for evaluating the subspaces learned by different frameworks.
Connection to linear discriminant analysis (LDA). One can argue that it is not surprising for self- supervised methods to have better performance than classical spectral methods. The more interesting comparison would be against supervised method, which would help uncover the capabilities of contrastive methods. Therefore, we focus our attention on Linear Discriminant Analysis (LDA) [9]. LDA is supervised LDR method that leverages the class information to learn the subspace where the data distribution conditioned on the class label (i.e. underlying component index) is maximally separated (see Appendix HH). We highlight that assuming AeD (Def. 2.2) is strictly weaker than assuming a labeled dataset. We show that this relaxed condition enough for learning the Fisher subspace and gives similar performance to supervised LDR methods.
3 Fisher subspace
For any target dimension r, the linear map that has the top r eigenvectors of E We (a as its columns maximizes the Fisher discriminant. Similarly, for any mappings A₁ and A2, if Co)(Ar) S Co)(Az), then we have ((A) S J(2)). In other words, J(-) is monotonic in the subspace of the matrices. Based on this, will have the following definition, which is necessary in measuring the optimality of projections learned by different methods.
Definition 3.1 (Fisher Subspace). Given a SharedGMM (Def 2.1 parameterized by {wk, 14, ] its Fisher subspace (9] denoted by SF, is the smallest subspace that achieves the maximum Fisher discriminant, which given by:
Sp Span (E- (B)ME(K)
(1)
We also use unique property of the Fisher subspace [ is the smallest subspace preserving the class posterior probabilities for Gaussian mixtures with shared covariance, which we formalize next.
Lemma 3.2. Let {wk, pk: ]) be a SharedGMM and Pr(z k|2)) be the posterior probability of x being drawn from the component 2. Let SF be the mixture's 's Fisher subspace and Ap be projection matrix such that Coll (A)) Sp. Then for any x ~ F,
Pr(z k k/x)
Pr(z k(AFx) Vk [K]
Crucially, this property implies that projecting GMM onto its Fisher subspace will not result in mode collapse or erroneous mode merging. Moreover, any clustering algorithm operating in the lower dimensional subspace will observe the same class probabilities as in the original space. Owing to these properties of the Fisher subspace, we argue that is the optimal subypace for projection, particularly for the class of shared covariance GMMs. Under the fully labeled SharedGMM setting, one could deduce that the subspace learned by (nulti-class) LDA, SIPA, A, for SharedGMM coincides with Fisher subspace Sp.

---

SsvD
Ssv
Su Sev SEnA
SEDA
SLDA
(a)
(b)
(c)
Figure 1: (a) (Spherical Gaussians) SSVD SI.DA and projection onto Ssv p leads well seperated
GMM. (b) (Non-spherical Gaussians) laarge variance one direction means SSVD & SLDA and projection onto SsvD leads mode collapse. (c) (Shifted non-spherical Gaussians) Sμ + SLDA
3.1 A simple example: Parallel pancakes 17
Spectral methods 17 the standard methods for LDR. They are based the principle of finding the best-fit subspace which is (generally) given by the top singular vectors of data covariance.
<(r) Formally, the r-dimensional SVD subspace (24) SVD for GMM F is described the top left singular vectors of Ex- [=x="" A vanilla spectral dimensionality reduction method hence projects the data points onto the r-dimensional SVD subspace.
[17] proved that for spherical GMMs (i.e. Σ I), SVD subspace is equal the mean subspace Sn Span (pan(u). Moreover, using Eq. ①I we can derive that SLDA = Sμ = SSVD. Hence, projection onto the SVD subspace is the optimal dimensionality reduction method for spherical GMMs.
However, easy to construct examples demanding for a sophisticated method. Spectral methods aim to maximally explain the data, leading to possibly "bad" projections when the variance in certain directions dominates the separation between the means. Figure we we present such an example; consider two component GMM that resembles "parallel pancakes" such that the components are narrow and separated along one direction, and spherical all other directions. The two dimensional SVD subspace for this GMM is given by plane parallel to the pancakes. The Fisher subspace (and equivalently LDA subspace), however, is the plane passing through the means; formally, SsVD & Sp. In fact, the SsvD has the smallest Fisher discriminant among all woo-dimensional subspaces. Finding the optimal subspace for non-spherical GMMs is known to be non-trivial 17.
Contrastive learning for Gaussian mixture models
We analyze solving the dimensionality reduction task for GMMs with two popular self-supervised methods, SimCLR and SimSiam. Both methods build on using augmentation pairs, but differ in their optimization objective (specifically how they regularize). We show that both objectives are able to effectively leverage the augmentations and learn mappings onto the (optimal) Fisher subspace.
4.1 Optimization objective
InfoNCE loss [9 is popularly used in contrastive learning methods (like SimCLR). It learns representations that pull data points and their augmentations close together while pushing them away from other points in the embedding space. We define the InfoNCE objective for linear mappings as,
Linfo (a
(A"
20))" (A"x)'
A
log E #NF
exp ((A" xx A²))))
(2)
#~F
where (x, .)) F is a sample from augmentatiooneenable distirbution (Def 2.2 such that F and ä ~ F' are independent draws from the mixture. The attractive term (i.c. the first term) keeps and & close by maximizing their dot product, while regularization term penalizes proximity to random samples. In other terms, the first term increases inter- component variance while the second term decreases data variance and hence intra- componenm variance.
SimSiam loss [ is another popular self supervupe loss without the negatives.
A AT æ | AT x
AT |ATE
E
StopGrad
(
5

---

Unlike InfoNCE, Simsiam only operates on augmentations (from F) and doesn't t use negatives. The loss parameterized by two matrices; the mapping A and the "prediction head" Ap, which is not utilized for projection. Observe that Simsiam might be prone to collapsing solution, i.e., mapping all points onto the same vector. Prior work [2] 15 26 has argued that Ap, which makes the optimization asymmetric, and StopGrad, which zeros out the gradients flowing through it, play crucial role together in the guiding the training dynamics, preventing the occurrence of collapse. Our goal is to analyze the fixed point of this loss function. For tractability, we introduce some simplifying assumptions and examine a modified loss:
Lsiam ()
B( [(A a) ( a" + + EEZE [|A">>)] [13]
We remove the StopGrad operation and set the prediction head (i.e. Ap) to I. We also trade the normalization term with regularization term weighted by &. The attractive term in SimSiam behaves similar to InfoNCE loss, while the regularization term penalizes the norm of the projected points. The loss scales linearly with || ||² therefore, we impose norm constraint when optimizing for A.
We have already established that maximizing total variance on ils own in the projected space is not the preferable strategy for dimensionality reduction. In the presence of augmentation (and negatives), self- supervised objectives could take a finer-grane grained approach by maximizing inter-component variance and keeping iomponet variance low, simultaneously. Next, will formalize claim that self-supervised learning is a good proxy for fully supervised techniques (e.g.. LDA).
4.2
Main theorem
Nex, we prove that InfoNCE and (simplified) Simsiam can find the Fisher-optimal projection matrix for the class of SharedGMMs.
Theorem 4.1. Suppose F is SharedGMM parameterized by {wk, pk: D]kE[K] and F is its Augmentation-enabled Distribution with bias 5. Let Sp be the Fisher subspace (Eqn 7 of F. Denote Amin as the minimum non-zero eigenvalue of Σー= (Ex (attsn 2 Σ- and
A argmin (A),
Asiam argmin Csiam(A)
AgR
AERdx², ||A||2S1
Then for some
2 K, Stnto Col(AAtt)), Ssiam Col( we have:
Sinfo C Sp. 1 1, then Stafo SS
Ssiam C Sp. 1f0 < <
Bduii then Ssiam = Sr
I+hmin.
Distribution provably sufficient for learning the Fisher subspace for SharedGMMs matching the performance of supervised LDDA, The result implies that class labels which are typically assumed by supervised LDR methods like LDA, are not necessary.
Remark 4.2. Access to Augmentation-enall
The proof of Theorem 4.1 can be found in Appendices C and D For population setting, the fact that
self-supervised methods learn mapping onto a subset of the Fisher space implies that the projected points do not capture any noise and only contain (a subset of) the useful features. Note that spectral methods do not have guarantees of this form. The theorem also states if we have perfect augmentations for InfoNCE loss, i.e. 8 1, we cover all directions in the Fisher space and learn it exactly. The equivalence is true for SimSiam loss when the regularization coefficient & upper bounded appropriately. On related front, this precludes learning of the collapsing solution for SimSiam loss, verifying the claims in prior work on a simplified version of the loss function. Most importantly, self-supervised methods in question learn the same subypace as supervised dimensionality reduction methods like LDA which needs the knowledge of underlying components for all the samples.
Technical difficulties. Although we do not provide a proof for Sinto SF when 0- < 8 < 1, we
conjecture that it is true, for which we include an empirical discussion in Section 6 Figure 4a suggests that Sinoo = Sp even for 5 < 1, analysis of which we leave as future work. Additionally, we do not have an exact characterization of InfoNCE solution; we characterize just the column space of Anio Our empirical results show that the mapped points are well-separated compared to a projection onto the Fisher subspace (see Fig 2d.. Furthermore, while we prove that both contrastive and non-contrastive objectives learn the same subspace, further investigation is required for their direct comparison. In Appendix F. we provide an example where Atnto strictly better than A ASiSSiam (for
< K). We also present empirical study the effect of r (see Figure 4c

---

5 Multi-modal Gaussian mixture models
Previously, we considered a setup where point and its augmentation follow the same distribution. It a natural assumption for methods like SimCLR and SimSiam where augmentations of points (i.e. images) are defined by transformations (e.g., cropping, color jittering) on the image. This assumption may not hold in general. Text embeddings models like DPR [18] or image-text embedding models like CLIP [20] consider input samples pair of points following (possibly) different distribution.
For instance, each sample could a pair of an image with its corresponding caption (for CLIP) or a search query with relevant document that answers the query (for DPR). The objective in this multi- modal setup is to learn a joint representation space for both modalities. These joint representations can be used downstream for finding similarities between data points belonging to different ambient spaces, and are surprisingly competitive with fully supervised representations [20). Our goal is to theoretically analyze the representations learned by such model, specifically the CLIP model. Next, we define CLIP-GMM to capture multi-modal data in a theory-friende setting.
Definition 5.1. (CLIP-GMM) A CLIP Gaussian mixture (CLIP-GMM) is defined as the probability distribution Felip = ΣkE[K] WyN[H Σv) x (H Er) where wy are the mixture weights, (pv,A By and { (kE[K) Ex are the parameters for the two coordinate spaces.
CLIP-GMM is a mixture of product distribution over Gaussians. To elaborate, sampling pair (ay, aT) ~ Felip is two step process. We first sample an underlying component index, and then draw independent samples from the component with the same index in the respective coordinate space. Felip can be used to define marginal distribution over each coordinate space, for instance, Fv = Lke(k] WKN(uvv Ev).
5.1 Optimization objective
Following CLIP [20], we want to learn different representation functions for each modality. We let these functions be linear mappings. i.e., Av [r Ar E Rd2xr, Recall that optimal mapping matrices for each coordinate space would have their column space equal to that of Fisher subspace for the marginal distributions Fv, Fr. Concretely, let the representation of a sample (æy,, Tr) be given by (A) A where Av, Ar are the mappings. Then, we define CLIP InfoNCE loss (20 as,
Lelip-
E
(A ATæv]
E
log
E lav~FV
|exp ((AT AT=v)) [ATxv)
(æy (
Each coordinate space serves as augmentation for the other. Similar to InfoNCE, CLIP InfoNCE attracts embeddings of æy and ær via the first term, while regularizing with the Sum-Exp term.
5.2 Results
Our main result for CLIP-GMM states that we can learn a subset of Fisher subspace for the constituent modalities by minimizing the CLIP InfoNCE loss. We state our results in the following theorem. Theorem 5.2. Suppose {ur,, pV.,, (WK,,V,ss KT,K, Ey, ET Eeeke is CLIP-GMM Let the Fisher subspace of Fy be Sv, and Fr be Sr.r. Denote Av, Ar. as the optimal solution of the CLIP InfoNCE loss,
AV,Aj
argmin
Celip Ar)
Aye xr, A
For any r 2 K, let Sv.dip Col(A;) and Sr,elip Col(AA)) Then, Sv,elip S Sy.f and Sp.elip ST.F..
The proof of Theorem 5.2 can be found in Appendix E In simple words, CLIP InfoNCE learns the subset of the Fisher subspace instead of the exact space, which is weaker than the single-modal setting. This is due to the fact that the means in the two spaces can vary arbitrarily. Hence, one can choose means and covariances adversarially such that particular directions in the Fisher subspace of either Fv or Fr does not contribute to the componetm distance and are not learned by the CLIP
InfoNCE. For certain special configuration of model parameters, such as ΣT Pr. = ET² μv.k. we can achieve ST.elip = ST. and Sv.elip = Sv,F. The result still shows that self-supervised learning for multi-modal data is better than non-supereieed methods. Accessing augmentations are strictly weaker than the knowledge of the labels, and they also naturally occur in the multi-modal setting. The fact that contrastive losses learn subset of the optimal subspace verifies their capabilities.
7

---

ARB
AMI
(a) Noise in augmentation
(b) Variance 1 SF
ARI
614
65
Sodan
Sindaw
(X)
(o (1))
(c) Dimension of projected space
(d) Variance within Sp
Figure 2: We empirically validate our theoretical findings for four main settings. (a) InfoNCE is robust noise in augmentations for various values of 6. (b) InfoNCE (and SimSiam) are invariant to variance orthogonal to Fisher subspace. (c) InfoNCE outperforms spectral methods for every mapping dimension. (d) InfoNCE learns good scaling within the Fisher subspace
6 Experiments
We validate our theoretical findings with experiments on synthetic and real data. For the synthetic data, study the effect of noise 8 in Augmentation-emablle Distribution, rank of the projection matrix and condition number of covariance matrix on learned representations. For the real data experiments, compare self-supervised methods against baselines for clustering CIFAR 100 dataset on low dimensional subspaces using K-Means.
6.1 Setup
We evaluate different linear dimensionality reduction methods for SharedGMMss. We adopt two-step process; the first step is dimensionality reduction with the target method, and the second step is clustering with out-of-the-box clustering algorithm (i.e. K-Means ). We compare the methods based on their clustering performance in the second step using well-known metrics.
we use Adjusted Rank Index (AR/) and Adjusted Mutual Information (AM/) to measure the quality clustering algorithms. We give the ground truth number of clusters to K-Means as input. ARI and AMI both vary between O and 1, with 0 corresponding to random clustering and corresponding to perfect clustering.
Metrics. Following prior work 14
Data generation. For synthetic experiments we consider K = 10 with equally likely component mixtures. For the CIFAR experiments, randomly sample K 20 classes out of 100. See SuppI for details.
Methods. We consider learning mapping matrix A using gradient descent with respect InfoNCE and SimSiam loss and we compare them againts baselines. Ambient is clustering in ambient dimension. Random projects onto random r-dimensional subspace. Optimal projects onto top directions in Fisher subspace. PCA projects onto r-dimensional Ssv LDA finds projection that maximizes inter-class variance with respect intra-class variance (Optimal in the synthetic case).
Synthetic experiments
We observe that InfoNCE often performs better than Optimal, projection onto the Fisher subspace. interpret that InfoNCE loss learns to scale within the subspace leading to better clustering performance (see App G.2 for orthonormalized plots of InfoNCT/SSiSian)) Moreover, SimSiam shows sub-par performance compared to InfoNCE. We believe this is due the difficulty in optimizing the SimSiam objective (projected gradient descent) compared to InfoNCE.

---

Table 1: Clustering performance of linear dimensionality reduction methods on CIFARI00:
Bolded and underlined values indicate the best and second- best scores for each column. We report clustering performance for K-Means on r = 5, 10, 15, 19-dimensional subspaces using the linear mappings learned by 6 different methods. We measure the clustering performance using ARI and AMI, which reported as pairs, in the format A ART AMI. InfoNCE. and L.DA outperform the rest by significant margins in all mapping dimensions. LDA shows the best performance in terms of ARI, while InfoNCE achieves higher AMI values across the board.
Mapping Dimension
Method
5dim
dim
15
dim
30 dim
Random
0.014751
0.04694
0.01 0000244
0.01894 00.05796
0.02446
0,07188
0.02503 (0,07856
PCA
0.03360
0.09254
0.032921
0.09509
0.03435
00009427
0.03154
0.09269
0.03361 10,09562
Ambient
0.03160
0.09481
0,031601
0.09481
0.03160 0.09488
0,03160
0.09481
0.03160
10.09481
SimSiam
0.02870
0.09581
0.02732
0.091 14
0.03707 10. 10830
0.02868
10. 10.09498
0.03562
10.10540
InfoNCE
0,05065|
0.12730
0,05138
0.12801
0.05402 00.22835
0.05285
0..12947
0,05182 0.12258
LDA
0.05067
0.12660
0.05489|
0.18728
0.06352
| 00.19399
0.05970
0.11933
0.05970
0011
Effeet of noise in augmentations. We vary the noise parameter for the AeD with everything else held constant. We can see in Fig 4aght InfoNCE is robust to the variation in 6.
Effect of variance orthogonal to Sr. Increasing the variance orthogonal to the Fisher subspace is equivalent to making the pancakes in Fig flatter. We quantify "flatness" as miin mi) / (E). Fig shows that InfoNCE is (almost) invariant "flatness". while PCA and Ambient fail when the
variance orthogonal to Fisher subspace is large (i.e. Twin (ai) (2) (2i is small).
Effect of projection dimension. We vary the dimension of the lower dimensional space that we aim to learn. InfoNCE's performance increases with increasing r and finally plateaus.
Effect of variance within Sp. We show that InfoNCE learns a "good" scaling within the Fisher subspace. We set the dimension of ambient space equal to the dimension of mean subspace (i.e. d = K 1). We select K/2 orthogonal directions and increase the variance in the subspace spanned by these direction by a factor of o max ( (Σ). Since none of the baselines learn scaling in the subspace, they perform almost the same, while InfoNCE outperforms all the baselines.
6.3 Real-data experiments on CIFAR-100
We consider images from CIFAR-I00 as inputs instead of generating data from synthetic GMM. We consider 20 fine-labeled classes (10K images) where each class corresponds to single component. We convert images into 256-dimensional vector by subsampling, grayscaling, mean scaling. Images belonging to the same class will be the augmentations of each other. We still consider a linear model and hence have the same baselines. We measure the clustering performance using ARI and AMI on mapping to r 5, 10, 15, 19-dimensional subspaces in Table [ Surprisingly, InfoNCE could still outperform LDA in certain metrics even in the real data setting where naturally, our data distribution assumptions and augmentation will no longer hold. LDA has the best performance for all dimensions in terms of ARI (first value in each column) which indicates that it excels at maintaining local groupings. On the other hand, InfoNCE achieves larger AMI values underlining that contrastive learning could be better at preserving overall class distributions.
6.4 Real-data experiments on ImageNet
We consider images from ImageNet as inputs instead of generating data from a synthetic GMM. We consider 20 labe-le classes (10K images) where each class corresponds to single component, and images belonging to the same class serve as the augmentations. Since image classification highly non-linear, we construct a setup to map them to a linearly seperable space. For this we represent each image using Res 's last layer representations (i..e, 2048 dimensional vectors). We measure the clustering performance using ARI and AMI on mapping to r 5, 10, 15, 19-dimensional subspaces in Table 2 We can see that InfoNCE consistently outperforms LDA in this setting.

---

Table 2: Clustering performance linear dimensionality reduction methods on ImageNet: Bolded and underlined values indicate the best and second best scores for each column. We report clustering performance for K-Means on r = 5, 10, 15, 19-dimensional subspaces using the linear mappings learned by 6 different methods. We measure the clustering performance using ARI and AMI, which reported as pairs, in the format ART AMI. InfoNCE. and L.DA outperform the rest by significant margins in all mapping dimensions. LDA shows the best performance in terms of ARI, while InfoNCE achieves higher AMI values across the board.
Mapping Dimension
Method
5
dim
15m
dim
30 dim
Random
0.03002
0.08296
0,072311 10. 14005
0.13140 [ 20,23009
0.14405
10,23363
0.26177 00,36685
PCA
0.23408
00.46447
0.39843 0.591
0,49903
10.65898
0,50954
10.66559
0.56355 10.70163
Ambient
(148641
0.67233
0.486411
0.67233
0.48641
00.62233
0.48641
0..6733
0.48641
10.67234
SimSiam
0.37581
0.58233
0,56084
0,70246
0,58705 10. 0.72227
0.60259
0.731599
0.62970
00.74476
InfoNCE
0.84451
0.88831
0.98182 0,98115
0.92963
0.97390
0,99621 0.99579
0.93317
0.97721
LDA
0,471121 0.69828
0.66967
0.83895
0.82550
0.891003
0.94744
0,94990
0.94745
10.94990
7 Conclusion and limitations
We study contrastive learning in the classical setting of linear dimensionality reduction for GMMs for which we define generalized notion of imperfect augmentations. Our main result underlines that the contrastive methods learn the Fisher-optimal subspace for the class of shared-covariance GMMs; going beyond the capabilities of unsupervised methods and matching that of fully supervised strategies. Our work particularly focuses on linear mappings, and it is an important open problem to theoretically verify performance of contrastive methods when projectors are non-linear, which is usually the case in practice. Similarly, we acknowledge that our results are not immediately generalizable for data distributions beyond GMMs, which we will investigate as future work.
References
[1] Dimitris Achlioptas and Frank McSherry. On spectral learning of mixtures of distributions. In International Conference on Computational Learning Theory, pages 458-469. Springer, 2005.
[2] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: pttp:/902..92290 2019.
[3] Alice Bizeul, Bernhard Schölkopf, and Carl Allen. A probabilistic model behind self-supervised learning. arXiv preprint 2002::::0020019999 2024.
[4] S Charles Brubaker and Santosh S Vempala. Isotropic pea and invar-invvrrantt clustering. Building Bridges: Between Mathematics and Computer Science, pages 241-281, 2008
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural 2020.
information processing systems,
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020. URL https::/ppooetta httpp::/prroeeedings .mlr......r................... mlr press/ http//v1///////// chen20j .htll
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15750-15758, 2021.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint XXXx:2003.04297 2020.
[9] Keinosuke Fukunaga. Introduction statistical pattern recognition. Elsevier, 2013.
10

---

[10] Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch. Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- self-supervised learning. Advances in neural information processing systems, 33:21271-21124,, 2020.
[11] Jeff Z HaoChen and Tengyu Ma. theoretical study of inductive biases in contrastive learning. arXiv preprini arXiv:2211 1.191144,9,,,. 2022.
[12] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 14:5000-55111 2021.
[13] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint 1rXiv:2112.09) 2021.
[14] Heinrich Jiang, Jennifer Jang, and Jakub Lacki. Faster dbscan via subsampled similarity queries. Advances in Neural Information Processing Systems, 33:22407-22419, 2020.
[15] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09344, 2021.
[16] Daniel D Johnson, Ayoub El Hanchi, and Chris Maddison. Contrastive learning can find an optimal basis for approximately view-invariant functions. arXiv preprint arXiv:2210.0168 083, 2022.
[17] Ravindran Kannan, Santosh Vempala, et al. Spectral algorithms. Foundations and Trends@ in Theoretical Computer Science, 4(3 ((3-4):157-288. 2009.
[18) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020.
[19] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1rXiv:1807.03748, 2018
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[21] Arora Sanjeev and Ravi Kannan. Learning mixtures of arbitrary gaussians. In Proceedings of the Thirly-Third Annual ACM Symposium Theory of Compuling, STOC '01, page 247-257, New York, NY, USA, 2001. Association for Computing Machinery. ISBN 1881133499, doi:
10.1 00.1145/807552.38880 URL stttps:///ol: org/ 10. 1145/3807555 380808
[22] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning, pages 19250-19286. PMLR, 2022.
[23] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021.
[24] Santosh Vempala and Grant Wang. A spectral algorithm for learning mixture models. Journal of Computer and System Sciences, 68(4):841-860 2004.
[25] Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16570-16579, 2022.
[26) Zixin Wen and Yuanzhi Li. The mechanism of prediction head in non-contrastive seff-supervised learning. Advances in Neural Information Processing Systems, 35:24794-2488 2022.
11

---

[27] Runtian Zhai, Bingbin Liu, Andrej Risteski, Zico Kolter, and Pradeep Ravikumar, Understand- ing augmentation- based self-supervised representation learning via rkhs approximation and regression. arXiv preprint arXiv:2306.007780 2023.
12

---

Appendix
A
Additional Lemmas
Lemma A.1. The function given by [( = log(E, exp(c" x;)) where {ai}, c E I& is strictly convex in E Span((x;)).
Proof. Using the definition of strict convexity we need to prove that:
f((a;); + (1 - -))22) <)(((ai):c)) +1 2)(((=));ea)
when 6₁ # 02 Taking exp(.) on both sides
exp(S((aa) (Ae, + (1- x)e2))) (x)p(A(((a):cc)))) ())((()))))
Simplifying the LHS we have:
exp(f((z:): (Ac1 + (1- A)c2)))
Σ exp((Acc (1 2², x;)
Σ exp(>ef=x)) 2.) exp((1 2)05 wi)
Using Holders inequality C 24yh S CE |x (2, bm where 1 + = 1) with
} and
= T-X we have
Σ exp(AAcf (oo(AeTT;)))exe exp((((((1 - 2)05 mi))
exp Ae x
exp (1 --)c 2 (1-1)
A
A
exp(cfxx1))
exp(c xz)
(xx(AA((()):c))))x)p) exp((1 ();((=)):)2))
Note that Holders equality holds only when of x; x; for all i. For E Span(æ;), this equality holds only when C₁ C2. Hence, for e₁ # C2 have strict inequality i.c.
exp(AcT xx)) exp((1 x)) xx))
AcT expl
(1- 2)c x; (11-)) exp 1-
=xxp(\[((x.):ci)) exp (1)))))))()
This gives strict convexity. Hence proved.
13

---

B Main Proposition
In this section, state and prove key proposition which is used to prove main result (Thm 4.1 Proposition B.1. Suppose parameterized by {wxx psg TRE(K) be a spherical gaussian mixture model and F be its augmenntation-enabled gaussian mixture with bias ô 2.2 Let SF be the fisher subspace (Eqn) 1 of F and A be the optimal solution of the InfoNCE loss (Eqn
A argmin ((A)
AcRaxr
Then given r 2 K, Col(A)) C. Sp. Moreover if = 1, then Col(A") = Sp.
We prove the proposition in two parts. In the first part we prove that Col(A") S Sp for any > 0. In the second part we prove that 8 1, then Col(A') Sp.
B.1 Column space of A is subset of Fischer Subspace We prove that: Col(A*) S Sp when > 0
Proof.
(A xxx)²
(A ( =x AAT xx By,
where B AAT, i.e. B positive semi-definite (PSD) matrix of rank r. We substitute into InfoNCE loss to get:
=
Ex xx B + Eg [log((2 [exp(x B
relax the rank-constraint on throughout the proof. We show that rank of our optimal solution 8 < K which satisfies the rank constraint implicitly (as K r).
Note The above loss function strictly convex in (using Lemma A.1) and the minimization of is over convex set (i.e. set of B E $4))
Let B" be the optimal solution. Denote the eigendecomposition of B as UAUT, where > 0 (as
B 0) and U unitary matrix. Equivalently:
B" B" ===>MR Auuuy
(4)
where u, are columns of U. Consider the indices where the eigenvalue A >> 0 as I. The solution (ppn((u)ke))
A' hence [v),u,] iez' We aim show that Col(A")
The condition Span(u;Jiez follows because there can't be more than K orthogonal vectors (i.e. columns of U) in a subspace of dimension K (as there are only K means spanning the subspace).
Span [ implicitly implies that rank of B < K. This
Suppose the condition is not true. Then there exists a unit vector v, such that v μs = for all k E []] and v" u, for some i where X; 0.
We construct a new matrix U, whose columns are reflection of U through the plane with normal vector a given by R = I 200". The reflection matrix is defined such that R44 R py pk. Define from the constructed Ú.
Ú RU (I 2vv")U
B ÜA RB* R"
U U is still unitary matrix (product of unitary matrices), is identical to U in Span [(E[K]
-TU-URRTR = ----
0 ps = U² Re ps - ps
Up = URP = Up
14

---

The first term in the loss L at B can be simplified as
Ex, de (x' B'
Ex del lxvt" B3]
(L at (xtte ". |, )²)
Σ we pu (ōfus + (1 (1155( Σ wyHlz)) B²)
Σ Wapuii + +(1 (1-6)) 88) Σ wpply) w. B²)
J
This is where we use the fact that the augmentation is generated from the augmentation oracle. We can now see that
Ex, a ax" Bi
WKMes T + (1-1)) (1-6)( wkMla( Σ Ewjpy)? BB)
(6 Σ WaMaitk T +11- (1--)) 5)( Σ wkHkk(( Σ " RBR")
(6 R papER
(1
wy R pA [ R" 1T)"", B²
6 WHHLRH T + (1- - 6)( Σ WklHk)( > wT [w, B²)
= Ex [x B
Now see analyze the second term in L where we prove
Ex [log [exp(x 7 B=)])] = Ex (og (Bxxxp((T)))))))) B'a)])
For this we show that the random variables x B 3 and x" Bi have identical distribution. We first simplify x" B* below.
x" Bi:
=x" "..
=(x 1b= + p.) T RB* ( μs + pm)
=((x -4=) R + μ R B" (R(ā us) + Rus)
=((x -4:)) ) TR. + μ B" R(i -p2) +
where R= n² (I 200 11 22 v) n2. Now we prove that their distributions
are identical.
15

---

Pr(x Bä 1 c)
=Pr(((x uz)1 R BB pæ) + m2) S
-Pr(((0-
mz) -4)l
+ n! B (( ps) + ps) cc
=Pr(xBB* sc)
We the fact that (x (I=) 2ve")
(I-2((() is identically distributed (æ m4) i.e. I).
Now since x= B" क. xx" Be are identically distributed, second term in the loss are also identically distributed and have the same expectation i.e.
Eu [log (Ea (ex(pp(( B"=)])] = (og (exp(x' Bε)))
This proves that L is identical for B and B'. But since loss is strictly convex have L( B (C(BB) L(B)) L(B"). This contradicts the fact that B is optimal in Sd. Hence proved.
B.2 Column space of A is equal to Fischer Subspace
We prove that: Col(A*) SF when 5 =
Proof. Consider the eigendecoopoostion for the optimal solution B" (Eq 5). Suppose there is direction Span (()As))) which is not in (o;an((u
Without loss of generality assume T u. vi I (if not then project onto the null space and re-normalize). Now since X;- = Vi & I. we can rotate the eigenvectors [uyligz with unitary matrix such that u for some This operation doesn't 't change B"
That implies there exists u, such that u, E Span {Me and X3 = 0. We now show that
dc
0% B-B- <0
Hence X, > for optimal B"
First consider the derivative of first term of L with Xy
OE xx Bi
OB weptikk Ox,"
0g
WaMaaa u u,
Σ wi (Hit uzy)
Σ
wxa
k
where ak = 11 uj. Since u, E Span ((an((1x)) }x) that implies there exists a non-zero 0g. Hence we have that 00 lxx" Ba)
0 for any B + 0.
8A,
16

---

For the second term in the loss we have
GE log En (exp(2) x Bi))) 0 Ba (exp(x* _ B On"
BA;
En
Es (exp(2 x=" B
Ba (ex _ B (at U (1 UT)!
RFag
Es cepp(11 (x B
Eg eexp(x' (x: Bi) ( Uj)]
En
x ui)
Es /exp(x x B&)
exp(x: B
=En
(x T Uj EB
x² Uj
E (exp(xx T Bí)
=Ex
(x" BB g x, B) us
where define g æ., B) as:
( B) A exp(x" B
Eg eep(xx B
with Es (á: B)] 1. Also define x uz(ufx and ø
(I uI- Hence 21+
Notice that since X = 0 for B", we have
g v. B²) = g x. B²) = | z. B*)
We evaluate the inner term in the above expression at B B
Eg g(ă; x B" )²" us
w.y EE (par)) 1)
g(0; x, B o") Us
Σ WiBBaMMHH.l g B²) (l);,,B')*" B")="u
We look at each of the expectations individually
Ex 1)
g(0: B') u.
= =EanN(m. 1)
g(0;,, B" (ax z Euy))
-Eg~Nimc
g(,; B" BBaaa
E
g (')
=ay En- g(0;2, B*)
a.:i4(x: B")

---

where ž the noise in z and (x; B²) (a-N(r g(ā: x, B²) We use the property that $1
independent of g( æ, B*) g( x. B3) to show that its equal to 0.
Ni (0 B B))
-Ra-N(uel I)
g00 x. B*)
EE-Niu.)) z!
(0 B*)
* *0 0
Hence we have
F g x. Ba) uj
=
Σ
an le (z:; B*)
We have Ee wxhx(xx) = -
Σ wx B)
Σ W.
g( xc, B)
explx:= B Es (xp(xx BTa)
We Ba
exp(x B
Es
Ex(exp(x" xx BT)
While g(ē; æ, B), the weight gives point by (w;; B) can be interpreted weight æ gives to cluster & (i.e. expectation of over cluster).
Going back to the expression at B B
E (x" U Bir g( ar B" Uj
EEp
Σ Whagha B") x T uu)
(x T uu )1 B
Σ
Wkaa Ezg
(hx B")
Σ
Wkakk Σ
Wk Wer
(o (a)
k
18

---

h inherits the property (x; B*) = (2) B²) from g (this is because A 0 in B*). Evaluating the inner expression we have
(xx u2)) BA)
E 1)
(aw + z ue B¹)
- Bee Bee
he B*)
+ +IE
(2f 2ffu)ha B*)
=a Bee
ax SKck
where z is the noise æ. We use the property that 21 is independent of hi (x: B²) ha B*) to show that its equal to O.
Σ wwFW. k
Σ wh BinNi
(a; B*)
E Ex-NiHu
Σ wkhh (2; (x;
since Ex wagha(;; B*) =1
Define matrix F E RKXK Fig Süg and W (01, "2....,ax]""
diag(wi, w2,--., and
Em
(x u BB
g( xr B") ui
Σ WKGk
Wk W..
(x (uTu uj)he( B*)
WKOkk
ak fw.
WWWF = WWW.)WWFWWWWWW
Eigenvalues of FWF W equal to eigenvalues of FW (since this is similarity transform).
F
And if is eigenvector of FW, then /We v is the eigenvector for /W F F.W.
Since F has expectation of exponentials as its entries i.e. F., > 0 and W > from definition.
Hence we have that every entry of FW is strictly greater than O. The Perron-Frobenius eigenvalue given by
min ((FW) by ST max Σ (FW)), iin
j
But we have that for every i the sum is equal to 1.
[(FW) (().....)) = Ebguy =1
Hence r 1 and the Perron vector is simply = [1,1.... 1] E R Hence we show that eigenvector
with largest cigenvalue (i.c. 1) for / WWF. W is WW. Hence we have that
19

---

WWFWWWVWWW < 1
Σ
The equality only holds when Span (1, i.e. all as equal. This is not true (since Σσ a2 and there exists a nonzero ag). Hence
Eg
(x: u U2 g a. B') uj
Σ
wyak
The loss for the whole term is
0C
O xx" B
0A B=B-
0 Blo Exg (exp(ax Bi))
0dg
Σ wxaf + Σ wxay
Hence proved.
C Proof for InfoNCE loss
In this section, we use Proposition B.1 to prove our main theorem. We recommend the reader to first go through Proposition B.1 and it's proof (in Appendix B) to understand the proof for the main theorem.
Proof.
C = -x
Bt] + Ex
(Iog [exp((" Bi)))]
where B = AAT, i.e. B is positive semi-definite (PSD) matrix of rank r. The above step follows from Proposition B.1 We relax the rank-constraint on B. We finally show that rank of B" i.e. the optimal solution < K which satisfies the above condition as K <
Now consider change of variables given by x' = Σ 2z. This implies that x' now fol- lows GMM with means given by {Σ- p and isotropic covariance I (as B[x'x"] - Σ- Σ - T I).
The loss hence be written as
C En a [x=TT B
+ Ew (2g (exp(x & E BE++)))
Now let = - BEB B' also a PSD matrix. Hence loss be written
C = Eg ar [x" B'a') + Eg (log(Ear [exp(x a B'ž')])]
Let the optimal 3 be denoted by E. Arura and let Z be set of indices with X; > 0 (note A; 2 Ovi). But from Proposition B. we know that, where Span (u: Jizz = (pan( where (1)) is the set of means.
20

---

For an invertible E and u E
pix we have
Span(u;Jiez = Span
Span liez / 1 p
Span | duiiiEz = Span E p1x)
Substituting and ="= Σ B² Σー we get:
B" 3"=²0 Σ A,
(V
Hence the optimal solution in the original space A
VA 2n, ver
We proved that
Span E 10-iu)eer uujie Span ppantE-paheejyy This implies that column space of Col(A*)
Span Σ uiiiiz = SpantE-1 Hence proved.
D Proof for SimSiam Loss
In this section, use the methodology in part of Proposition B.I to prove the theorem. First we use the strict convexity of the loss to show that the solution lies in the fisher subspace. Afterwards, by differentiating the loss function w.r.t. any direction in the fisher subspace, we show that all directions in the subspace should be learnt assuming sufficient capacity.
Proof. We write the objective by computing the expectations as
(As(A)=
-
(A TTTT'AA AT
+ E 2m
||A x:
(2,
F
(AAT )
+ E mm
E
(AA ax. T
(a.a)~
-AA
(x (a F
+ EIATT
xx
(AAA
WEMLHT + +EAAAT, Σ WaMtiu E)
(BB +E(BEE)
())
(E' (-0 +()E -iME - +EI)
(B', (-8 +
where we have B = AAT, M = =Wwuuaul B' -'= and M' = 3-1ME 1, Note
that the loss is linear in B (and B'). Since we restrict ourselves to || ||2 S 1, it is also true that ||B||2 1.
We now show that the optimal B' has column space (and row space, since symmetric) lying in column space (and row space) of M'. Let B" be the optimal solution. Denote the eigendecomposition of B as UAUT, where A = (as B 2 0) and U is unitary matrix. Equivalently:
B' = Σ N
(5)
21

---

where are columns of U. Suppose the condition is not true. Then there exists unit vector such that M'v 0 and B'v # O.
We construct new matrix U, whose columns are projection of U onto the plane with normal vector given by R = I vv" The projection matrix is defined such that RM M'. Define B from the constructed U.
U RU (1 - (I-vo" UAUT = -RB'R Rd
Now through some algebra we see that loss at B is less than at B and hence B can't be optimal.
(B.(-6+)) +EI)
(-* M'R & R)
=B",, (-)++))) + R- R-1)) ((+(((RRRI))) = =B", (-5 + M + +E(B", (R" R-RI
Now show that (B", (R' R I)) and since >>> 0, loss at B is less than B". The fact is intuitively clear and aa formal proof is as follows:
tr (RB'R") = Σ A Ru: Σ ( (B',
The strict inequality is due to fact that B'v o O, i.c. there exists a i with A; >> 0 and v²uu * 0 (and a projection matrix.
hence Ru,
<
u. 2. |u, s ||u, [32 true generally because
Hence we showed that column space of B" (i.e., optimal B') lies in column space of M'. Now show that it spans the whole column space. Suppose not. Let B" be the optimal solution with decomposition with notation as used above. Then WLOG there exists u, which E Span MM' and (if it doesn't i exist can rotate the uj 's with 0 eigenvalues that u, aligns in the subspace). We take derivative w.r.t. X and show that it's negative.
0Lss Oi, B'=B"
Ø(B', (-6+G)M +5T)
ON; =u ((-5 + M) +
=(-6+ E)u? M'u; +
f < be M's I4w" M'' for all directions v in (pan(M) (as u, E Span(M') then 0L2s is 8x,
Bo'M'e Ito M'v is monotonic oJ M'n. The minimum value of TJ M'v is the smallest non-zero eigenvalue M' denoted by Amin- Hence if <
Axmin we are good. I+Amin
Now showed that 3" (i.e., optimal B') spans the complete column space of M', Hence using the facts B' Σ 3 and A A, we can argue that for the optimal value A denoted by A', Col(At) = Sp.
E Proof for CLIP Loss
In this section, we use the first part Proposition B. to prove the theorem, i.e. we use the proof of Col(A") C SF.
22

---

Proof. We write the objective by substituting the functional form of as:
C EE,, n. [o]] A. A.A x..
+ +En [log( Bir [exp(a? a.)))
= - ,2 [ Bæ.) + Em, [log(Ea (exp(xt Bū.)])
where B - A.A. We can further do change of variable by ai E," and r.y = EV a.. The means are now Pi, = Σ, f and H E. pe,k- Define B' = Σ BEJ. Now xi and ø have components with covariance being isotropic. Now we have
L E,,, [ Br.,) Ez, log(Es, (exp( ai B
Eat a². [ B' x Ea; [Eo (exp a /T B' Bl
We argue that optimal B has its row space equal to (pan( and its column space equal to ) We present the argument for column space (row space argument follows similarly).
Suppose there exists unit vector such that v² ft,x 0 for all & E [K]. 7 B' * 0. Then we can define new matrix B' RB ( 2ve" B B'' where
-- 2vv" is reflection matrix. Following arguments from Proposition B.1 we can prove that x; B' é' is identically distributed to 2. B' Hence using this we can show that L(B")) L(B'). Then by the strict convexity of the L(B" (L(B))
loss function L we have that C( B +'', Hence proved.
L(B"). Hence B can't be optimal.
2
Representation collapse in non-contrastive learning
Consider two component GMM in d dimensional ambient space. The means of the components lie on the and y axis (i.e. the first two dimensions), equidistant from the origin. Both components have isotropic covariance. We plot the first two dimensions below in Figure 3 Note that for this setup
Figure 3: The means GMM components the first two dimensions.
the rank of the fisher subspace two (i.e., K = 2) and the fisher subspace is the x plane.
Consider learning a mapping matrix onto a one-dimensiona subspace (i.e. < K). InfoNCE type contrastive objectives would learn the subspace given by the pink line (i.e. gaussian would be projected onto the pink line and hence would be well seperated). This is stated without proof, but know that InfoNCE loss would learn a subspace lying in the fisher subspace. And through some basic algebra convince ourselves that the solution would be the pink like (i.c. the line + y 0).
23

---

ARR
ARR
ARI
Fuzal (aaa
AMM
ANll
Cond
AMI
8
1
(a) Noise in augmentation
(b) Variance orthogonal to SF (c) Dimension of projected space
Figure 4:
further extend the results presented Figure 2 with orthonormalized InfoNCE and SimSiam mappings
But for non-contrastive objectives like SimSiam, while we can only prove that the optimal solution lies in the y plane. SimSiam objective is not able to distinguish between the lines y, and hence might lead collapse of representations. But as stated in the Theorem 4. this only the case if K. For r 2 K, the SimSiam objective learns the complete Fisher subspace (and only the fisher subspace).
0 and
G Experimental Details and Additional Results
G.1 Data Generation for Synthetic Experiments
Ambient Dimension We consider the ambient dimension to be 100 for all experiments (except for scaling curves in Figure 2d where we let ambient dimension equal to K 1).
Means of Components For generating means of the components, we first sample K 1 times from a O mean, I normal distribution in ambient dimension space. We chose the K'th mean such that the means sum upto O. Hence the means lie
dimensional space.
Covariance Matrix To construct a covariance matrix we start with a unit variance matrix. We then upscale the variance in (Ambient- NumMeans+1] dimensional space orthogonal the subspace by factor of к, where x = mmax / /min (E). We take k to be 1/0.1 = 10.0 by default. choose random NumMeans/2 orthogonal directions, and increase the
For scaling experiments, covariance in those directions. Note for condition number experiment we take k to be 1/0.z where a goes from 1 to 9.
G.2 Orthonormalized InfoNCE and SimSiam
We plot the ARIAMI numbers for orthonormalized InfoNCE and SimSiam matrices in Figure 4 Specifically, we do QR decomposition these matrices and take the orthogonal matrix the projection matrix. In these plots the convergence issues of InfoNCE loss become more apparent, as that InfoNCE lagging behind optimal.
G.3 CIFAR-100 Experiments and Additional Results
In this section, we present seeded version of our results in the manuscript for the CIFAR- 100 experiments. We randomly sample 20 classes from the datasets and collect the data points that belong these 20 classes. The randomly selected subset images are then normalized before inputting the the mapping method. Once the methods learn linear maps, is then applied on the normalized data.

---

Table 3: Clustering performance linear dimensionality reduction methods on CIFARI00: Bolded and underlined values indicate the best and second-best scores for each column. We re- port clustering performance for K-Means on r 5, 10, 15, 19-dimensional subspaces (mapping dimensions) using the linear mappings learned by 6 different methods. We measure the clustering performance using ARI and AMI. We run the methods for different seeds, corresponding random 20-class subsets of CIFAR- 100 datasets. The average scores over 5 runs are reported with standard deviation given in paranthesis. InfoNCE and LDA outperform the rest by significant margins in all mapping dimensions. LDA shows the best performance in terms of ARI, while InfoNCE achieves higher AMI values across the board.
Mapping Dim 5
Mapping Dim 10
Mapping Dim = 15
Mapping Dim 19
Method
ARI
AMI
ARI
AMI
ARI
AMI
ARI
AMI
Random
0.020 (000))
0.061 m.000)
0.026 (ic0))
0.078
0.028 00.00)
0.084 mocr)
0.033
(0007:
0.094 m005
Ambicnt
0.039 (ume))
0.113 0L013)
0.039
mucte)
0.113 (003
0.039
oLIMM
0.113
0.039
(uase)
0.113 ouns)
PCA
0.038 (omn))
0.110 morg
0.040
mocos
0.113 (mm--)
0.040
0.114 move)
0.041
(moon)
0,116 01015)
SimSiam
0.035 (0000)
0.111 ouremo
0.037
muc00)
0.113 (0422)
0.035
0.110 mv22))
0.035
(D000)
0,110 (LO0R)
InfoNCE
0.058 (0000))
0.146 ouenoo
0.064
0.156 souns)
0.063
0.157 (u004)
0.066 (ouoo:
0.159 (um14)
LDA
0.060
0.145
0.074
0.153
0.074
0.148
0.076
0.149
Finally, cluster the (linearly) transformed data using K-Means where the number of classes is the same as the number of components in the GMM. We repeat this procedure for 5 times with different seeds for the random number generator. We report the results in Table 3 which are consistent with the results in the main text.
H Linear Discriminant Analysis (LDA) and Fisher LDA
H.1 Linear Discriminant Analysis
The classical binary class LDA objective is defined as a bayes-optimal solution for classification under the assumption that the data is generated from two-component Gaussian mixture model with identical covariances. For non-shared covariance for two component case we don't 't have closed form solution and the problem referred as Quadratic discriminant Analysis. The LDA solution for woo-oompneent GMM with shared covariance is the subspace projection on which leads to no loss in likelihood of data, It is given as:
SLDA= B -M U22
H.2 Fischer Linear Discriminant Analysis
Instead of defining the LDA objective to be the Bayes-optimal under the assumption that data is generated from a two-component Gaussian mixture model with shared covariance, Fischer LDA considers an alternative objective. It does away with both the GMM and the shared covariance assumption and assumes data to be coming from two different distributions, where each distribution belonged is defined by it's class. Fisher defined the separation between these two distributions to be the ratio of the variance between the classes to the variance within the classes i.e.
10 (M 22))
S Fischer
1 [ E1 + W22 E
The solution maximizing this is given by ( + - - 112) is termed the Fisher subspace. When the data is generated from a two-component shared covariance GMM, the solution coincides with E-1(p 422 learnt by LDA.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose a standard theory framework in machine learning, and provide theoretical results that validate our claims. We also conduct experiments corroborate theoretical claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have section for limitations at the end of the paper. We also follow up the theorem statements with discussions the implications and shortcomings of the results.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: We clearly state the assumptions we make for the theorems. particular, we refer to the data and augmentation distribution assumptions and clearly define the objective functions with their properties.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
We state the details of our experimental setup. For the synthetic experiments, we give the recipe for data generation and for the real data, we describe how the data is randomly subsampled.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
27

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We are planning to release our code after we prepare a clean and distributable version which is also de-anonymized.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpcc/pccpccpcccceoeeoooop /Codeeeeee seaibbmmssionPo. ssinnnoissiiycc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the experimental setup in details in the manuscript. We provide details on data generation, split and also properties of the algorithm(s) used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see the Appendix for a complete overview of our results with statistical details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss the details related to compute resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tt:://neeiispee http://rrrpps.cppllli/iceece/eet htcp/ppulicc/thhcccccieeeeeeeeid
Answer: [Yes]
Justification:
We abide by the NeurlPS Code of Ethics to the best of our abilities.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We study a fundamental theory problem in machine learning. Our work and results do not have any negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work does not pose any such risks related its content; we do not release new models datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our theory and numerical experiments do not any existing assets that requires license. Any academic resource, model, dataset and algorithm used are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any assets that requires licensing and/or documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The theory-oriented scope our work does not require such approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The development process does not involve L.M use as any important part of
it.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuri)) http:e/r/eearrggcccco///ooeeeerr / for what should or should not be described.
32


Reviews:
[{"review_id": "q79WO27wsI", "reviewer": "Reviewer_nsJs", "summary": "The authors investigate a Gausian Mixture Model variant of contrastive learning. They show a noisy label contrastive setup can converge to the same subspace as discovered through LDA (which needs class labels). They perform quantitative experiments on small scale datasets to demonstrate the theory and show how it varies with key parameters like noise fraction.", "questions": "-Line 63 Missing clip cite\n- Can you help me understand the impact of the shared covariance on the model. Is this an assumption that all clusters have the same shape? If so this seems a little restrictive and I wonder whether your work could be applied to the general GMM case. \n- (For the authors curiosity) One paper that seems relevant to your work is “I-Con: A Unifying Framework for Representation Learning” from the looks of it, your theorem might represent a new row or column in their periodic table of representation learning algorithms.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n-\tGood attention to detail in the mathematics and notation\n-\tInteresting theorem connecting contrastive learning with LDA and Fisher subspaces\n-\tReasonable analysis of results and experiments\nWeaknesses:\n-\tThe work is good but comes off a bit dense at times, perhaps include a overview figure in the work that captures the main plot, this will make it easier for people to quickly glean the important aspects of your work.\n-\tMinor typos\n-\tSmaller scale datasets for evaluation", "was_revised": true, "final_justification": "A reasonable paper, keeping my score at accept", "rebuttal": "**Remark 1 : Including an overview figure for the paper**\n\nThanks for the suggestion. We will try to include one in the camera-ready version of the paper. We plan to add a figure capturing the data model for shared-covariance GMMs and display the properties of representations learnt by different self-supervised loss functions. We will also include a table that explains the main theoretical results for contrastive losses and the properties of the subspaces learned in comparison to optimal Fisher subspace.\n\n**Remark 2 : Comment on the shared covariance assumption**\n\nYou are correct with your assessment; shared covariance means all components have the same covariance, and the data samples per component are distributed over the ambient space in the “same shape”. We wanted to expand the typical spherical (identity) covariance assumption and consider a general covariance matrix shared among components. The assumption though is necessary for the theoretical analysis. One of the steps in our proof requires an affine transformation to make each component’s covariances isotropic, and the shared (non-spherical) covariance assumption is the most general we could come up with (please see Supplementary C). Moreover, the Bayes optimality of the Fisher subspace is only true for shared covariance GMMs and we do not have a reliable measure of optimality beyond that. That said, we are exploring more general extensions of our analysis with appropriate optimality metrics. \n\nIn practice, however, we still get good numbers beyond shared covariances. For general Gaussians, we empirically show that InfoNCE learns representations that match (or even outperform) LDA on clustering metrics. We also include a new set of results on ImageNet. Specifically, we take images corresponding to 20 random classes from ImageNet-1K and use a pretrained ResNet-50 model to get the non-linear mappings. We then learn a linear map from the ResNet embedding space (2048 dimensional) to our target space. We present the results below (presented as ARI/AMI). \n\n| **Method** | **5 dim**             | **10 dim**            | **15 dim**            | **19 dim**            | **30 dim**            |\n| ---------- | --------------------- | --------------------- | --------------------- | --------------------- | --------------------- |\n| Random     | 0.03026 / 0.08296     | 0.07231 / 0.14005     | 0.13140 / 0.23209     | 0.14405 / 0.23363     | 0.26177 / 0.38685     |\n| PCA        | 0.23408 / 0.46447     | 0.39843 / 0.59117     | 0.49903 / 0.65898     | 0.50954 / 0.66159     | 0.56355 / 0.70163     |\n| Ambient    | 0.48641 / 0.67233     | 0.48641 / 0.67233     | 0.48641 / 0.67233     | 0.48641 / 0.67233     | 0.48641 / 0.67234     |\n| SimSiam    | 0.37581 / 0.58233     | 0.56084 / 0.70246     | 0.58705 / 0.72227     | 0.60259 / 0.73159     | 0.62970 / 0.74476     |\n| InfoNCE    | **0.84451 / 0.88831** | **0.98182 / 0.98115** | **0.92963 / 0.97390** | **0.99621 / 0.99579** | **0.93317 / 0.97721** |\n| LDA        | 0.47112 / 0.69828     | 0.66967 / 0.83895     | 0.82550 / 0.89103     | 0.94744 / 0.94990     | **0.94745 / 0.94990** |\n\n\n\n**Remark 3 : Relevant work “I-Con: A Unifying Framework for Representation Learning”**\n\nWe appreciate the reviewer pointing out other relevant work. Our method occupies the same block as InfoNCE i.e., uniform over positive pairs supervisory signal and gaussian learnt representations. Though for us the uniform pairs are defined as points belonging to the same cluster instead of a set of given positive pairs."}, {"review_id": "DQyu9Pzdpf", "reviewer": "Reviewer_79Vv", "summary": "This paper presents a new theoretical perspective on self-supervised learning via contrastive learning, aiming to explain why the simple strategy of contrasting or relating pairs is so effective. The authors formulate the representation learning problem as a form of dimensionality reduction for Gaussian Mixture Models (GMMs) with a shared covariance matrix, referred to as sharedGMMs. Their central finding is that contrastive methods learn the Fisher-optimal subspace—surpassing traditional unsupervised methods like SVD and approaching the performance of supervised techniques such as Linear Discriminant Analysis (LDA). To allow for analytical tractability, the analysis is restricted to linear mappings, and the optimality of representations learned via contrastive learning is studied under the assumption of perfect augmentations for the InfoNCE loss. The authors also distinguish between two categories of contrastive learning objectives: uni-modal and multi-modal GMMs, the latter modeled after CLIP-like architectures. In both scenarios, they demonstrate that contrastive learning with InfoNCE recovers a subset of the Fisher-optimal subspace. For empirical validation, the paper evaluates various linear dimensionality reduction methods on both synthetic sharedGMM datasets and real-world data (CIFAR-100), showing strong alignment between theoretical predictions and experimental results.", "questions": "•  The authors state, “Although we do not provide a proof for $S_{Info}=S_{F}$ when $0<\\delta<1$ we conjecture that it is true, ...” (lines 256-7). Doesn’t this result depend on the noise level introduced by the augmentation process? Is it reasonable to assume that the learned subspace remains optimal under imperfect augmentations?\n\n•  The authors say that *“both contrastive and non-contrastive objectives learn the same subspace”* (lines 261–262). However, Theorem 4.1 suggests that $S_{Siam}$ recovers the Fisher subspace even without perfect augmentations ($\\delta < 1$), while the results for InfoNCE are tied specifically to the assumption of perfect augmentations. This appears inconsistent with the general conclusion (without making use of negative samples). Could the authors clarify this discrepancy?\n\n•  In practice, augmentation techniques do not rely on independently sampled transformations. Instead, each augmentation is typically generated conditionally, based on the original sample (e.g., type-preserving augmentations). Would it not be more realistic to model the augmentation process as a conditional distribution $p(\\hat{x}∣x)$? How would this change the theoretical conclusions, especially under statistical dependencies introduced by type-based augmentations?\n\n•  Theorem 4.1 assumes $r \\geq K$, but in all experiments, the projection dimension $r$ appears to be smaller than the number of clusters. Could the authors explain this mismatch between theory and experimental setup?\n\n•  The synthetic experiments are not clearly described. What is the data distribution? What is the ambient space dimension? Why does the \"optimal\" method perform worse? Is the data generated from a sharedGMM? Why is $r=K−1$ used in the numerical experiments?What is the rationale behind the design of the cluster means?\n\n•  The authors write that *“InfoNCE loss learns to scale within the subspace leading to better clustering performance”* (lines 336–337). Could the authors elaborate on what is meant by \"scaling within the subspace\" and how this contributes to improved clustering?\n\n•  In Definition 2.2, the equation for AeD, should $K \\neq K'$ be satisfied? \n\n•  The paper defines a \"good projection\" as one that performs well in classification tasks. Is this definition sufficient for evaluating the quality of representations more broadly, especially in unsupervised or transfer learning scenarios?\n\n•  Section G.1 in the supplementary material appears to be incomplete. \n\n•  In line 63, the citation is missing.", "limitations": "I think the main limitations of the proposed analytical framework for studying contrastive learning are as follows:\n\n•\tIt relies on the assumption of shared covariance in the GMM model, which is a strong condition and may not hold in many practical scenarios.\n\n•\tIt lacks an explicit analysis or quantification of the effects of augmentation noise and the overlap between components in the GMM, both of which are critical factors in real-world applications.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n•\tThe proposed analytical approach to studying contrastive learning through the InfoNCE loss, even in the simplified setting of linear mappings, is both insightful and well-motivated. The theoretical connection to supervised LDA is particularly compelling.\n\n•\tThe analysis of optimal projection learning for multi-modal data is valuable. Demonstrating that multi-modal InfoNCE learns a subset of the Fisher-optimal subspace contributes meaningfully to the understanding of contrastive objectives in complex settings.\n\n•\tThe introduction of the Augmentation-enabled Distribution (AeD), which includes a hyperparameter to control the correlation between a sample and its augmentation, is novel and provides a more principled framework for modeling augmentations compared to traditional distance-based approaches.\n\n•\tThe manuscript is fairy well written and well-structured, making the theoretical contributions easy to follow.\n\n**Weaknesses**\n\n•\tTheorem 4.1 is restricted to the ideal case of perfect augmentation ($\\delta = 1$). However, such a scenario rarely holds in practice. The analysis does not account for more realistic conditions, such as limited numbers of positive or negative samples, or imperfect augmentations.\n\n•\tThere is no discussion of the number of positive and negative samples required for effective contrastive learning. This is a crucial factor, especially when sample balance and augmentation noise can significantly impact learning outcomes.\n\n•\tThe experimental analysis is quite limited. The use of CIFAR-100 as the primary dataset may not be well aligned with the assumptions of the theoretical framework. As shown in Table 1, the reported performance across all methods is quite low. A more suitable approach might involve first learning a nonlinear embedding for the image data that better satisfies the sharedGMM assumption, followed by applying linear mappings. This could also be extended to non-image datasets to further validate the theory.\n\n•\tWhile the paper touches on the role of augmentation and the distinction between perfect and noisy augmentations, a more quantitative analysis would be valuable. Specifically, measuring the effect of augmentation noise on the ratio of inter-component to intra-component variance would strengthen the empirical insights.\n\n•\tThe experimental details, particularly for the synthetic studies, are not quite clear and need to be clarified. More transparency in how the synthetic data is generated and evaluated would improve reproducibility and understanding.\n\n•\tAlthough the paper provides theoretical analysis for the multi-modal contrastive learning case, it lacks corresponding experimental validation. The authors could consider using a subset of the datasets from the CLIP paper to demonstrate their findings in practice.", "was_revised": true, "final_justification": "The authors provided a detailed rebuttal and addressed many of my questions. They provided a clearer explanation of the assumptions and limitations of the theoretical work. Additionally, they provided a new set of results on the ImageNet dataset.", "rebuttal": "**Remark 1 : Theorem 4.1 is restricted to the ideal case of perfect augmentation. Analysis doesn’t account for more realistic scenarios.** \n\nIn our analysis, we try to show that for any direction in the Fisher subspace *not included in the optimal solution*, the gradient of the loss function is negative. This leads us to upper bound the gradient of the loss by $ - \\delta \\cdot \\sum_k w_k a_k^2 + \\sum_k w_k a_k^2$. This upper bound comes from the Perron-Frobenius theorem (at the end of Supplementary B). The gradient is indeed non-positive if $\\delta = 1$. When $\\delta <1$, our analysis cannot guarantee this, but we believe that the same is true even for relatively large values of $\\delta$. However, this would require a more fine-grained analysis that takes into account the properties of mean subspace and the covariance matrix. \n\nAdditionally, we do study the effect of noise empirically as shown by our synthetic experiments. We believe that introducing noise doesn’t change the optimal subspace in the infinite sample limit, though we expect to have adverse effects on the number of samples required for it. This ties to the finite sample analysis question, which is on our future works list. \n\n**Remark 2 : Reliance on shared covariance assumptions**\n\nWe wanted to expand the typical spherical (identity) covariance assumption and consider a general covariance matrix shared among components. The assumption though is necessary for the theoretical analysis. One of the steps in our proof requires an affine transformation to make each component’s covariances isotropic, and the shared (non-spherical) covariance assumption is the most general we could come up with (please see Supplementary C). Moreover, the Bayes optimality of the Fisher subspace is only true for shared covariance GMMs and we do not have a reliable measure of optimality beyond that.\n\n**Remark 3 : Lack of experimental validation for multi-modal scenario**\n\nOwing to lack of a good small scale multi-modal dataset and a time crunch, we unfortunately could not make it to the rebuttal period. A possible experimental setup would be to take ImageNet images and generate synthetic text captions for them using an LLM. We believe this is a toy scenario and we were not sure about its value before the reviewer’s eye. If you would be willing to see such results, we will conduct a small scale experiment in the above-described manner and make them ready for the discussion period. Alternatively, we could provide an analysis of (a subset of) real CLIP embeddings, which we were not able to complete due to the limited period allocated for the rebuttal preparation. We will be working on this in the meantime.\n\n**Remark 4 : Analysis doesn’t account for more realistic scenarios like a limited number of positives and negatives. No discussion on number of positives and negatives required for effective contrastive learning**\n\nWe acknowledge that our analysis is designed for the infinite sample setting and there does not seem to be a trivial extension to finite samples. The main technical difficulty lies with the LogSumExp term in the loss function of InfoNCE. We believe that our reviewer would agree that such an analysis is still non-trivial in the linear setup we propose. That said, we are exploring possible directions which would enable us to make meaningful comments about the establishing bounds on samples for learning subspaces that are comparable of Fisher subspace. Our goal in this paper is to quantify the absolute limits of contrastive learning in the limit and we defer the finite sample study for our future work. \n\nFinite-sample analysis with sample complexities and convergence rate analysis would also require a careful optimization algorithm design to solve the InfoNCE or other losses, which is not trivial and beyond the scope of our infinite sample analysis. We do leave the finite-sample analysis to future work, which we believe is an independent interest to the community.\n\n**Remark 5 : Limited experimental analysis**\n\nWe agree with your comment on our CIFAR-100 experiments. Following your suggestion, we conducted another set of experiments using the ImageNet dataset. Due to character limits, please refer to *Remark 2 for Reviewer nsJs* for the new result.\n\n**Remark 6 : Experimental details are not quite clear**\n\nWe apologize for the lack of clarity and detail, and thank you for your constructive comment. We will resolve the ambiguities and also fill out any incomplete details in the supplementary sections as pointed out. To summarize the details; for the synthetic experiments we consider a 10-component shared-covariance GMM, with an ambient dimension of 100. The covariance matrix is a diagonal matrix. To simulate a parallel pancake-like setup, we increase the variance in directions orthogonal to the mean subspace which leads to poor clustering performance with methods like PCA. We particularly choose this setup, as unsupervised methods struggle with this kind of data.\n\n**Remark 7 : Would it not be more realistic to model the augmentation process as a conditional distribution. How would this change the theoretical conclusions, especially under statistical dependencies introduced by type-based augmentations?**\n\nConditionally generated samples, as studied in [HaoChen et al., 2021], serve as an alternative framework for analyzing self-supervised learning. Prior work exploring these directions often deal with theory-friendly approximations of InfoNCE loss, i.e., replacing LogSumExp by a squared term and are not relatable to some Bayes-optimal notion of representation that our work deals with. Using probabilistic modelling is a more realistic approach, however, the conclusions will not be enough to explain possible advantages of contrastive learning against, for instance, fully supervised learning. \n\nRegarding the effect of considering augmentations as conditional distributions for our setting: we also tried assuming a naive conditional distribution $\\hat{x} \\sim \\mathcal{N}(x, \\sigma)$, i.e., the augmented sample is a “noisy” version of the point $x$. Assuming this distribution, we were not able to recover the Fisher subspace with the InfoNCE loss. We posit that an ideal conditional distribution should be more informative than the original sample itself, and hence should provide some information about the cluster it belongs to . A stronger conditional distribution (our AeD) which assumes the augmented sample to be biased towards the underlying component of the original sample, is appropriately informative enough to learn the Fisher subspace. Identifying a less restrictive (when compared to AeD) conditional distribution, which can learn the Fisher subspace is indeed an interesting question. \n\nWe admittedly study a much simpler setup, but this helps us prove stronger results, i.e. showing equivalence to fully supervised methods, and our findings are partially recovered by the synthetic experiments and surprisingly validated by the real-data examples (both the old CIFAR-100 experiments and the new, non-linear ImageNet results). \n\n*HaoChen et.al., 2021. Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss, 2021*\n\n**Remark 8 : Projection dimension r>K.**\n\nWe tried to study a more difficult problem with fewer embedding dimensions to possibly make a stronger conclusion empirically, and we unfortunately did not have experiments with $r > K$. Following this remark, we run additional experiments with r=30 when for the 20-class dataset. We refer our reviewer to the last column of our ImageNet results in Remark 5.\n\n**Remark 9 : What does scaling within the subspace mean**\n\nThanks for pointing this out. Our terminology was inaccurate and admittedly loose; let us be more precise. By scaling within subspace, we mean that methods such as LDA learn just a projection matrix (i.e., projection onto a subspace). While methods like InfoNCE learn an affine transformation (projection onto a subspace + affine transformation within the subspace) which leads to more suitable representations for clustering. We interpret that this is possibly the reason InfoNCE does better than LDA (Optimal in synthetic experiments) which only projects onto optimal subspace. We will make this clear in the camera ready version.\n\n**Remark 10 : Definition of a good projection and its sufficiency**\n\nWe believe our reviewers refers to our definition of “favorable” mapping that maximizes Fisher discriminant. Our definition is indeed dependent on learning discriminative representations for each cluster. When each cluster corresponds to a broad class like “dogs”, these representations might be sub-optimal for certain transfer learning tasks (like distinguishing between different dog breeds). But when the clusters are finely defined (via augmentations), we get “good” representations that are discriminative enough for the sub-structures in the data. Naturally, the quality of the representations are tied to how fine-grained and/or noisy the augmentations are (which is the only information through which we could discover “classes”). Under this scenario, we believe our definition is reasonable, however, we would be delighted to hear more on this if our reviewer has better suggestions."}, {"review_id": "NULpSLFRb7", "reviewer": "Reviewer_xYCQ", "summary": "This paper setup a framework to analyze the contrastive learning through the perspective of GMMs, which projects the unlabeled augmentation pairs in InfoNCE into the Fisher-optimal subspace. This paper discusses the ability of infoNCE to recover the optimal shared-covariance projections, which states that minimizing the CLIP InfoNCE loss gives subspaces for each modality that lie within their respective Fisher optimal subspaces. Then they validate their theoretical findings with the synthetic data and CIFAR100 data.", "questions": "Q1: Your contributions show that InfoNCE recovers the Fisher‐optimal subspace under GMMs and extends to multi-modal CLIP losses—but could you clarify what concrete practical benefits or downstream applications that we can expect from these theoretical guarantees? This part seems ambiguous to me.\n\n\nQ2: In Theorem 4.1, you show that InfoNCE (and SimSiam) recover the full Fisher subspace only when the augmentation bias delta = 1. Could you clarify how sensitive the model is with these hyperparameters in practice when we try to ensure near-optimal projection recovery?\n\n\nQ3: For the multi-modal CLIP-GMM result (Theorem 5.2), you prove that CLIP InfoNCE learns only a subset of each modality’s Fisher subspace—could you suggest how we identify or recover the “missing” directions, and what downstream impact this subset selection has on retrieval or classification performance?\n\n\nQ4: The current scope of experiments is somehow narrow.  Only linear mappings, with no experiments on any multi-modal CLIP-style data or validation on deep, non-linear embeddings or real CLIP retrieval benchmarks. Could you comment on whether and how your CLIP-GMM theory holds up when applied to actual CLIP embeddings (e.g. ResNet+Transformer) on image–text retrieval tasks?\n\n\nQ5: Table 1 shows LDA achieving the highest ARI while InfoNCE leads in AMI—could you explain which properties of the learned projections drive this divergence, and what this implies for choosing one method over the other?\n\n\n\nMinor:\n\n1. Missing citations in line 63. will be generated by separate GMMs. “For instance, in the CLIP []”\n\n2. It contains more space in Definition 3.1.1.  \n\n3. In Figure 1, why (a) is well-separated while the (b) is the mode collapse? From the figure, the (b) looks more separated. \n\n4. Why in Line 187 the SVD subspace is italics while in line 189  is not?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "For strengths across the dimensions of originality, quality, clarity, and significance:\n\n\nQuality:\n\n\n\nTheory is good in proving that InfoNCE and SimSiam objectives recover the Fisher subspace under a shared‐covariance GMM, which discusses how the contrastive and non‐contrastive losses align with supervised LDA. Besides, this paper extends the analysis to a CLIP-style multi-modal GMM and shows that contrastive loss learns discriminative subspaces in each modality without explicit class information.\n\n\n\nClarity:\n\n\nIntroduces a clean, analytically tractable model (Augmentation-enabled GMM) that captures “noisy” augmentations and paired multi-modal data. Clearly describe the difference between the collapse scenario with the well separated scenario.\n\n\nSignificance:\n\n\n\nThere are a few papers trying to illustrate how contrastive learning works so well. However, this paper provides an interesting perspective to show that, InfoNCE and related losses recover the Fisher-optimal subspace matching supervised LDA. They extend this to multi-modal settings, which might potentially guide the future design of self-supervised and vision–language models.\n\n\n\nOriginality:\n\n\n\nThe main point in this paper, including casts contrastive learning as linear dimensionality reduction on GMM, defines “noisy” augmentations with AeD, and proves that minimizing InfoNCE recovers the Fisher-optimal subspace. These results are original to me.\n\n\n\n\nWeakness:\n\n\n\nW1: In this paper, the feature extraction through the matrix A looks like a linear mapping. The work does provide a clear analysis, but I want to know how these Fisher-subspace results extend to real encoder architectures or with more non-linear setup.\n\n\nW2:  Validation is restricted to small synthetic mixtures and a toy CIFAR-100 clustering setup (with grayscale, downsampled features and known K for K-Means). There’s no demonstration on large-scale vision or vision–language benchmarks to support the theories are useful under the CLIP situations.\n\n\n\nW3: Natural images or languages lie on highly curved, non-linear manifolds.  Their complex, non‐linear, multi‐modal manifolds could not be well captured by any finite mixture of Gaussians with shared covariance. All theoretical guarantees assume access to the true data distribution (infinite samples). It remains unclear how many real examples or negatives are needed for InfoNCE to work in practice without convergence-rate analysis.", "was_revised": true, "final_justification": "The authors have basically addressed my concerns.", "rebuttal": "**Remark 1 : How do the results (and Fisher optimality) extend to real encoder architectures with a more non-linear setup** \n\nWe show that for linear setting, InfoNCE loss learns the Fisher subspace, which we motivated as the optimal subspace for clustering. We do not know of an equivalent definition of optimality for representations under non-linear mappings, which also seems prohibitively difficult to come up with without assumptions on the mappings. \n\n**Remark 2 : Lack of demonstrations for large scale vision / vision-language benchmarks.**\n\nWe realised the toy nature of our setup and hence have extended our experimental setup to the ImageNet dataset. This setup helps us argue about distributions more general than shared-covariance GMM. Specifically, we take images corresponding to 20 random classes from the ImageNet and use a pretrained ResNet-50 model to get non-linear mappings for the images. We then learn a linear map from the ResNet embedding space (2048 dimensional) to our target space. We present the results below (presented as ARI/AMI). \n\n| **Method** | **5 dim**             | **10 dim**            | **15 dim**            | **19 dim**            | **30 dim**            |\n| ---------- | --------------------- | --------------------- | --------------------- | --------------------- | --------------------- |\n| Random     | 0.03026 / 0.08296     | 0.07231 / 0.14005     | 0.13140 / 0.23209     | 0.14405 / 0.23363     | 0.26177 / 0.38685     |\n| PCA        | 0.23408 / 0.46447     | 0.39843 / 0.59117     | 0.49903 / 0.65898     | 0.50954 / 0.66159     | 0.56355 / 0.70163     |\n| Ambient    | 0.48641 / 0.67233     | 0.48641 / 0.67233     | 0.48641 / 0.67233     | 0.48641 / 0.67233     | 0.48641 / 0.67234     |\n| SimSiam    | 0.37581 / 0.58233     | 0.56084 / 0.70246     | 0.58705 / 0.72227     | 0.60259 / 0.73159     | 0.62970 / 0.74476     |\n| InfoNCE    | **0.84451 / 0.88831** | **0.98182 / 0.98115** | **0.92963 / 0.97390** | **0.99621 / 0.99579** | **0.93317 / 0.97721** |\n| LDA        | 0.47112 / 0.69828     | 0.66967 / 0.83895     | 0.82550 / 0.89103     | 0.94744 / 0.94990     | **0.94745 / 0.94990** |\n\n\n**Remark 3 : All theoretical guarantees assume access to the true data distribution (infinite samples). It remains unclear how many real examples or negatives are needed** \n\nWe acknowledge that our analysis is designed for the infinite sample setting and there does not seem to be a trivial extension to finite samples. The main technical difficulty lies with the LogSumExp term in the loss function of InfoNCE. We believe that our reviewer would agree that such an analysis is still non-trivial in the linear setup we propose. That said, we are exploring possible directions which would enable us to make meaningful comments about the establishing bounds on samples for learning subspaces that are comparable fo Fisher subspace. Our goal in this paper is to quantify the absolute limits of contrastive learning in the limit.\n\nFinite-sample analysis with sample complexities and convergence rate analysis would also require a careful optimization algorithm design to solve the InfoNCE or other losses, which is not trivial and beyond the scope of our infinite sample analysis. We do leave the finite-sample analysis to future work, which we believe is an independent interest to the community.\n\n**Remark 4 : Practical benefits following from our results.**\n\nOur analysis seems to show that while both InfoNCE and CLIP loss are effective at filtering out the noise directions, InfoNCE loss (when considering perfect augmentations) can provably recover all discriminative directions (i.e. learns the complete Fisher subspace). We prove that CLIP loss (at least in the linear setting) is not able to capture the complete subspace and hence the representations are possibly less suited for image-only tasks like clustering. \n\n**Remark 5 : How sensitive is the model to the augmentation noise parameter in practice?**\n\nWe believe the model is robust to the choice of the noise parameter and we empirically verify it with the synthetic shared-covariance GMM experiments. In Figure 2 (a) in the paper, we plot the the clustering performance (AMI/ARI) w.r.t. different augmentation noise $\\delta$. We can see that InfoNCE achieves almost perfect clustering even at relatively high noise levels ($\\delta = 0.4$). Please let us know if you would be interested in further experiments on this for the discussion period.\n\n**Remark 5 : For multi-modal clip what impact does not recovering the Fisher subspace have for downstream performance? How can we identify or recover the missing directions?**\n\nOur results seem to suggest that for a given model capacity, not recovering the full Fisher subspace can possibly lead to collapsing. Since Fisher subspace ensures the maximal separability (non-necessarily perfect separability), there might exist a subspace where certain components are almost indistinguishable, leading to poor performance. \n\nOne possible solution for recovering the missing directions might be to first train image and text encoders separately, and then fine-tune them for CLIP loss. Then, training the image encoder with the InfoNCE loss would help with learning discriminative representations.\n\n**Remark 6 : How the CLIP-GMM theory holds up when applied to actual CLIP embeddings on image-text retrieval datasets**\n\nDue to the size of the CLIP dataset, we were not able to complete experiments on it. We are working on a subset of the dataset and hopefully, we will try to make the results ready for the discussion period. \nMoreover, our theory assumes a linear map and the results could be a result of the limited mapping capacity of these linear maps. It is unclear whether our findings would transfer to representations learnt on highly non-linear networks. \n\n**Remark 7 : Discrepancy between the ARI and AMI scores for LDA and InfoNCE**\n\nA core difference between ARI and AMI is that ARI emphasizes pairwise clustering performance; it looks into how many pairs of points originally belonging to the same cluster ended up together. ARI could be smaller than AMI if the original cluster sizes are not balanced or the final clusters have size imbalance. \n\n**Remark 8 : Minor comments**\n\nWe will work on them as pointed out by the reviewer. Regarding the comment on Figure 1, in this figure we wanted to show how the separability of representations after projection onto their SVD subspace vs onto their fisher subspace. The GMMs are well separated in their ambient space in both Fig. 1 (a) and 1 (b). Though for Fig1 (b), projection onto the SVD subspace (y-axis) leads to a mode collapse, while projection onto the Fisher subspace (x-axis) ensures that the underlying components still remain well separated."}, {"review_id": "BvS1oKiGUg", "reviewer": "Reviewer_7TSE", "summary": "The paper studies contrastive learning through the lens of linear dimensionality‑reduction for Gaussian‑mixture models (GMMs). By introducing an \"*augmentation‑enabled distribution*\" that biases each augmentation toward the same mixture component as its anchor, the authors prove that (i) InfoNCE and a simplified SimSiam objective recover the Fisher–optimal subspace for shared‑covariance GMMs, thereby matching the performance of fully‑supervised LDA, and (ii) a CLIP‑style multi‑modal InfoNCE learns a subset of the Fisher subspaces for each modality.\n\nExperiments on synthetic mixtures and CIFAR‑100 corroborate the theory, showing that InfoNCE outperforms PCA and approaches (sometimes exceeds) LDA on clustering metrics.", "questions": "- [minor] Nonlinear Extensions: Do the authors have insights or conjectures on how their linear‑theory might generalize to deep nonlinear encoders? Could kernel methods bridge this gap?\n\n- [minor] Finite‑Sample Bounds: Can the authors extend their analysis to finite datasets? What sample complexity is required for the empirical InfoNCE solution to approach the Fisher subspace?\n\n- [minor] Role of Temperature and Batch Size: InfoNCE performance in practice depends on hyperparameters like temperature and negative sampling. How do these factors interact with the theoretical model?", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- [theory] Sharp, closed‑form characterization of when InfoNCE is equivalent to LDA. Introduces a clean, tunable notion of “noisy” augmentations ($\\delta$‑biased draws).\n\n- [theory] Through comparison to spectral methods (SVD) and supervised LDA, the work offers valuable insights of when and why contrastive losses can match or exceed traditional methods in the linear setting.\n\n- [empirical validation] Systematic synthetic study isolates effect of (i) augmentation noise, (ii) anisotropy, (iii) projection rank. Real‑data CIFAR‑100 experiment shows gains over PCA and parity with LDA.\n\n**Weaknesses**\n\n- Linear Mapping Assumption: The analysis is restricted to linear projectors, whereas modern contrastive methods use deep nonlinear encoders. It remains unclear how these results extend to non‑linear settings.\n\n- Simplified SimSiam Analysis: The treatment of SimSiam employs a greatly simplified loss (no StopGrad or prediction head dynamics), raising questions about applicability to actual implementations.\n\n- Result for $0 < \\delta < 1$ is only conjectured (stated but unproven, l. 256‑260).", "was_revised": true, "final_justification": "I would like to thank the authors for their rebuttal, which addresses most of my concerns. I agree that the non-linear setting is non-trivial, and I find the paper’s theoretical results both interesting and valuable for advancing our understanding of contrastive learning. Overall, I am inclined toward recommending acceptance.", "rebuttal": "**Remark 1 : Restriction of analysis to Linear Mapping Assumption**\n\nWe show that for linear setting, InfoNCE loss learns the Fisher subspace, which we motivated as the optimal subspace for clustering. We do not know of an equivalent definition of optimality for representations under non-linear mappings, which also seems prohibitively difficult to come up with without assumptions on the mappings. \n\n**Remark 2 : Simplified SimSiam analysis**\n\nThe reviewer raises a valid point. The analysis including the stop-grad and the prediction head introduces new challenges we cannot tackle without loosening the analysis. Our paper deals with the fixed point analysis of these loss functions. Such fixed point analyses are not affected by the stop-grad and the prediction head component. Though we agree these are important components essential to the success of SimSiam, for our results (i.e., learnt representations exist in the Fisher subspace) the presence of these components does not affect the results. To study the contribution of the stop-grad and the prediction head, one would have to take a dynamical systems approach to parameter learning, which is more complex than our setting. It would also be unclear whether there would be a well-defined notion of optimality for this setting.\n\n**Remark 3 : Proof for noise augmentations is only conjectured.**\n\nIn our analysis, we try to show that for any direction in the Fisher subspace *not included in the optimal solution*, the gradient of the loss function is negative. This leads us to upper bound the gradient of the loss by $ - \\delta \\cdot \\sum_k w_k a_k^2 + \\sum_k w_k a_k^2$. This upper bound comes from the Perron-Frobenius theorem (at the end of Supplementary B). The gradient is indeed non-positive if $\\delta = 1$. When $\\delta <1$, our analysis cannot guarantee this, but we believe that the same is true even for relatively large values of $\\delta$. However, this would require a more fine-grained analysis that takes into account the properties of mean subspace and the covariance matrix. \n\nAdditionally, we do study the effect of noise empirically as shown by our synthetic experiments. We believe that introducing noise doesn’t change the optimal subspace in the infinite sample limit, though we expect to have adverse effects on the number of samples required for it. This ties to the finite sample analysis question, which is on our future works list. \n\n\n**Remark 4 : Non-linear extensions**\n\nWe show that for linear setting, InfoNCE loss learns the Fisher subspace, which we motivated as the optimal subspace for clustering. We do not know of an equivalent definition of optimality for representations under non-linear mappings, which also seems prohibitively difficult to come up with without assumptions on the mappings.\n\n**Remark 5 : Finite-sample bounds**\n\nWe acknowledge that our analysis is designed for the infinite sample setting and there does not seem to be a trivial extension to finite samples. The main technical difficulty lies with the LogSumExp term in the loss function of InfoNCE. We believe that our reviewer would agree that such an analysis is still non-trivial in the linear setup we propose. That said, we are exploring possible directions which would enable us to make meaningful comments about the establishing bounds on samples for learning subspaces that are comparable of Fisher subspace. Our goal in this paper is to quantify the absolute limits of contrastive learning in the limit and we defer the finite sample study for our future work. \n\nFinite-sample analysis with sample complexities and convergence rate analysis would also require a careful optimization algorithm design to solve the InfoNCE or other losses, which is not trivial and beyond the scope of our infinite sample analysis. We do leave the finite-sample analysis to future work, which we believe is an independent interest to the community.\n\n\n**Remark 6 : Role of Temperature and Batch Size**\n\nThis is indeed a great questions, and it is relevant to the finite-sample analysis and the convergence rate of the optimization process. Clearly, batch size is not applicable to our analysis framework as we focus on the limiting behavior. Although temperature is more relevant, it should not effect our conclusion, beyond constants in the calculations, that InfoNCE would learn the Fisher subspace under perfect augmentations."}]

Author rebuttal:

Meta-review: This work theoretically investigates the benefit of contrastive learning under Gaussian mixture models (GMMs). As a key contribution, the paper proves that the standard contrastive loss - InfoNCE retrieves the optimal linear mapping in terms of the separability of Gaussian clusters, from a infinitely large number of augmented data pairs in which both data points, the original and the augmented, are generated from the same Gaussian component. The same optimal linear mapping can also be obtained by LDA in a fully supervised manner, demonstrating the competitiveness of contrastive learning w.r.t. the supervised approach. Empirical results on synthetic and CIFAR100 data also support the comparability between contrastive and supervised learnings, in a less restrictive (and noisier) setting where the augmented data are generated from different clusters than the original ones with various probabilities. The multi-modal learning is also studied by considering separate GMMs with matching components, for which the optimal linear mapping is only partially recovered except under certain special configurations of model parameters.

The provided analysis is well motivated and carefully laid out. Its relevance and novelty were unanimously recognized by Reviewers. Several restrictions of the theoretical results were pointed out, such as linear mapping, infinite sample regime, and the conditional distribution of augmented data that depends only on the cluster index of original ones. Removing them would provide deeper insight into contrastive learning but require non-trivial extensions beyond the scope of the current analysis, as argued in Authors’ response. While they do not undermine the main messages of this article, these limitations should be better outlined and discussed in the manuscript, as per the suggestion of Reviewer 79Vv. Another criticism shared by several reviewers is the narrow scope of the experimentation, which is partly addressed by the new results on ImageNet provided during rebuttal, and can be further strengthened with experiments on multi-modal learning.
Decision: Accept (poster)