InfoNCE Induces Gaussian Distribution
Authors: Roy Betser
Eyal Gofer
Meir Yossef Levi
Guy Gilboa
Venue: ICLR 2026 Oral (2026)
Keywords: Contrastive learning
Gaussian distribution
InfoNCE
Primary area: learning theory
TL;DR: Contrastive learning based representations can be well approximated by a multivariate Gaussian distribution.

Abstract: Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive
unlabeled data for both task-specific and general (foundation) models.
A prototypical loss in contrastive training is InfoNCE and its variants. 
In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. 
We establish this result in two complementary regimes. 
First, we show that under certain  alignment and concentration assumptions, projections of the high-dimensional representation asymptotically approach a multivariate Gaussian distribution.
Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results.
We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior.
This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. 
The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.

Published as a conference paper at ICLR 2026
INFONCE INDUCES GAUSSIAN DISTRIBUTION
Roy Betser" Eyal Gofer" Meir Yossef Levi Technion Israel Institute of Technology
Guy Gilboa
ABSTRACT
Contrastive learning has become cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specifie and general (foundation) models. A prototypical loss in contrastive training is InfoNCE and its variants. In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. We establish this result in two complementary regimes. First, we show that under certain alignment and concentration assumptions, projections of the high- high-dimensional representation asymptotically approach a multivariate Gaussian distribution. Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results. We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior. This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. The resulting Gaussian model enables principled analytical treatment of learned representations and expected to support a wide range of applications in contrastive learning.
INTRODUCTION
Self-supervised learning with contrastive objectives has transformed modern representation learning.
enabling scalable training of encoders without labels (Oord et al. 2018 Chen et al.) 2020 He et al.) 2020: Radford et al. 2021). Among these objectives, the InfoNCE loss balances two pressures:
positive pairs are aligned while the batch is repelled encourage uniformity (Wang & Isola 2020).
This uniformity is often described geometrically as "spreading out" the data on the hypersphere (Chen He 2021), but a deeper probabilistic question remains: What is the actual distribution of representations trained with InfoNCE?
Answering this question is not only of theoretical interest. A Gaussian characterization directly motivated by recent empirical findings suggesting that "more Gaussian" representations can correlate with improved downstream performance (Eftekhari Papyan| 2025). It also provides principled basis for practical methods that model contrastive representations as Gaussians for tasks such as classification, uncertainty estimation and test-time adaptation (Baumann et al. 2024 Morales-Alvarez et af. 2024). Moreover, assuming Gaussian structure makes quantities such as entropy, likelihood and KI. divergences available in closed form, which underpins density- -baed diagnostics (Lee) cct al. 2018 Betser et al., 2025). These benefits are already exploited in applied work, with recent studies empirically observing and leveraging approximate Gaussian behavior in self-supervised representations (Baumann et al. 2024 Balestriero et al. 2025 Betser et al.) 2026). Yet, despite these developments, a principled population-leve explanation of why contrastive objectives such as InfoNCE give rise to Gaussian structure in representation space remains lacking.
Analyzing the population InfoNCE objective, formalize the emergence of asymptotically Gaussian representations through two complementary analytical routes. key ingredient is a novel alignment bound based
(HGR) maximal correlation, which limits achievable alignment according to augmentation mildness (Sec. 3.I.. In the empirical idealization route, motivated by empirical training dynamics, alignment reaches plateau and the objective reduces to constrained uniformity problem on the hypersphere; combined with norm concentration, this yields Gaussian structure for both normalized (to a unit norm) and unnormalized representations (Sec. 4.1)
'These authors contributed equally to this work. Corresponding author: roybelcampus techn on. ac.

---

Published as a conference paper at ICLR 2026
Contrastive learning pipeline
Gaussian Distribution
Figure 1: Illustration. Contrastive learning yields (approximately) Gaussian representations.
In the regularized roule, population-level analysis shows that adding vanishing convex regularizer prioritizes the isotropic solution, yielding the same asymptotic Gaussian behavior without relying on training dynamics (Sec. 4.2). Together, these analyses shed light on why Gaussian structure can emerge under the InfoNCE objective at the population level.
We complement our theoretical analysis with empirical studies on synthetic data and CIFAR- 10 (Krizhevsky et al. 2009) images, using encoders of increasing complexity: linear layers, MLPs with nonlinear activations, and ResNet-18 (He al. 2016). By comparing contrastive and supervised training, we isolate the role of the training objective, beyond effects of data architecture. We further observe similar Gaussian statistics in representations learned by general self-supervised foundation models, including DINO (Caron et al. (021). motivating broader examination of Gaussian structure across sulf-supe objectives. Our main contributions are:
Bounded alignment. In the large-batch limit, the alignment induced by the InfoNCE objective is bounded by the strength of the data augmentations.
Uniformity the sphere. Along both routes analyze, normalized representations converge toward the uniform distribution on the unit sphere.
Asymptotic Gaussian structure. Within this framework, both normalized and unnormalized representations admit asymptotically Gaussian behavior under the InfoNCE objective. Empirical support. Accompanying our asymptotic analysis, we provide finite-dimensional empirical evidence on synthetic and real data, illustrating the emergence of Gaussian behavior across multiple settings and encoder architectures.
2 RELATED WORK
Contrastive learning and InfoNCE. The InfoNCE loss (Oord et al. (018) is the standard objective in self-supervised representation learning and underlies methods such as SimCLR (Chen et al. 2020), MoCo (He et al. 2020), and CLIP (Radford et al. (02T). It balances alignment of positive pairs with batch-wise repulsion that promotes uniformity in representation space (Wang & Isola 2020 Chen He 2021). Prior work has studied these effects from geometric and optimization perspectives, identifying phenomena such as hyperspherical uniformity and feature concentration (Chen & He 2021 Caron et al. 2021 Draganov al. (025). Other empirical studies model contrastive represen-
tations as approximately Gaussian (Baumann al. 2024 Aovales-Alvaez al. 2024). However, the probabilistic law induced by the InfoNCE objective itself remains theoretically unexplained.
Isotropy and Gaussian structure. Several works aim to promote isotropic or Gaussian- like representations through explicit regularization or architectural design, including whitening-has based objectives, variance-covariance control, and neural collapse phenomena (Ermolov et al. [222]] Papyan et al. 2020 Bardes et al. 2022). Related self-supervised approaches based on joint-embedding predictive architectures (JEPA) also yield highly regular representations and have been shown to encode density-related structure that can be exploited with Gaussian models (Assran et al. 2023 Bardes et al., 2024 Balestriero et al. 2025 Balestriero & LeCun 2025). However, these works primarily observe or exploit Gaussian- like structure rather than explain its origin. Our work instead shows how Gaussianity emerges directly from the population InfoNCE objective.
2

---

Published as a conference paper at ICLR 2026
Hyperspherical geometry and Gaussianity. A classical body of work studies the geometry of high- nigh-dimensional uniform measures on the sphere and their connection to Gaussian distributions (Vershynin] 2018 Wegner 202I). Related geometric ideas also appear in hyperspherical variational families and radial Bayesian priors, which leverage approximately uniform distributions over the hypersphere (Davidson et al. 2018: Farquhar et al. 2020 A central result in this literature is the Maxwell-Poineeee spherical central limit theorem, which shows that exed-dimensional projections of the uniform distribution on gd-1 converge to a Gaussian
the dimension grows (Maxwell 1860, Poincaré 1912; Diaconis & Freedman 1987). Although developed independently of contrastive learning. these results provide the mathematical basis for why spherical uniformity induces Gaus- sian structure in high-dimensional representations. Our analysis connects this classical theory to contrastive learning by identifying regimes in which the InfoNCE objective induces such uniformity.
Additional theoretical perspectives. Complementary lines of work study theoretical properties of representations learned with contrastive objectives. Identifiability analyses characterize when latent variables semantic factors can be uniquely recovered under structural assumptions on the geteraenng process (yyarinen & Morioka 2016; Hyvarinen et al., 2019: Zimmermann et al.) 2021 Roeder et al., 2021 Reizinger et al.,) 2024); these results concern conditional or component- level structure and do not make claims about the marginal distribution of representations. Separately, clustering guarantees for contrastive representa- tions (Saunshi et al., 2019 HaoChen et al.) 2021). focusing on class- conss-coniitioall geometry rather than the overall distribution. Concretely, class cass-specific clusters may remain well separated even when the overall embedding distribution is approximately Gaussian. Our work does not address recovery or class structure; instead, it analyzes the marginal distribution induced by the population InfoNCE objective.
2
task-driven analyses establish class separability
3 SETUP
Data domain. Let ((X)) be a standard Borel space (a standard setting in probability) with a base probability Pbase- We draw Xo ~ Pbase single data item (e.g., an image).
Pairs via augmentation. Contrastive learning is built around pairs of related examples rather than individual samples. To form such pairs, sample Xo ~ Pbase and produces stochastic variations of it. Formally, given Xo. we draw two independent augmentations
augmentation channel A, which takes base
X,Y ~ A(- X)).
(I)
Here X and Y two views of the same underlying example (e.g., different crops color jitter). We denote by px the marginal distribution a single augmentation and assume it is nonatomic (a mild technical condition achievable in practice by infinitesimal dither). pxy denotes the joint distribution of a pair of augmentations (X,Y).
InfoNCE loss. Let f: Rd, d 2 2, be Borel-measurable encoder that maps input data rep- resentations. InfoNCE operates on nornmaalede representations, defined as f(x) == (=)/ (=) if [[(())]] > O, and f(x) == co for fixed arbitrary co E s otherwise. Given a batch of N paired augmentations {( y1) N drawn i.i.d. from pxy, define u, u:: f(x:) and U; == f(y)). The empirical InfoNCE loss is
N Σ log im1 Σ exp( ( (u, v1))
exp(= 0z))
LInIONCE = N
(2)
with fixed temperature > 0. Since and Uj are unit-normalized, (u,, 03) equals cosine similarity. The numerator measures the similarity of the positive pair (u)) The denominator compares each anchor u, to all candidates {v, J.y where j # i serve as negatives. This softmax encourages u, to rank its true partner highest while remaining distinct from negatives, preventing collapse.
Population InfoNCE. The empirical InfoNCE loss in Eq. (2) depends the batch size N. As N + co, the empirical averages converge to expectations. Let
/ = f.px;
x =:= (f, N.PxY,
3

---

Published as a conference paper at ICLR 2026
be the marginal distribution of representations and the joint distribution of positive pairs, respectively. Here f.pxx denotes the pushforward measure of px by f. which is the distribution of f(X). As shown by Wang & Isola (2020) Theorem 1, Eq. (2)), in the infinite-negativs limit N + co the empirical InfoNCE loss (up to the additive log N term) converges the following population functional. With
= 1/TT for fixed T > 0:
=(,,T) = -a ~m [u-u] + ((u);
((n) == Enw log Enaay expl (u.()xu.u).
The first term measures alignment of positive pairs, while the second is a uniformity potential depending only on j..
3.1 ALIGNMENT BOUND
We now introduce a new term that quantifies the degree of augmentation. The augmentation channel A limits how much positive-paii alignment can be induced. We quantify this with the augmentation mildness parameter
Var (E) Xol) Var(g(X))
1/2
sup gEL (px)
[0, 1].
Var(g)>0
which measures how predictable functions of the view X are from the base Xo. This quantity (HGR) maximal correlation, denoted (X, ie..
equals the squared Hirschfeld-Gebel
722 = p., (X, Xo) [Hirschfeld 1935 Gebelein 1941 Rényi 1959) (see Appendix A. Intuitively, ?/2 0 when X is (effectively) independent of X (very strong/noisy augmentations), and 7/2 when is fully determined by Xu (no augmentation noise).
Example. Consider the Gaussian channel X = AXo + VI VI-A3e where Xo ~ No, 1) and ~ (0, 1) are independent. In this case, X and Xo are jointly Gaussian with Pearson correlation A, the maximal correlation satisfies Pm, Xo) = |A], and thus 7/2 = A² (Appendix A.2).
alignment bound). Let X, A(- Xo) be conditionally independent given the base sample Xo, and let u = f(X), v = f(Y) be normalized representations in Sd i.e. ||u|| ||l||| 1. Then
Proposition (Augmentation-conteee
B(,,,)==lu v] S 72 + 12) [Im(
m(u) == E[u] Biu],
(6)
where 1/2 = (X, is the squared HGR maximal correlation between the view and the base, and I is the marginal law of u.
The proof appears in Appendix A.3 This bound links the alignment of positive pairs to the structure of the statistical dependence induced by the augmentation channel. While HGR maximal correlation has been studied in statistical dependence analysis (Huang & Xu 2020: /Zhang et al.) [024), it has not previously been used to control alignment in contrastive learning. Existing work studies augmentations empirically (c.g., Tian et al. (2020)) but does not derive bounds of this form. This result formalizes how the strength of data augmentations fundamentally constrains achievable alignment under the InfoNCE objective.
4 GAUSSIANITY FROM INFONCE
We study why minimizing the population InfoNCE objective (Eq.E) yields (approximately) Gaussian now-dimensional projections learned representations, for both normalized representations on the sphere and unnormalized representations in RRd. Our analysis proceeds along two complementary routes, which differ in the strength of the assumptions they require.
Empirical idealization. We first analyze idealized regime with infinite data, ambient dimension + oo, and sufficient optimization. Guided by empirical observations, we assume alignment plateau and thin-shell concentration; these assumptions enable simple derivation of Gaussian projections.
Regularized route. To reduce reliance on training dynamics, we study a regularized variant of the population objective. Introducing vanishing convex regularizer and assuming attainable alignment at uniformity ensures a unique minimizer and yields the same asymptotic Gaussian structure. This route provides alternative explanation independent of training behavior.
4

---

Published as a conference paper at ICLR 2026
4.1 GAUSSIAN PROJECTIONS ALIGNMENT PLATEAU
Proposition provides an upper bound on achievable alignment. In the sequel we do not assume this bound tight; instead, we model training as reaching a plateau that lies strictly below the bound.
Assumption 1 (Alignment plateau). After sufficient training, the positive-pair alignment saturates at ceiling: concretely.
[(u.))~<(u-d) 1/2
Tplat:
where plat S. O is constant error term representing the difference between the alignment value at plateau and the maximal correlation defined by the augmentations (112).
Empirically, alignment saturation has been reported in some contrastive-leaniig settings (Wang & Isola, 2020), which motivates considering plateau model as a plausible scenario rather than universal requirement. our experiments (Fig. 2 Appendix Figs. 7 we frequently observe high alignment alongside improving uniformity with larger dimensions and batch sizes, suggesting that alignment may saturate before uniformity in at least some regimes. An extension that places plateau exactly at the alignment bound (Eq. (6 is discussed in Appendix D
Corollary 1 (Gaussian k-projections at the plateau), Suppose the alignment plateau condition (Eq. 7 holds, and consider the population objective (Eq. D.. Let μ³ denote the global minimizer supported on Sd Then, + for every fixed k 2 1 the k-dimensional marginal of u² satisfies
Vdu = NN(O.T Tw):
(8)
where ug denotes the projection of onto fixed k-dimensional coordinate subspace and Tg the k x k identity matrix.
The proof is provided in Appendix C.I and follows from two lemmas. The first establishes that & ((u) attains global minimum at the uniform law (Wang & Isola 2220)) while the second invokes the central limit theorem on the sphere (Diaconis & Freedman 1987) to deduce Gaussian projections.
4.1.1 GAUSSIAN PROJECTIONS FOR UNNORMALIZED REPRESENTATIONS.
So far we analyzed normalized representations on the sphere. We now extend the result to the original, unnormalized encoder outputs 2 = f(X) E IR Write = ru, where = ||||| is the representation radius and = =/||=|| E gd-1 the normalized direction.
Assumption (Thin-shell concentration). We assume the representation radius concentrates:
1,
To
d
where ro E o0) is deterministic constant.
Norm concentration is widely observed in contrastive learning: unnormalized representations cluster around characteristic radius (Wang Isola 2020 HaoChen et al. [222) [evv Gilboa (005).. This thin-shell effect (Klartag 2023) is further promoted by weight decay, which penalizes and stabilizes a common scale. particular, Draganov et al. (2025) show that appropriate weight decay suppresses norm inflation and tightens the dispersion of representation norms, lending empirical support to Assumption Consistent with these reports, our experiments exhibit progressively sharper radius histograms dimension and batch size increase (Figs.
growth
Proposition 2 (Gaussian projections for unnormalized representations). Let f(x) E Rd be the unnormalized representation and u u= z/ z||. Assume u ~ (the uniform distribution on Sd-1) and ro (0, oc). Then any fixed k-dimensional subspace, that Assumption 2 holds, i.e., F d
Vdza N(0, TT)))
+ oo),
(10)
where zy denotes the orthogonal projection of onto that subspace and In the k x k identity.
See proof in Appendix
C.2
5

---

Published as a conference paper at ICLR 2026
uniform
less uniform
more uniform
0.5
0.5
more uniforth
Cosine similarity
0.0
Cosine similariey
0.5
1.0
consistent optimal alignment
consistenc optimal alignment
0.5
un unrmoryy alignment
uniformity
aligoment
0.0
32
64
128
256
512
0.0
64
128
Dimensions
Batch size
Figure 2: Uniformity vs. aligmment across settings. A simple linear encoder trained on synthetic Laplace data exhibits (i) near-optimal alignment across all configurations and (ii) steadily improving uniformity as batch size or dimensionality grow.
4.2 GAUSSIAN PROJECTIONS USING REGULARIZATION
Proposition shows that alignment is limited by the augmentation channel Eq. 6 the uniform distribution (jz ===0) .) the mean vanishes, m(a) = 0, and the bound reduces to E [- 72- Assuming this ceiling is attainable at uniformity, the uniform distribution becomes asymptotically optimal for the population objective. We work in regularized setting, where the regularization vanishes as d + 30. As before, this has direct implications to the representation projections, which are approximately Gaussian (Theorem 2 This result shows that Gaussianity can be obtained without relying on the stronger thin-shell or plateau conditions.
We constrain f take values in B S Rd, which is either some closed ball centered at with positive radius or Rdd We take the original loss and add two new losses: one to penalize large squared norms, and the other to encourage high entropy (we comment that both commonly regarded desirable goals, irrespective of our setup). Specifically, for fixed B, A > O,
J(J) ((²) aB r) + ( + \E2z0p1Z|(²),
(11)
where p f.Px is the unnormalized pushforward probability. Define the truncated Gaussian 7X
(dz) CB, cB, A||==1 12(2)d2,
CB,
1 e->llell' dz.
(12)
If p < 1x (<< denotes absolute continuity, so is absolutely continuous with respect to 7x) then
dp dp
dr
kL(ph) log
log
d + A,,112112 log eg
(13)
dz
that is, equality up to an additive constant. Since p(B) if * 7ht then both (L and (p) are +00. Thus, it is equivalent to minimize
J( (p)) (u,) [u.v] 3KL ( 7))
(14)
and we thereby also implicitly restrict p to satisfy p < 7 and in particular p(B) =
Our goal is to prove that for / > 9o, taking the angular probability approaches optimality and the optimal radial probability is that of 7 If = Rd this means that Gaussian p approaches optimality. Furthermore, d co, fo 0.
This will be done in several steps. First, can be decomposed into a radial part and an angular part. We show that the radial part can be chosen optimally in straightforward way.
Proposition 3. Let p(dz) ((du))x(dr u) and 7 (dz) addu) u) in polar coordinates 2 ru. Then x & is an optimal choice, yielding x KL(u|Jo).
The proof is given in Appendix B.I The above proposition reduces the optimization problem for unnormalized embedding to normalized embeddings only. It also describes an optimal probability

---

Published as a conference paper at ICLR 2026
for embedding norms, in contrast to the original InfoNCE loss, which is completely oblivious to embedding norms.
It is important to note that because px. any probability p E P(B) has 9.Px for some encoding g. In addition, any μ E P(S--1) has μ1 === px for some encoding, and since B contains a ball around 0, there is encoding s.t. f. Thus can legitimately speak about "choosing" p or μa, since suitable encodings exist that induce them. In addition, we may also define:
working with standard Borel space with a nonatomic
Definition 1. For every μ E P()
Align(u) sup { B(f(X) f(Y): f measurable, (f)-Px - =
(15)
f
As was noted, the supremum is always taken on nonempty set. We can write J(p) = (()) - (-llin(())) + [KL(H)))),
(16)
and it holds that inf (/-f.xx=w) J() = [(n), and consequently inf, J(f) = inf i) J(u).
The reason is that (l) can be approximated arbitrarily well by an encoding, and the KL divergence optimized by taking the radial distribution given in Proposition 3 We can therefore focus on optimizing (()).
The assumption for which we will prove our result is the following:
Assumption 3. holds that am1 Al))n(o)) d-000 O.
We will require one more technical lemma before proceeding to prove the result.
Lemma 1. If 2 2, then KL(p]lo) 2 C(d - ()]]m())) where > 0 is a universal constant.
Proof is provided in Appendix B.2. To understand the constant, see /eershynnn 2018 tion 2.6.1).
Proposi-
Theorem 1. Let There is a universal constant C 0 s.t. for 3 B2 Bo
a(1-m) %
C(d-1)
Under Assumption
((a inf, J(4)
d++20
Assuming further that Align(o) 172 yields that J((o min, J(p).
Moreover, as > 00, 30 +
Proof. Write 6(d) 7/2 Aligu(a). For every μa we have that k(x) D(a) 2 0 (Wang & Isola
2020 Theorem 1). In addition.
Ajign( Align(a) ) I + (1 m2)11m - (12 5(d)) = (1 72)1lm(()) 5(d) (17)
by Proposition 0 Lastly,
{T(u|}o) KL(ollo) = KL(n]lo) 2 C(d - (0_-1)]lmmpp)) m
(18)
by Lemma 0 Therefore,
+ J(p) j(o) = (0()) - (())) - ((Align(a) - [(K((l)) KL(ollo))
2 aa 12) (2)[(m((]]1 ao(d) + BC(d - 1))mm(()
(-a(1 (2) + 3C(d - 1)) [|][1]11 ad(d) 2 -ao(d)
(19)
where the last inequality is by the choice of
If we assume that cc)(d) d=20 O, then J(or - inf, J(n) S ad(d), so (((] inf, J(w) d+00 0. If we assume further that Align(a) 1/2, then 5(d) 0, and since J(a) j(n) for every J. ((a) min, j(x,, completing the proof.
Since the optimal radial component of the distribution is known, we can draw conclusions w.r.t. p as well. For example, we can directly obtain the following corollary.
Corollary 2. Let B Rd (d 2 2) and 3 2 Br- If Align(o)) = 172. where o is the uniform distribution on gd-1 and U/2 is the augmentation mildness, then N (0, (2) is an optimal choice for p.
7

---

Published as a conference paper at ICLR 2026
Norms mean
60
0.8
Batch size
40
Density
Dim
Density
Dim = 128
32
20
10
128
20
46
Norm Value
Norm
Dimensions
0.20
AD Value
0.45
DP Value
0.50
32
64
CV
0.40
0.45
0.10
64
96
128
0.35
32
96
120
32
96 120
512
Batch size
Batch size
Batch size
Figure 3: Synthetic data experiments. Left: representation norm statistics vs. batch size (curves denote dimension), showing thin-shell concentration with increasing d and N. Top middle/right: norm histograms illustrating radius tightening. Bottom: normality diagnostics (AD, DP), with averages in the Gaussian acceptance range.
EXPERIMENTS
We empirically evaluate the distributional geometry of representations learned with the InfoNCE objective. The experiments designed to test three theoretical predictions: (i) concentration of representation norms on thin shell, (ii) emergence of Gaussian low-dimensional projections, and (iii) the dependence of these phenomena on contrastive learning.
We consider three settings of increasing complexity: synthetic data with linear encoders, CIFAR-I0 with both contrastive and supervised training, and pretrained foundation-scale models. In all cases, we analyze both normalized and unnormalized representations. All reported trends are stable across runs; figures show representative seeds, with full implementation details in Appendix E..1
Metrics. We quantify Gaussian structure using complementary diagnostics targeting radial and coordinate-wise behavior. To assess norm concentration, we measure the coefficient of variation (CV) of representation norms:
std( f|| (1 mean( (Il = (1
CV
(20)
the learned representations and N is the number of samples. A small CV indicates concentration of || around a characteristic radius, consistent with thin-shell behavior.
To evaluate Gaussianity of low-dimensional projections, apply two standard one-dimensional normality tests to individual coordinates: (i) the Anderson-Darling (AD) test (Anderson & Darling 1954), where AD 0.752 corresponds to failure reject normality, and (ii) the D D'Agostino-Pearson
(DP) test (D'Agostino & Pearson 1973). where > 0.05 indicates failure to reject the Gaussian hypothesis. These tests probe marginal normality of fixed coordinates, as predicted by the spherical central limit theorem.
Taken together, CV captures global radial structure, while AD and DP test coordinate-level Gaussian- ity. This combination provides strong inite-sample indicator of approximate Gaussian behavior and provides evidence against common heavy-tailed mixture alternatives, which typically fail least one of these diagnostics.
Synthetic data experiments. We begin with controlled synthetic settings to validate our diagnostics and isolate the mechanisms predicted by the theory. We consider three synthetic data distributions: (i) an i.i.d. Laplace(O, 1) distribution, (ii) a Gaussian mixture with 25 equally weighted components and random means, and (iii) fully discrete sparse binary distribution (1024-dimensional vectors). Each dataset contains 10k samples, and we train linear encoders using InfoNCE while varying the representation dimension and batch size. In addition, explicitly track alignment and uniformity as functions of batch size and dimension (Fig. 2 to probe the saturation behavior predicted by our Assumption
8

---

Published as a conference paper at ICLR 2026
Features
1.0
Not Geustian
0.20
value
0.8
8
0.18
0.7
Geussian
0.9
0.6
25
100
100
100
Epochs
Epocns
Epochs
Figure 4: CIFAR-10 training dynamics. A two-layer MLP trained with InfoNCE on CIFAR-10 exhibits increasing Gaussianity over training. Left: representation norms concentrate indicated by declining CV (Eq. 20). Middle: the AD statistic decreases from non-Gaussian levels into the normal range. Right: the fraction of coordinates passing the DP normality test rises steadily.
Figure 3 shows that, for Laplace inputs, representation norms progressively concentrate as both batch size and dimension increase, evidenced by a monotonic decrease in the coefficient variation (CV). Norm histograms further illustrate the emergence of thin-shell concentration. Normality diagnostics (AD and DP) indicate that individual coordinates fall well within Gaussian acceptance thresholds, with perfect per- cor-connee compliance (Table IT.
Across all three synthetic settings, including strongly non- non-Gaussian mixture inputs, the learned representations exhibit low norm variation and strong coordinate-wise Gaussianity (Table [ indi- cating that marginal Gaussian structure emerges independently of the input distribution. The same phenomenon is observed for the fully discrete binary dataset: although representations are initially far from Gaussian, training drives pronounced norm concentration and coordinate- wiit normality. Since this distribution admits no invertible mapping to a continuous Gaussian, the observed structure cannot be explained by latent Gaussian recovery.
In parallel, aligmment quickly approaches a stable ceiling determined by the augmentation channel (Fig.2) (2. while uniformity continues to improve. This behavior is consistent with the saturation route and the emergence of isotropic Gaussian structure in high dimension. Together, these controlled experiments support the assumptions underlying theoretical analysis and motivate the study of Gaussianity more realistic settings.
CIFAR-10 experiments. We next study whether Gaussian structure emerges in realistic vision setting. We train a two-layer MLP with a single ReLU nonlinear activation using the InfoNCE objective CIFAR-10, and evaluate representations the test set throughout training.
Figure f shows consistent trends across training: representation norms concentrate over time, as indicated by steadily decreasing CV; the AD statistic drops from non-Gaussian levels into the normal regime; and the fraction of coordinates passing the DP test increases monotonically. These dynamics illustrate the joint emergence of thin-shell concentration and coordinate-wisg Gaussianity as optimization progresses. These trends mirror the synthetic setting and show that norm concentration and Gaussianity also emerge in realistic contrastive training.
Table Gaussianity diagnostics across data and training settings. We report norm concentration (CV) and Gaussianity via AD and DP tests (average statistic and fraction of compliant coordinates). Lower AD and higher DP indicate closer Gaussian agreement. Binary BO/E50/E 100 denote evaluation
epochs 0/50/100; other results from the end of training. Results unnormalized embeddings.
Synthetic (I (Linear)
CIFAR-10 (ResNet-18)
Metric
Laplace
GMM
Binary EO
E50
E100
Supervised Contrastive
CV
0.08
0.08
0.36
0.12
0.09
0.5
0.09
AD Avg. (< 0.752)
0.38
0.39
1.64
0,41
0.42
3.3
0.43
AD Norm. Feat.
100%
100%
30%
93%
97%
6.2%
96.1%
DP Avg. (> 0.05)
0.49
0.46
0.02
0.44
0.46
0.041
0.39
DP Norm. Feat.
100%
100%
15%
89%
98%
3.9%
94.5%
Gaussian?
X
X

---

Published as a conference paper at ICLR 2026
Contrastive vs. supervised training. To isolate the role of the training objective, we compare supervised and contrastive learning using the same ResNet- 18 architecture on CIFAR-10, following the SimCLR training protocol. Both models share identical initialization and capacity, differing only in the objective: cross-entropy supervision versus InfoNCE. As shown in Table I supervised training yields representations with high norm variability and strong deviations from Gaussianity, with most coordinates failing both AD and DP tests. In contrast, InfoNCE training produces concentrated norms and Gaur-aa per-coordinate distributions. These results indicate that the emergence of Gaussian structure is not explained by the data or architecture alone, but is also direct consequence of the contrastive objective.
Table 2: Gaussianity diagnostics for pretrained models. Coordinate- wise Gaussianity via AD and DP tests (average statistic and fraction of compliant coordinates). Test thresholds are indicated in headers. Results shown for Unnormalized / Normalized embeddings.
Training
Model data)
AD 0.752)
DP & D.05)
Guussian?
Avg.
Norm. Feut. (%)
Arg.
Norm. Feut. (%)
Supervised
RosNet-34 (MS-COCO)
10.01 79.638
0.0% / 0.00%
2.2% 2.2100 7 73.2 112 10
0.0% / 70.0%
DenseNet (MS-COCO)
2.982 2.8588
42.2%
0.1550 1442
49.3% 49.0%
DINO (MS COCO)
0.44/0.44
97.0% 19.1%%
0.45
99.2% 99.3%
CLII (Image, MS-COCO)
0.47 / /0.49
96.8% 99.00%
0.42/0.39
99.6%% / 99.44%
Self-sapervised
CLIP (Text, MS COCO)
0.53/0..5
94,0%.
99.4%
CLIPP (Image. Sketch)
0.4/0.4
94.8% / /94.7%
0.44/0.42
93.3% / 93.2%
CLIP (Image. Painting)
0.41 /0.4
95.3% 19.11%
0.43//0.
94.2% / 93.9%
Pretrained models. We further examine whether Gaussian structure persists in large pretrained repre- sentations. On the MS-COCO validation set (Lin et al. (0014), we compare self supervieee backbones
CLIP (ViT-L/14 image and text encoders) (Radford et al.] 202T) and DINO (ViT-B/32) (Caron et al.,
2021 against supervised TaageNet-pretranne (Deng et al. 2009), ResNet34 (He et al. (2066), and DenseNet (Huang et al. 2017). Normality diagnostics (Table 2 show that self self-supervised models exhibit near-Gaussian coordinate distributions, while supervised models deviate substantially. We further evaluate the CLIP image encoder on ImageNet-R (Sketch and Painting domains) to test robustness beyond natural images, and again observe strong Gaussian signatures. Although CLIP and DINO are not exact instances of the unimodal InfoNCE setting, these diagnostics suggest that similar isotropic Gaussian-lis like statistics may arise broadly in self supervised objectives.
6 DISCUSSION AND CONCLUSION
We showed that InfoNCE trained representations admit an asymptotic Gaussian law, via two routes: an alignment-plateau analysis with thin-shell concentration, and a regularized surrogate with milder assumptions. Experiments on synthetic data, CIFAR-10. 10, and pretrained models (MS-COCO and ImageNet-R) are consistent with these assumptions and the Gaussian hypothesis, revealing norm concentration, alignment saturation, and near- Gaarsian projections. These results indicate that the Gaussian convergence remains informative well before the infinite-dimensional limit. This Gaussian view justifies common modeling choices (e.g., likelihood scoring, OOD detection) and suggests that explicit isotropy promoting regularizers may act as principled surrogates for InfoNCE's implicit bias. However, limitations remain: our results are asymptotic, relying on high-dimensional limits and idealized assumptions that may not capture all practical regimes. We therefore view our asymptotic framework as a principled starting point rather than a complete description of all practical regimes. For finite dimension d and batch size N, projections are close to Gaussian, with deviations vanishing as d, N + co. Quantitative bounds follow from classical Berry-Esseen (Vershynin) (0188) rates in high dimension and uniform laws of large numbers for empirical objectives (Wellner et al. 2013)) In particular, the minimizer of the empirical InfoNCE loss deviates from the population minimizer by O(N-1/2) according to Wang & Isola (2020) Thm. 1), and the distribution of fixed-k projections deviates from Gaussian by O(d-1) according to Diaconis & Freedman (1987) (see Theorem 2 in Appendix C.I). Thus, for large but finite d, N. the Gaussian limit provides a representative and empirically useful approximation. In addition, we do not analyze optimization dynamics or prove that training attains these minimizers in practice; our results are asymptotic and characterize the population optima under the stated assumptions. Overall, we provide principled asymptotic explanation for Gaussianity in contrastive representations, grounding empirical observations and opening new directions for analysis and practical design.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work is theoretical and empirical in nature, focused on understanding the statistical behavior of representations trained with contrastive learning. We do not foresee direct negative societal impacts. Potential downstream applications of Gaussian modeling (c.g.. density estimation, OOD detection) could influence decisions safety-critical domains, and care must be taken ensure robustness and fairness.
REPRODUCIBILITY
We provide detailed descriptions of theoretical assumptions, proofs, and experimental protocols. Datasets (synthetic data, CIFAR-10 (Krizhevsky et al., 2009), and MS-COCO (Lin al., 2014)) are publicly available. Architectures, hyperparameters, and training settings are fully specified (Appendix E. and to ensure reproducibility. code for the experiments is released here
ACKNOWLEDGMENTS
We would like to acknowledge support by the Israel Science Foundation (Grant 1472/23) and by the Ministry of Innovation, Science and Technology (Grants No. 5074/22, 8801/25).
REFERENCES
Venkat Anantharam, Amin Gohari, Sudeep Kamath, and Chandra Nair. On maximal correlation, hypercontractivityy, and the data processing inequality studied by Erkip and Cover. arXiv preprint arXiv:1304.6133, 2013.
Theodore W Anderson and Donald A Darling. A test of goodness of fit. Journal of the American statistical association, 49(268) 765-766, 1954.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Kendall Atkinson and Weimin Han. Spherical harmonics and approximations on the unit sphere: an introduction, volume 2044. Springer Science & Business Media, 2012.
Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint X /. 108544, 2025.
Randall Balestriero, Nicolas Ballas, Mike Rabbat, and Yann LeCun. Gaussian embeddings: How JEPAs secretly learn your data density. arXiv preprint arXi:22510.05949, 2025.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICRegL: Self-supervised learning of local visual features. Advances in Neural Information Processing Systems, 35:8799-8810, 2022.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.084/ 2024.
Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Amno Solin, and Martin Trapp. Post-hoc probabilistic vision-language models. arXiv preprint arXiv:2412.06014 2024.
Roy Betser, Meir Yossef Levi, and Guy Gilboa. Whitened CLIP as a likelihood surrogate of images and captions. In 42nd International conference machine learning. 2025.
Roy Betser, Omer Hofman, Roman Vainshtein, and Guy Gilboa. General and domain-specinn zero- shot detection of generated images via conditional likelihood. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026.
11

---

Published as a conference paper at ICLR 2026
Wlodzimierz Bryc and Amir Dembo. On the maximum correlation coefficient. Theory of Probability & Its Applications, 19(1:1332-138, 2005.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Ralph D' Agostino and Egon Pearson. Tests for departure from normality, Empirical results for the distributions of b2 and vbi. Biometrika, 60(3):613-622, 1973.
Tim Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspherical 2018.
variational auto-encoders. arXiv preprint /804.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Persi Diaconis and David Freedman. Asymptotics of graphical projection pursuit. The annals of statistics, pp. 793-815, 1984.
Persi Diaconis and David Freedman. A dozen de Finetti-style results in search of theory. Annales de I'IHP Probabilités et statistiques, 23($2):397-423, 1987.
Andrew Draganov, Sharvaree Vadgama, Sebastian Damrich, Jan Niklas Böhm, Lucas Maes, Dmitry Kobak, and Erik Bekkers. On the importance of embedding norms in self-supervised learning. arXiv preprint arXiv22502.09252, 2025.
Paul Dupuis and Richard S Ellis. A weak convergence approach to the theory of large deviations. John Wiley & Sons, 2011.
Daniel Eftekhari and Vardan Papyan. On the importance of gaussianizing representations. arXiv preprint wwxiiv225050006: 2025.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In International conference on machine learning, pp. 3015- 3024. PMLR. 2021.
Sebastian Farquhar, Michael A Osborne, and Yarin Gal. Radial Bayesian neural networks: Beyond discrete support in large-scale Bayesian deep learning. In International Conference Artificial Intelligence and Statistics, pp. 1352-1362. PMLR, 2020.
Hans Gebelein. Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung ZAMM-Journal of Applied Mathematics and Mechanies/Zeitschrfff für Angewandte Mathematik und Mechanik, 21(6):36-377) 379, 1941.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self supervised deep learning with spectral contrastive loss. Advances in neural information processing systems, 34:5000-5011 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738, 2020.
12

---

Published as a conference paper at ICLR 2026
Hermann O Hirschfeld. A connection between correlation and contingency. Mathematical proceed- ings of the cambridge philosophical society, 31():5200-3 524, 1935.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Shao-Lun Huang and Xiangxiang Xu. On the sample complexity of HGR maximal correlation functions for large datasets. IEEE Transactions on Information Theory, 67(3):1951- 1980, 2020.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd international conference on artificial intelligence and statistics, pp. 359-868- PMLR, 2019.
Bo'az az Klartag. Logarithmic bounds for isoperimetry and slices of convex sets. arXiv preprint 149388, 2023.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images, 2009.
Kimin Lec, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Meir Yossef Levi and Guy Gilboa. The double-ellipsoid geometry of CLIP. In 42nd International conference on machine learning, 2025.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 15227122-77625, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona. Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Kanti Mardia and Peter E Jupp. Directional statistics. John Wiley & Sons, 2009.
James Clerk Maxwell. li, illustrations of the dynamical theory of gases. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 20(130): 1860.
Pablo Morales-Alvarez, Stergios Christodoulidis, Maria Vakalopoulou, Pablo Piantanida, and Jose Dolz. Adapte:: enhanced uncertainty estimation in CLIP few-shot adaptation. arXiv preprint 091.0184 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 2018.
Vardan Papyan, XY Han, and David Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Henri Poincaré. Calcul des probabilités. Gauthier- Villars, 1912.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cross-entropy is all you need to invert the data generating process. arXiv preprint arXiv:2410.2 1869, 2024.
13

---

Published as a conference paper at ICLR 2026
Alfréd Rényi. On measures of dependence. Acta mathematica hungarica, 10(3-4):441-2 44-444,,,11 1959.
Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pp. 9030-9039, PMLR, 2021.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International conference on machine learning, pp. 5628-5637. PMLR, 2019.
Gabor Szeg. Orthogonal polynomials, volume 23. American Mathematical Soc., 1939.
Yonglong Tian, Chen Sun, Ben Poolc, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference machine learning, pp. 9929-9939. PMLR. 2020.
Sven-Ake Wegner. Lecture notes on high- high-dimensional data. arXiv preprint M.Xiv:2111008844 2021.
Jon Wellner et al. Weak convergence and empirical processes: with applications to statistics. Springer Science & Business Media, 2013.
Hongkang Zhang, Shao-Lun Huang, and Ercan Engin Kuruoglu. HGR correlation pooling fusion framework for recognition and classification multimodal remote sensing data. Remote Sensing, 16(10):1708, 2024.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. International conference on machine learning, pp. 1 12990. PMLR, 2021.
LLM USAGE
Portions of this manuscript, including text editing, reference search, ideation, mathematical deriva- tions, and summarization, were assisted by a large language model. The model was used interactively to refine exposition, suggest formulations, and check consistency of notation, but all results, proofs, and experiments were implemented and validated by the authors. All mathematical claims, experi- mental details, and citations were independently verified. No content was included without author review and approval.
OVERVIEW
This appendix provides complete proofs for all propositions, corollaries, lemmas, and theorems, along with additional derivations that did not fit the main text. We also include supplementary experiments and implementation details. The appendices are organized as follows:
A. Proof and details of the alignment bound (Sec. A
B. Proofs of some regularization surrogate-related claims (Sec. B
C. Proof of the aligmment plateau approach. These include general claims, some are used in the regularization surrogate proof as well (Sec. C).
D. Discussion about exact alignment bound at plateau (Sec. D
E. Experimental details (Sec.)
14

---

Published as a conference paper at ICLR 2026
A
HGR MAXIMAL CORRELATION AND THE ALIGNMENT BOUND
A.1
HGR DEFINITION AND BASIC PROPERTIES
The Hirschfeld-Geheeee Retpeeddd-GGeeeee---RnRee Rényi (HGR) maximal correlation (Hirschfeld 1935 Gehelein
1941
Rényi 1959)
between random variables X and Y is
Pm (X, Y)
sup =|
((((X)((()) [0, 1]. Var(o)=Var(H)=1
(21)
An equivalent "explained-variance characterization (Gebelein, 1941 Rényi 1959)
Var(Eig M)) Var(g(X))
(x,Y)=
sup (px) Varlg(X))>0
(22)
Here px is the marginal law of X, and (px) denotes the square-integrable (measurable) functions of X under px- The numerator is the variance explained by the optimal 7.2 predictor B[g(() |Y] and the denominator is its total variance. Hence, the ratio is a (generalized) coefficient of determination,
i... the fraction of variance of g(X) predictable from Y. in [0, 1].
HGR satisfies a (multiplicative) data-processing inequality (DPI): if X - Y - 2 is Markov chain, then
(x, 2) < Pm Pm ZY Rényi 1959).
(23)
Our representations normalized (u, v E Sd-1), hence bounded and each coordinate is in L See also the operator-theoretie theoretic interpretation (Anantharam et al. 2013).
A.2 GAUSSIAN EXAMPLE
If two random variables X and Y are jointly Gaussian, then the HGR maximal correlation between them equals the absolute value of their Pearson correlation coefficient:
A Co
Pm (X, Y) |A],
(24)
Vaar(X)aan
This is special case where the supremum in the HGR definition is achieved by simple linear functions, More precisely, the optimal transformations are just standardized versions of X and Y themselves. In other words, nonlinear functions cannot increase correlation beyond the linear one when the joint distribution Gaussian. This result is well established; see, for example, Brye & Dembo 2005).
A.3 PROOF OF THE ALIGNMENT BOUND
We prove the inequality
Elu v| I 722 + (1 12)) [m]
(25)
for normalized representations f(x) and f(Y) on sd-1, where m(n) = R[[u] R(u] is their common mean.
Step 1: mean-residual decomposition. Since u and share the same marginal μ,, their means coincide:
m(u) = -Elu] Elu].
(26)
Define residuals
ū = v - m(H). i = e m(#),
(27)
so that
[[[]] 0. Expanding the inner product yields
E uu
m] [((m(p) + ū) (m(#) + o)] [m(4)]12 + Elū 6).
(28)
The cross terms vanish because BMn] E[i] 0.
E(m(1)) .i] = m() Elū] O,
(29)
and
Blū mm(m) = Blūj m(n) 0.
(30)
15

---

Published as a conference paper at ICLR 2026
Step 2: bound the residual correlation via HGR. Fix coordinate & E {l...., d) and set (X) = Ukk
hy) = U2.
(31)
Then (X) = (Y) 0 and, by the Markov structure X - Xo - Y the DPI for HGR maximal correlation gives
Pm (X,Y) 2 Pm Pm, Pm (Xo. Y) //2 v/12 122
(32)
in Anantharam et al. (013))
For any real-valued, square-integrable functions (()), h(Y) with zero mean, we can apply the definition of HGR maximal correlation from Eq. 21) together with the Cauchy-Schwarr inequality to obtain:
([((X)((Y)) S (x, / Var(g) Var(h).
(33)
This inequality holds even when y and are not normalized, since any such functions can be rescaled to have unit variance. In our case, the random variables X and Y are conditionally independent given Xo, and identically drawn from the same augmentation channel A(- Xo). Therefore, the Markov chain X + Xo - Y holds, and the multiplicative data-proces processing inequality (Eq. 32) gives:
Pm (X,Y) I Pm (X, Xo) Pm PY, Xo)
7/2-
(34)
Substituting Eq.
34 into Eq.
33 yields:
(Xx M(Y)] & 1/2 Varlg) Var(h).
(35)
Applying Eq.
35 to (gk, h₂) and summing over coordinates.
Σ E Ük
1/2
Var(ūn Var(t)) S 12
Var(ūk)
Var(ūk); (36)
where the last step is Cauchy-Schward for sequences.
Step compute the marginal variances. Because ||u||| = ||l||| = 1 and m(p) Elu] E[e]].
Var(ūx) = [] = L[[lu m(N)]l?] = [[[[||]] [m([)]] = 1- 1-1m(()))? (37)
and identically
Var(ūx) = 1 =1
(38)
Step 4: conclude.
Combine Eq. 36
Eq. 37
and Eq.
38
to get
i
[[m[))]]]
(39)
B REGULARIZED SURROGATE PROOFS
B.1 PROOF OF PROPOSITION 3
Proof. For any encoder f with angular law y the KI. term satisfies (by the KI. chain rule, see e.g.
Dupuis & Ellis 2011 Theorem B.2.1)
kT( KL(Ullo) + / [K[(s( I (u) || E( u)) p(du).
(40)
where p(dz) (((d)))(dr u) and 7. (dzz)) (dd))((dr | u) in polar coordinates Thus, at fixed μ. the KL term is minimized by choosing xe(- u) = E((- u) p-a.s., and then (T((p (T.(ullo).
16

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF LEMMA 1
Proof. We can assume l < a, otherwise (L(ullo) = too and the claim is trivial. The claim is also trivially true if m((4) 0, so assume m(u) o O. By the Donsker-Varadhar variational formula
Dupuus & Ellis 201T. Lemma 1.4.3)
(L(pllo)= sup (umulp(())) log Buwa [estn]]}
(41)
where the supremum is taken over bounded measurable functions yp g8-1 R. Taking (p())
tw - for some unit vector w E Ie and & E IR, we have
(L(p) 2 Eunu [tw u| log Buwa [e [ww] tw m(x) log Euwa [etwm]] Suppose we showed that
(42)
log Euwa [e [e S ta/a
(43)
for some a > for every choice of and w. Then picking 2 [1(2)) and w we have
a 1m((())]' KT(u|lo) 2 tw - m())) - fa/a/aaaa a |m(x)))2 10
a 1I
(44)
is left show Eq. 43 with a 4C - 1). Now, since g(u) w is 1 1-Lipschitz on the sphere, then by corollary of Lévy's 's isoperimetric inequality, for all s 2 0,
a (1gl 2 x) 2e 1 (d-1)s
(45)
where we used the fact that the median of g Since Eg O, this implies that for some universal
C" > O,
20'²22
log E9
S d
(46)
(Vershynin 2018 Proposition 2.6.1). This satisfies Eq. 443
with a 20 and taking = 1/(80
we are done.
C ALIGNMENT-PLATEAU PROOFS
C.1 NORMALIZED REPRESENTATIONS
Lemma 2 (At the plateau the loss reduces to uniformity). Under Assumption 7 the population InfOCEE objective (Eq. 4 takes the form
I()) (()) - a E[u-r]
((1) - aa(12 + Tppat)
(47)
hence minimizing J over probability laws on gd-1 is equivalent to minimizing (u)). Moreover, (()) is uniquely minimized by the uniform law a on 1d-1
Proof. At the plateau, E[u-r] is the constant in Eq. 7 so the alignment term is independent of jr, leaving the uniformity potential (()) as the only objective. By Wang Isola 20200, Appendix A), & is uniquely minimized at the uniform distribution on the sphere, i.c. For consistency, the plateau value in Eq. 7 must be feasible at μ = a.
Remark. In Eq. 7 Tplat 0. By Eq. (6) at
(m(p) 0) the alignment ceiling is 1/21 the plateau value is not guaranteed to be feasible at f a and must be verified.
Lemma 3 Maxwell-Poincanl (Diaconis & Freedman 1984 Let Ua be uniform on Sd-1, where Ua,i denotes the i-th coordinate of Us. Fix k N. Then
Vd(Uda....., Uak)) = NNO, I2) (d + o0).
(48)
A concrete rate of convergence was given by Diaconis & Freedman 1987) Theorem 2. (Diaconis & Freedman 1987 If < k < d 4. then
2 33
drv( VoVVU,UU....... Ua,a), 2) < d 3 where 7 NN TR) and dry denotes the total variation distance.
(49)
Clearly, Lemma 3 and Theorem 2 hold for any l indices, or for any orthonormal projection of U₂ to I dimensions. Combining Lemmas and get Corollary 1
17

---

Published as a conference paper at ICLR 2026
==0 (uniform)
==1
k=3
K
10
VMF exponential titt
Figure 5: vMF exponential tilt distribution for different concentration scales kappa (x).
C.2 UNNORMALIZED REPRESENTATIONS
We now prove Proposition 2 by reducing to the normalized case established above.
Proof. Let - f(X) E Rd denote the unnormalized representation and write polar decomposition with | 0 and u 2/|22 Sd By Lemma 2 at the alignment plateau the population objective reduces to minimizing ((u), whose unique minimizer is the uniform law a on gd-1 Hence the angular component of any global minimizer satisfies
Sd-1 Assumption
further gives thin-shell concentration of the radius:
ro (0, 00).
d=
For any fixed k 2 1 and any fixed k-dimensional subspace, let Ps be the corresponding orthogonal projector and set uk == Psu. By the Maxwell-Poincanl spherical CLT (Lemma 3)
Vdu = Ix)
(d +
(50)
Let 2 = P& ru.. Since P 2000 yields
ro and Eq. (50) holds, Slutsky's theorem (an der Vaart
d+
vd rvdu: + N(O,, rg
(d + +d++0) oo).
(51)
This proves Proposition
D EXACT ALIGNMENT BOUND IN PLATEAU DISCUSSION
The following analysis begins from the alignment ceiling (Eq. 6) under generalized plateau assumption (extending Assumption D] the expected alignment is determined by the augmentation mildness 7/2 and the squared mean norm [m(())] up to negligible residual (noted Tplat in Eq. Substituting this relation into the population InfoNCE objective (Eq. yields the surrogate
Jn(m) (()) q (m)]1
a 12):
(52)
where ((u) is the uniformity potential of [Wang & Isola 2020). Thus, at the plateau, the population loss reduces to a trade-off between uniformity and the mean vector length.
Stationary points. The surrogate involves the spherical convolution operator P with kernel E
which diagonalizes in spherical harmonics by the Funk-Hecke theorem (Atkinson & Han 2012). Analyzing the Euler-Lagrange condition shows that in high dimensions Ph must asymptotically take an exponential tilt form Ph(E) x exp(3w E). Inverting this relation via Gegenbauer expansions and their decay properties (Szeg 1939 indicates that, under mild regularity, the stationary density
is well-approximated in its leading modes by either the uniform law or von Mises-Fisher (yMF) tilt (Mardia Jupp 2009). This captures the dominant low-degree structure in high dimensions, though more complex stationary forms cannot be excluded.
Implications. Consequently, in high dimension the stationary points of the plateau surrogate are well-approximated by either the uniform distribution (when m(u) 0) or Mises-Fisher (vMF) tilt aligned with an axis w (when m mp)) + 0); see Fig. The vMF concentration parameter quantifies the strength of angular concentration around (larger i narrower cone). This
18

---

Published as a conference paper at ICLR 2026
perspective helps explain why contrastive encoders often yield nearly uniform representations, with occasional vMF-like bias. For example, in CLIP, where narrow-cone structure (a modality- dependent angular bias) has been observed (Liang et al. 2022).
E EXPERIMENTAL DETAILS
E.
IMPLEMENTATION DETAILS
Code and reproducibility. Code is available here All experiments were implemented in PyTorch with torchvision. Training was performed on single 3090 NVIDIA RTX GPU with CUDA 11.8.
Synthetic Laplace data experiments.
Dataset. Laplace(O, 1) vectors of dimensions ddata 1024. We of 20k samples for training, and 5k samples for testing.
Representation dimensions. The dimensions of representations vary: d E {32, 64, 128, 256).
Batch size. Batch size in our experiments varies: N {8, 16, 48, 64, 96, 128).
Training objective. InfoNCE loss with temperature {0.1, 0.2). We report results for T = 0.1. but note that results are similar.
Augmentations. Fach synthetic sample æ is perturbed to form two correlated views
21 = Ax+ V A1
= Ar + v 1 A² E2.
(53)
where E1. No0, I) are independent. The parameter A E (0, 1) controls the correlation between views. After this linear Gaussian mixing, we apply light, independent jitter: additive Gaussian noise with std 0.2, feature dropout with probability 0.1, and random multiplicative scaling by exp(A 0.12)). Unless otherwise stated, we use A 0.6 (results for A E {0.2, 0.5, 0.8) appear in Fig.
Optimization. Optimizer: Adam. Learning rate 10-3 We ran 50-250 epochs depending on setup; unless stated otherwise, we report results at 1 150 epochs.
Evaluation metrics. norm concentration (CV), mean norm values, Gaussianity diagnostics (AD/DP) tests and uniformity vs. alignment comparison (based on cosine similarity).
Additional synthetic data experiments.
For the Gaussian mixture setting:
We generate 10k samples in Idd from a mixture 25 equally weighted Gaussian components (1024 dimensions) with randomly sampled means and shared isotropic covari- ance. The component means are drawn independently at initialization and fixed throughout training.
Dataset.
Augmentation. Positive pairs are generated by independently sampling views from the same underlying mixture component.
Training. A linear encoder maps inputs to a 256-dimensional representation space and is trained using the InfoNCE objective for 100 epochs.
Evaluation metries. Normality diagnostics include norm concentration (CV) and coordinate-wise AD and DP statistics.
For the discrete binary setting:
Dataset. Each sample a sparse binary vector dimension 1024.
Augmentation. Positive pairs are generated by independently flipping a small fraction (0.1 1%) of zero entries to ones.
19

---

Published as a conference paper at ICLR 2026
Supervised
Contrastive
DenseNet
ResNet
CLIP; Text
CLIP: Image
0.2
DINO
Densiey
0.1
Densley
0.1
Densieg
0.5
Denage D
Densier
0.1
CV=0.115
CV-0.118
CV=0.055
CV+0.054
VV0.039
0
50
100
0
50
100
50
100
0
50
100
50
100
Figure 6: Thin-shell concentration across pretrained models. Radius distributions of representa- tions from supervised models (DenseNet, ResNet) and contrastive models (CLIP, DINO). All models exhibit thin-shell concentration, with contrastive methods showing tighter clustering (lower CV,
Eq. (20
Training. A linear encoder maps inputs a 256-dimensional representation and is trained using the InfoNCE objective for 100 epochs.
Evaluation metrics. Every 10 epochs, we evaluate normality diagnostics, including norm concentration (CV) and coordinate-win AD and DP statistics.
CIFAR-10 experiments
Dataset. CIFAR-I0, training set size 50k, test set size 10k.
Augmentations. We apply the standard MenCLR-style augmentation pipeline: random resized crop to 32 x 32 pixels with scale uniformly sampled from (0.2, 1.0), random horizontal flip, color jitter with strengths (0.8, 0.8, 0.8, 0.2), and random conversion to grayscale with probability 0.2.
Architecture. Two experimental settings: (i) basic encoder composed of two-layer MLP with nonlinearity, and (ii) ResNet-18 18 encoder following the SimCLR protocol for the contrastive setting and trained with standard cross-entropy for the supervised setting (pretrained on ImageNet (Deng et al.) 2009)).
Training objective. InfoNCE with temperature 0.1 (cross-entropy for the supervised setting).
Optimization. Adam optimizer, learning rate 10 weight decay = 10 14, batch size 256, epochs 100.
Evaluation metrics. norm concentration (CV), Gaussianity diagnostics (AD/DP) tests.
Pretrained model diagnostics
Models. CLIP (ViT-L/14, text and image modalities), DINO (ViT-B/32), ResNet-34 and DenseNet.
Full MS-COCO validation set (5k images) and the full ImageNet-R benchmark when noted.
Datasets.
Feature extraction. Last-layer embeddings; whitening applied when noted.
Evaluation metries. norm concentration (CV), Gaussianity diagnostics (AD/DP) tests and uniformity before and after whitening.
E.2 ADDITIONAL EXPERIMENTS
Thin-shell concentration in pretrained models. Fig. 6 visualizes the radius distributions of rep- resentations from supervised (DenseNet, ResNet34) and self-supervised (CLIP image/text, DINO) pretrained models on MS-COCO. All models exhibit thin-shell concentration, with radius values tightly clustered around a characteristic norm. Notably, contrastive models display significantly stronger concentration (lower CV) than supervised counterparts, consistent with the Gaussian di- agnostics reported Table This reinforces the underlying near- Gaar-Gans structure observed in self-supervised representations.
20

---

Published as a conference paper at ICLR 2026
Dim 512
Dim 512
Dir 32
0.8
0.10
0.10
uniFormiti
alignment
0.5
uniFormity
0.05
0.05
120
0.00
-0.5
-0.2
0.2
0.5
1.0
-0.5
-0.2
0.2
0.5
-0.5
-0.2
0.2
0.5
Casine simllarity
Casine similarity
Cosine simliavity
Figure Alignment and uniformity vs. batch size. Histogram view of cosine similarities for positive pairs (alignment) and negatives (uniformity), corresponding to Fig. 2 As batch size increases, alignment remains high while uniformity improves, with aegative-pair similarities concentrating near zero. The middle panel is zoom of the left; the right panel shows that very low dimensionality, increasing batch size yields little uniformity gain.
Beech size 128
Baech size 128
Baech size
0.8
0.10
0.10
uniformitd
alignment
Di Cimensions
0.5
uniformity
0.05
0.05
0.2
005-0.2
0.2 0.5
1.0
0.00
0.00
-05
Casine similarity
Cosine similaxicy
Cosine similaarty
Figure 8: Alignment and uniformity vs. dimensionality. Histogram view of cosine similarities for
positive pairs (alignment) and negatives (uniformity), corresponding Fig. 2 As dimensionality increases, alignment stays high while uniformity improves, pushing negative-pan pair similarities toward zero. The middle panel is zoom of the left; the right panel highlights that with very small batch sizes, increasing dimensionality offers limited uniformity improvement.
Figs. and 8 provide alternative visualizations of Fig. presenting the same experiments with different display. Both figures plot the distributions of cosine similarities for positive pairs (alignment) and for negatives (uniformity). As batch size (Fig. 7 or dimensionality (Fig. { increases, uniformity improves (negative-pair similarities concentrate near zero) while alignment remains consistently high across settings. These complementary views reinforce the observation from the main body: uniformity continues improve with larger batches and higher dimensions, whereas alignment appears to saturate early.
0.02
0.02
Rism
0.01
Rasi
Wirisene
Whitened
0.015
0.015
CLiP:Image
Density
CLIP,Text
Densiey
DINO
Density
0.01
0.005
0.01
0.005
0.005
0
0.5
-0.5
0
0.5
0.5 cosine similarity
cosine similarity
cosine simitarity
Figure Whitening and uniformity: unnormalized representations. Cosine similarity histograms of negatives for CL.PP (image, text) and DINO, before (raw) and after whitening. Unnormalized representations benefit from whitening, with distributions pushed closer to zero, reflecting enhanced uniformity. The y-axis ("Density")) represents the relative count in each bin (normalized by the total number samples) rather than the probability density function.
21

---

Published as a conference paper at ICLR 2026
x10
x x10
x10
15
Row Wiisered
Wintened
Wintened
1.5
1.5
counte
counre
count
10
1
CLIP; Image
CLIP; Text
5
DINO
0.5
0
0
0.5
-0.5
0.5
0
0.5
cosine simitariey
cosine simitariey
Cosine simitarity
Figure 10: Whitening and uniformity: normalized representations. Cosine similarity histograms of negatives for CLIP (image, text) and DINO, before (normalized) and after whitening. Normalized representations are already close uniform; whitening provides modest but consistent improvement.
Additionally, we assess uniformity in several pretrained models before and after whitening. Whitening consistently increases uniformity, indicating that these representations, which are already close to uniform (and approximately Gaussian; see Table 2). become more isotropic once decorrelated and rescaled. This effect holds consistently across pretrained models (CLIP image, CLIP text, and DINO), for both normalized and unnormalized representations, see Figs.
9 10
Thus, simple post hoc
projection via whitening can further enhance uniformity in practice.
Pearson=0.87
Pearson=0.86
Pearson=0.87
Log-Pidatal
-820
Log-Pidatal
-825
Log-Prdatal
-850
-850
-900
-925
-920
-950
0
0.2
-550
0.5
-940
P 0.8
750
-00
-550
750 Log Prembedddds
Log-Prembeddings
Log-Prembeddings)
Figure 11: Encoder "pushforward". On synthetic data, the encoder maps Laplace-disttibutee inputs to approximately Gaussian representations. Because both source and target families admit tractable likelihoods, we can score entire sets and observe consistently high correlation across different augmentation strengths.
We examine the correlation between the data distribution and the representation distribution. Using Laplace data as input and observing Gaussian representations the output, hoods for both input and output sets. Comparing these scores reveals strong correlation (Fig. [T]. indicating that the distribution indeed "pushed forward" through the encoder. This correlation remains stable across different augmentation strengths, showing that this "pushforward" behavior is insensitive to the level of augmentation.
compute likeli-
22


Reviews:
[{"review_id": "Absp8QDB7z", "reviewer": "Reviewer_j58r", "summary": "The paper presents theory that representations obtained with contrastive learning yield \"Gaussian representations\". The authors theoretially investigate this claim for normalized and unnormalized contrastive learning techniques and present a small scale theoretical study across popular contrastive learning models in computer vision.", "questions": "1. I might have a fundamental misunderstanding here, but the sentence \"Overall, we provide the first principled explanation for Gaussianity in contrastive representations\" seems off. Can you comment on how your theory relates to identifiability theory for contrastive learning, which \n2. What is the null hypothesis against the Gaussian distribution that is being observed, what would be the alternative? Will the Gaussian distribution also appear when the underlying data-generating process is non-gaussian distributed, or cannot me mapped into a feature space which is Gaussian-distributed?\n3 What is a \"Gaussian projection\"? (L 191) I assume this is a low-D projection of the full feature space on a 1D axis, where Gaussianity is tested? Doesn't the Gaussianity of a projection then trivially follow from the law of large numbers? If not, could you highlight why this finding is significant?\n4. What is the expectation in Eq. 8 over?\n5. Consider a mixture of Gaussian distribution, or a vMF mixture where each component represents one class; can you comment if the marginal distribution expected through contrastive learning for this case would be Gaussian? Previous work seems to suggest that a mixture of Gaussians would be recovered, and this seems to conflict with the presented theory here; could you comment?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors connect new theory on properties of contrastive representations with empirical experiments covering real world models\n- The paper is dense with theoretical results, but still fairly well structured and easy to follow\n- The authors release code for reproducibility", "weaknesses": "1. It seems like the authors missed existing identifiability literature for contrastive learning, e.g. https://arxiv.org/abs/1605.06336, https://proceedings.mlr.press/v54/hyvarinen17a/hyvarinen17a.pdf, https://arxiv.org/abs/1805.08651, https://proceedings.mlr.press/v139/zimmermann21a.html, https://arxiv.org/abs/2007.00810, https://arxiv.org/pdf/2410.21869. It would be good to discuss how the presented theory (which uses a different set of tools) relates to this work; the presented work does not talk about the data generating process, but I wonder if the results are fundamentally connected to identifiability; Gaussian representations (or their projections to the sphere) is what we would expect with the considered form of the InfoNCE loss (in particular cf. https://proceedings.mlr.press/v139/zimmermann21a.html) for this.\n2. \"In particular, the raw unnormalized representations have received little theoretical attention\": This statement seems ungrounded, indeed a lot of identifiability theory also studies the unnormalized case (see above).\n3. The experiments are not sufficiently rigorous to support the theory. It is unclear to me what the null hypothesis is, and how the authors test for Gaussianity. It would be quite helpful if a study with simulated data would be conducted that precisely illustrates the theoretical claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j8MFmWygz4", "reviewer": "Reviewer_BFsh", "summary": "This paper studies the distribution of the optimal representation obtained by optimizing the InfoNCE objective. It demonstrates, under alignment and concentration assumptions, that finite projections of the high-dimensional representation are Gaussian. The authors then weaken their alignment and concentration assumption to obtain a similar result. Finally, they give empirical evidence to confirm they claim on widely used pre-trained contrastive models (CLIP) and non-contrastive (DINO) models.", "questions": "Most of my questions are described in the weaknesses. I believe answering them will improve the quality of the manuscript, in particular regarding the match to the current practice in contrastive learning.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "-\tThe mathematical analysis is rigorous, the first claims (Corollary 1 and Proposition 2) are strong (even if they are directly obtained from two well-known results) and the section 4.2 is very technical but sounded. \n-\tThe empirical evidence given for real-world contrastive-based models gives credit to the theoretical analysis. There are not many works testing the Gaussian assumption on the representations of foundation models while it is often assumed for downstream applications (e.g. OOD).\n-\tThe exposition is clear, concise, and easy to follow (excepted maybe for section 4.2 which requires special attention for the numerous notations…).", "weaknesses": "-\tRegarding Assumption 3, the authors mentioned it is weaker than previous Assumption 1. It is still not clear to me the realizability of such assumption and why it would hold in practice. Assumption 1 may be easier to verify empirically (as the authors already mentioned) and it seems more intuitive.\n-\tIn the experiments, you consider DINO, a non-contrastive approach that does not introduce any uniformity term in its loss (which is key in your analysis since you always assume the alignment to be controlled by a constant). It is interesting to note that DINO representation seems to be Gaussian as well, but you do not justify it theoretically. Do you have any theoretical analysis linking the knowledge distillation technique in DINO with the uniformity loss optimized by InfoNCE? \n-\tIn your experiments, you also consider CLIP, which is a constative approach, but it does not fit well in your theoretical analysis. Indeed, the vision and text encoders are different (they do not share the same weights, and they are not defined in the same space) and the marginal distribution for each modality is different. Therefore, your uniformity term in eq. 5 seems to not fit this case. It is again interesting to note that image and text representations are Gaussian but you do not justify it theoretically.\n-\tFinally, it is funny to observe that the only foundation model that fits perfectly your analysis (SimCLR) is not present in your experiments!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YkxDxSnlaB", "reviewer": "Reviewer_vUPT", "summary": "The authors theoretically and empirically analyze representations learned by the InfoNCE contrastive loss, showing they become approximately Gaussian in high dimensions. They present two approaches to demonstrate their claim. First, under two assumptions: an “alignment plateau” assumption and a \"thin-shell\" assumption, the population-level InfoNCE solution is isotropic, so normalized features are uniform on the sphere and any fixed-$k$ projection is asymptotically Gaussian as dimension $d\\to\\infty$. Second, even under weaker conditions, adding a vanishing $\\ell_2$-norm/entropy regularizer biases the solution toward isotropy, yielding the same asymptotic Gaussian law. Empirically, the authors validate these claims on synthetic data, CIFAR-10, and pretrained models (e.g. CLIP, DINO). They measure thin-shell concentration of feature norms and apply normality tests (Anderson–Darling and D’Agostino–Pearson) to demonstrate that InfoNCE-trained features indeed exhibit Gaussian statistics in practice. The paper argues this formalizes prior observations of “near-Gaussian” features in contrastive learning and justifies Gaussian modeling for downstream tasks (e.g. OOD detection).", "questions": "- In practice, how does one verify the “alignment plateau” condition during training? Can alignment be measured online to check this assumption? Is the plateau different given different images ? Does that mean something about the quantity of information carried by the image ?\n\n- The vanishing regularizer is claimed to ensure isotropy at finite $d$. How should its strength be chosen in practice? Did you experiment with non-vanishing regularizers, and if so, how did they affect downstream performance?\n\n- The work emphasizes normalized (unit) features. Does the Gaussianity hold equally for unnormalized (pre-norm) embeddings? Fig.3 and authors remarks suggests yes, but clarity on this point would be good. It is unclear why would upstream representations (like the one used for downstream tasks, typically before the projection MLP 2 layers) would exhibit a multi-dimensional Gaussian distribution as well.\n\n- Do these results extend to other contrastive losses ? Empirically, on CLIP and DINO it appears to, but what is the theoretical argument to support this behaviour ? Is the Gaussian outcome specific to InfoNCE batch training?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- Theoretical rigor: The paper provides a novel and rigorous analysis using high-dimensional probability and spherical CLT tools (e.g. Hirschfeld–Gebelein–Rényi maximal correlation bound, polar KL decomposition, Maxwell–Poincaré CLT). It derives precise conditions (bounded alignment, uniformity on sphere) under which InfoNCE yields isotropic, Gaussian outputs.\n\n- Comprehensive empirical validation: The authors evaluate across diverse settings: a synthetic Laplace dataset, CIFAR-10 with a small encoder, and large pretrained models (supervised ResNet/DenseNet vs self-supervised CLIP/DINO). They consistently observe thin-shell norm concentration and high normality p‑values for InfoNCE-trained features (vs non-Gaussian behavior for supervised models).\n\n- Clarity and organization: The paper is well-structured. Key assumptions and results are clearly stated (e.g. via bullet-point summary of contributions on p.2). The exposition links the theory to concrete phenomena (alignment saturation, feature variance) and shows illustrative figures (e.g. Figure 3) that make the findings intuitive. Definitions and proofs are thorough and mostly easy to follow.", "weaknesses": "- Strong asymptotic assumptions: The theory relies on high-dimensional limits and idealized assumptions (infinite negatives, alignment plateau, perfect norm concentration). These may not hold in all practical cases. The authors acknowledge this, noting the results are asymptotic and “alignment plateau and thin-shell concentration… are not guaranteed to hold universally”. It remains unclear how well the Gaussian approximation holds for moderate $d$ or when these assumptions fail.\n\n- Dependence on regularizer: One route to Gaussianity uses an added regularizer (low-norm, high-entropy) that vanishes asymptotically. It’s not fully clear how sensitive the results are to this term in finite dimensions. Practical guidelines for choosing (or removing) this regularizer without harming performance are not discussed. Besides, it would have been interesting to study the impact of this regularizer on downstream classification performances on one or more different datasets (CIFAR10, STL10, IMAGENET100, IMAGENET1K).\n\n- Scope of experiments: Although varied, the experiments mostly focus on vision models and one dataset (MS-COCO) for pretrained features. It would strengthen the paper to test more domains (e.g. other modalities or tasks or non-natural images) to see if Gaussianity is universal for InfoNCE. Additionally, experiments on how violations of assumptions affect outcomes (e.g. weak augmentations) are not shown.\n\n- Overall, the Gaussianity of the latent space is not so surprising, it has been already shown in Wang and Isola, 2021 that the representations would converge toward a uniform hyperspherical distribution. And it is commonly known that there is a close connection between uniform hyperspherical distribution (von-Mises Fisher with concentration equals to 0) and high-dimensional Gaussian distributions known as the soap-bubble-effect as stated by Vershynin, 2018; Lecture Notes on High-Dimensional Data, Sven-Ake Wegner, https://arxiv.org/pdf/2101.05841; Hyperspherical Variational Auto-Encoders, https://auai.org/uai2018/proceedings/papers/309.pdf Davidson et al. 2018; Farquhar et al, 2021 Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning, https://arxiv.org/abs/1907.00865. The paper would benefit discussing these results and these papers in order to clearly state their contribution.\n\n- It is unclear why CLIP and DINO (what about v2 or v3 ?) exhibits Gaussian distribution properties even though they were not exactly trained like SimCLR: CLIP positive-pair probably implies different alignment propoerties than SimCLR, and DINO's loss is not exactly the same as SimCLR, the paper would benefit from a better discussion about other self-supervised methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5z8yRhgsdn", "reviewer": "Reviewer_yqFV", "summary": "This paper investigates the statistical structure of representations learned through contrastive learning with the InfoNCE loss, and demonstrates that such representations closely follow a multivariate Gaussian distribution. The authors provide two theoretical justifications: (1) under alignment and concentration assumptions, high-dimensional embeddings asymptotically approach Gaussianity; and (2) under weaker assumptions, introducing a small regularization term that encourages low feature norms and high entropy leads to similar asymptotic behavior. Theoretical analysis is further supported by experiments on synthetic data, CIFAR-10, and pretrained foundation models, consistently validating the Gaussian property of learned representations.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tTheoretical and empirical results align well. The experiments on synthetic datasets, CIFAR-10, and pretrained models convincingly support the Gaussianity of contrastive representations.\n2.\tThe paper is well-organized, with a clear logical flow that makes both the theoretical derivations and empirical validations easy to follow.\n3.\tThe work introduces a novel and meaningful perspective for understanding the contrastive learning objective, potentially offering a new theoretical framework for future studies in this area.", "weaknesses": "1.\tWhile the theoretical findings are solid, the paper could benefit from more discussion on how the Gaussianity of learned representations may translate into practical insights or applications. In the introduction, the authors claim that understanding representation distributions is not only of theoretical insights. However, the main body does not explore how this insight could improve contrastive learning methods or be leveraged in practice  This is my main concern.\n2.\tThe claim that contrastive representations follow a Gaussian distribution offers a novel theoretical view. However, it would strengthen the paper to compare this finding to prior work that also studies the representations of contrastive embeddings—such as [1] and [2], which show that contrastive features tend to form well-separated clusters aligned with supervised labels. These perspectives seem complementary: clustering and Gaussianity both describe structural regularities in learned features. Intuitively, it seems that their results could provide more guidance for our application as contrastive learning (as classification is one of the most important application and metric of learned representations). I wonder what are the advantages of the theoretical analysis in this paper. A clearer comparison could help clarify the added value or distinct implications of the Gaussian model over existing analyses.\n3.\tThe paper focuses on the unnormalized representation space, which is an interesting and often underexplored aspect. Most contrastive frameworks compute losses in the normalized space, while fine-tuning typically uses unnormalized embeddings. It would be valuable if the authors could discuss whether their theoretical analysis provides any insight into this transition—e.g., why Gaussianity might hold or break across normalized and unnormalized spaces, and what this implies for the design of pretraining-finetuning pipelines.\n4.\tThe paper progressively relaxes its assumptions. Nevertheless, it remains unclear how strong or realistic these assumptions are in practice. It would be helpful for the authors to clarify the degree of relaxation in each step and to discuss how far the theoretical conditions deviate from those typically observed in real contrastive learning scenarios. More empirical verifications or discussions of assumption validity could make the theoretical results more convincing and grounded.\n\n[1] Saunshi, Nikunj, et al. \"A theoretical analysis of contrastive unsupervised representation learning.\" International conference on machine learning. PMLR, 2019.\n\n[2]  HaoChen J Z, Wei C, Gaidon A, et al. Provable guarantees for self-supervised deep learning with spectral contrastive loss[J]. NeurIPS, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)