Rethinking Uniformity in Self-Supervised Representation Learning
Authors: Xianghong Fang
Jian Li
Xiangchu Feng
Benyou Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Collapse Analysis
Wasserstein Distance
Self-Supervised Representation Learning

Abstract: Self-supervised representation learning has achieved great success in many machine learning tasks. While many research efforts focus on learning better representations by preventing the model from the \emph{collapse} problem, less attention has been drawn to analyzing the collapse degrees of representations. In this paper, we present a formal study of collapse analysis via the \emph{uniformity} metric, which measures how uniformly learned representations distribute on the surface of the unit hypersphere. We fundamentally find that \textit{representation that obeys zero-mean isotropic Gaussian distribution is with the ideal uniformity} since its $l_2$-normalized form uniformly distributes on the surface of the unit hypersphere. Therefore, we propose to use the Wasserstein distance between the distribution of learned representations and the ideal distribution as a quantifiable metric of \emph{uniformity}. Moreover, we design five desirable constraints for ideal uniformity metrics, based on which we find that the proposed uniformity metric satisfies all constraints while the existing one does not. Synthetic experiments also demonstrate the proposed uniformity metric is capable to deal with the dimensional collapse while the existing one is insensitive. Furthermore, we impose the proposed \emph{uniformity} metric as an auxiliary loss term for various existing self-supervised methods, which consistently improves the downstream performance. 

Under review as a conference paper at ICLR 2023
RETHINKING UNIFORMITY IN SELF-SUPERVISED REP- RESENTATION LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Self-supervised representation learning has achieved great success in many ma- chine learning tasks. Many research efforts tends to learn better representations by preventing the model from the collapse problem. [Wang & Isola (2020) open new perspective by introducing a uniformity metric to measure collapse degrees of representations. However, metric is insensitive to the dimensional collapse. Inspired by the finding that rep- resentation that obeys zero-mean isotropic Gaussian distribution with the ideal uniformity, we propose to use the Wasserstein distance between the distribution of learned representations and its ideal distribution with maximum uniformity as a quantifiable metric of uniformity. To analyze the capacity on capturing sensi- tivity to the dimensional collapse, we design five desirable constraints for ideal uniformity metrics, based on which we find that the proposed uniformity metric satisfies constraints while the existing one does not. Synthetic experiments also demonstrate that the proposed uniformity metric is capable to distinguish different dimensional collapse degrees while the cxisting one in (Wang & Isola 2020) is insensitive. Finally, impose the proposed uniformity metric auxiliary loss term for various existing self-superviee methods, which consistently improves the downstream performance.
theoretically and empirically demonstrate this
INTRODUCTION
Self-supervised representation learning has become increasingly popular in machine learning com- munity (Chen et al. 2020: He et al.] 2020 Caron et al.] 2020 Grill et al. 2020 Chen & He 2021 Zbontar et al. 2021| ). and achieved impressive results in various tasks such as object detection.
segmentation, and text classification (Xie et al. 2021 Wang et al.) 202 Yang et al. 2021 Zhao (et al. 202T Wang et al. 2021 Gunel et al. 202 1). Aiming to learn representations that are invariant under different augmentations, common practice of self-supervised learning is maximize the similarity representations obtained from different augmented versions sample by using a Siamese network (Bromley al. 1994 Hadsell al., 2006). However, common issue with this approach is the existence of trivial constant solutions that all representations collapse to a constant point (Chen & He, (021). visualized in Fig. [ known as the collapse problem (Jing al. 2022).
Many efforts have been made to prevent the vanilla Siamese network from the collapse problem. The well- known solutions can be summarized into three types: con- trastive learning (Chen al. 22200) He al., 22200; Caron et al. 2020), asymmetric model architecture (Grill et al.) 2020 Chen & He (021), and redundancy reduction (Zbon- tar et al.) 2021 Zhang et al. 2022b). While these solutions could avoid the complete constant collapse, they might still suffer from a dimensional collapse (Hua et al., 2021) in which representations occupy a lower-dimensional sub- space instead of the entire available embedding space (Jing jet al., 2022). as depicted the Fig. [ Therefore, to show the effectiveness of the aforementioned approaches, need quantifiable metric measure the collapse degree of learned representations.
Consbnnt Collapse
Dimensional Collapse
The left figure presents constant collapse, and the right figure visualizes di- mensional collapse.
Figure 1:

---

Under review as a conference paper at ICLR 2023
To gain quantifiable analysis of collapse degree, recent works (Arora et al. 2019 Wang & Isola 2020) propose to divide the loss function into alignment and uniformity terms. For instance, recent objective functions such as InfoNCE (van den Oord et al. 2018) and cross-correlation employed in Barlow Twins (Zbontar et al. 2021) could be divided into two terms. These uniformity terms could explain the degree of collapse to some extent, since they measure the variability of learned representations (Zbontar et al.) (021). However, the calculation of these uniformity terms relies on the choice of anchor-positive pair, making them hard to be used as general metrics. Wang et al (Wang & Isola, 2020) further propose a formal definition of uniformity metric via Radial Basis Function (RBF) kemel (Cohn & Kumar 2007). Despite its usefulness (Gao et al.) 2021 Zhou et al. 2022), we theoretically and empirically demonstrate that the metric is insensitive to the dimensional collapse.
In this paper, we focus on designing new uniformity metric that could capture salient sensitivity to the dimensional collapse. Towards this end, we firstly introduce an interesting finding that representation that obeys zero-mean isotropic Gaussian distribution is with the ideal uniformity. Based on this finding, we use the Wasserstein distance between the distribution of learned representations and the ideal distribution as the metric of uniformity. By checking on five well-designed desirable properties (called 'desiderata') )) of uniformity, satisfies all desiderata while the existing one (Wang & Isola (020) does not. Synthetic experiments also demonstrate the proposed uniformity metric is capable to quantitatively distinguish various dimensional collapse degrees while the existing one is insensitive. Lastly, we apply our proposed uniformity metric in the practical scenarios, namely, imposing it as an auxiliary loss term for various existing self-supervised methods, which consistently improves the downstream performance.
theoretically demonstrate the proposed uniformity metric
The contributions of this work are summarized as: (i) We theoretically and empirically demonstrate the existing uniformity metric [Wang & Isola 2020) is insensitive to the dimensional collapse, and we propose new uniformity metric that could capture salient sensitivity the dimensional collapse; (ii) By designing five desirable properties, we open a new perspective to rethink the ideal uniformity metrics; (iii) Our proposed uniformity metric can be applied as an auxiliary loss term various self-supervised methods, which consistently improves the performance in downstream tasks.
BACKGROUND
2.1 SELF-SUPERVISED REPRESENTATION LEARNING
Self- self-suuer representation learning aims learn representations that are invariant to series of different augmentations. Towards this end, a common practice is to maximize the similarity of representations obtained from different augmented versions of sample. Specially, given a set of data samples {x1, X2, ..., x, ). a symmetric network architecture, also called Siamese network (Hadsell al., 2006). takes as input two randomly augmented views x; and x² from input sample x,. Then the two views are processed by an encoder network f consisting of backbone (e.g., ResNet (He) ctt al. 2016)) and a projection MLP head (Chen al.) 2020), denoted as g. To enforce invariance to representations of two views z é (x()) )) and z - g(f(x))), a natural solution is to maximize the cosine similarity between representations of two views, and Mean Square Error (MSE) is widely used loss function align their 2 normallze representations on the surface of the unit hypersphere:
z
(z 2!)
Catign
11 =
(1)
|
|
||
|
However, a common issue with this approach easily learns an undesired trivial solution that all representations collapse to a constant, as depicted in Fig. T
2.2 EXISTING SOLUTIONS TO CONSTANT COLLAPSE
To prevent the Siamese network from the constant collapse, existing well-known solutions can be summarized into three types: contrastive learning, asymmetric model architecture, and redundancy reduction. More details will be explained in this section.
Contrastive Learning Contrastive learning is one effective way to avoid constant collapse, and the core idea is repulse negative pairs while attracting positive pairs. SimCLR (Chen et al. 2020) is
2

---

Under review as a conference paper at ICLR 2023
one of the most representative works, which first proposes an in-batch negative trick that employs samples in a batch as negative samples. However, its effectiveness heavily relies on the large batch size. To overcome the limitation, MMCo (He et al. 2020) proposes memory bank to save more representations as negative samples. Besides instance-wise contrastive learning approaches, some recent works also propose clustering-basee contrastive learning by bringing together clustering objective with contrastive learning (Li et al. 2021 Caron et al. 2020).
Asymmetric Model Architecture Asymmetric model architecture is another approach to prevent constant collapse, the core idea is to break the symmetry of the Siamese network. A possible explanation is that asymmetric architecture could encourage encoding more information (Grill et al. 2020). To keep asymmetry, BYOL (Grill et al. 2020 proposes to use an extra predictor in one branch of the Siamese network. and use momentum update and stop-gradient operator in the other branch. An interesting work DINO (Caron et al. 202T applies this asymmetry in two encoders, and distills knowledge from the momentum encoder to another branch (Hinton et al. 2015). Chen et al. propose SimSiam (Chen & He, 202T) by removing the momentum update from BOL, its success demonstrates the momentum update is not the key to preventing collapse. Mirror-SimSian (Zhang) al. 2022a) further swap the diop-gradien operator to the other branch, its failure refutes the claim in SimSiam (Chen & He 202I), that the tlop-gradient operator is the key component to preventing the model from collapse.
Redundancy Reduction The principle for redundancy reduction to prevent constant collapse to maximize the information content of the representations. The core is to achieve decorrelation by making the matrix based on representations as close to the identity matrix as possible. Barlow Twins (Zbontar et al. (222)) tries to achieve this end on the cross-correlation matrix, while VI- CReg (Bardes et 2022) chooses on the covariance matrix. Instead of applying regularization to al. 2021) employs direct way make the covariance matrix equal
the matrix, to the identity matrix via feature- wise whitening. Zero-CL (Zhang et al. 2022b) further proposes the hybrid of instance-wise and feature-wise whitening achieve this end.
2.3 COLLAPSE ANALYSIS
While aforementioned solutions could effectively prevent model from constant collapse, they might still suffer from the dimensional collapse in which representations occupy a lower-dimensional subspace instead of the entire available embedding space, depicted in the Fig. [ The evidence of dimensional collapse was identified in contrastive learning by singular value spectrum of representa- tions (Jing et al. 2022). However, the singular value spectrum is in the form of pictures, making hard to conduct statistical comparisons among various approaches in terms of collapse analysis.
To gain quantifiable analysis of collapse degree, Wang et al. propose a formal definition of uniformity metric in (Wang& & Isola, 2020). via Radial Basis Function (RBF) kernel (Cohn & Kumar 2007). More specially, given a set of representation vectors (21, 22,... ..., ?n. } (z; E RR"), the uniformity metric is defined as follows:
1-1
Cu log n(n - 1)/2
e
tt> O.
(2)
i= j=1
Where t is fixed parameter (generally = 2). Despite its usefulness, we theoretically and empirically demonstrate this metric insensitive to the dimensional collapse in Sec. 4.2 and Sec. 4.3
3 A NEW UNIFORMITY METRIC
In this section, we focus on designing new uniformity metric that could capture salient sensitivity to the dimensional collapse. In Sec. 3.1 Sec.B.1| we introduce an interesting finding that the maximum uniformity could be achieved when learned representations obey zero-mean isotropic Gaussian distribution. To enforce the uniqueness of the ideal distribution, we adopt its Ig-normalized form. Interestingly, we theoretically and empirically demonstrate it is approximately Gaussian distribution in 3.2. Based on this principle, we propose to use Wasserstein distance between the distribution of learned representations and its ideal distribution as uniformity metric in Sec.
3

---

Under review as a conference paper at ICLR 2023
3.1 ZERO-MEAN ISOTROPIC GAUSSIAN DISTRIBUTION, THE MAXIMUM UNIFORMITY
we provide a theorem that states maximum uniformity could be achieved learned representations obey zero-mean isotropic Gaussian distribution (Z NO o I²))).
As shown in Theorem D
Theorem 1. Let random variable Z NN0, Im) (Z R its Iz-normalized form Y 2/||Z||2 uniformly distribute on the surface of umil hypersphere Sm-1 See App. A for the proof.
However, obeying zero-mest isotropic Gaussian distribution is a sufficient but not necessary for an ideal uniformity of its l2 lg-normalized form. For example, stated in Corollary [] independent random variables following zero-mean isotropic Gaussian distribution also achieves the ideal uniformity.
mixture of two
Corollary 1. For random variable Z, Z2 E T that both follow Gaussian distributions. Namely, Z1 ~ (0, (Im)) and Z2 ~ N (0, Im)) Let Z be a mixture distribution derived from Z₁ and Z2 with any binary selection probabilities. Its lz-normalized form Y Z |Z|| also uniformly distribute on the surface of the unit hypersphere St--1
The distribution to achieve ideal uniformity (i.e., the mixture of various zero-mean isotropic Gaussian distributions) is not unique due to the mixture form discussed in Corollary [I] in sense one could define different mixtures of gero-mean isotropic Gaussian distributions, each of which might have different norms encapsulated in ",,". -k,. Therefore, we turn to investigate the [2-normalized form of these zero-mean isotropic Gaussian distributions see Sec. 3.2
3.2 ON THE 12- IO-NORMADD GAUSSTAN DISTRIBUTION
This section will discuss the characteristics regarding a Iz-normalized distribution of a Gaussian distribution mixture, which is found to be close Gaussian distribution N(O,, 1 I.), from both theoretical aspect (in Sec. 3.2.I and an empirical aspect (in Sec. (.2.2).
m
3.2.1 THEORETICAL CONNECTION BETWEEN Y AND THE GAUSSIAN DISTRIBUTION
For simplicity, we denote the 12- n2-oomaalzed form of zero-mean isotropic Gaussian distributions as Y. Y Z//ZZ 12- Note that Y obeys uniform distribution on the surface of the unit hypersphere Y ~ U.. Interestingly, we found Y approximates Gaussian distribution N(O, Imm) when m large enough. Particularly, each dimension of Y, denoted as Y,, degrades to a Gaussian distribution (0, L) in terms of the Kullback-Leibler divergence when m is large enough, see Theorem
Theorem For random variable Y; in the i-th dimension of Z/I where Z NOO, I²Im) ) (Z E I)"),, then the Kullback-Leibler divergence between Y, and the variable Y; (0, converges to zero as - as follows.
lim DKL(Y =0
We firstly seek the probability density function (pdf) of Y, shown in App. C Since the probability density functions of both distributions are known, we could derive the Kullback-Leibler divergence between them. One trick is to expand logarithm term using Taylor expansion. Finally, obtain that the divergence has a limit of zero when m approaches infinity (Theorem proved). See App. D for the detailed proof.
A mixture distribution is the probability distribution of random variable that is derived from collection of other random variables. This could be implemented by first sampling random variable based given probability distribution w.r.t a ratio of each random variable, and then sampling a value based the selected random variable.
"Most recent self-supervised representation learning approaches learn representations with la norm con- straint (Zbontar et 2 Wang & Isola 2020] Chen & He 2021 Grill et al. 2020 Chen al. 2020). restricting the output representations to the surface of unit hypersphere, i.e., the (2-normalizer representation
(Y def
Z/| Z|I2) should be on the surface of the unit hypersphere mm- This suggests that directions of learned representation vectors (instead of the absolute amplitude of elements in the vectors) matter when capturing the semantic information of instances.
4

---

Under review as a conference paper at ICLR 2023
(a) m
(b) m
(c) m = 8
(d) m 16
(c) 772 32
(1) m 64
(g) m7 128
(h) 771 256
Figure 2: The binning density between Y; and Y; over various dimensions. visualization App.
Therefore, we make an assumption that Iz-normalized zero-mean isotropic Gaussian distribution (denoted Y) follows, or least is close to, an approximated Gaussian distribution Ao0, m... even m is moderately large. Note that (O, Lm) enjoys the merits of uniqueness (a proper distribution to design uniformity metric), and might be used as approximated distribution for Y.
3.2.2 EMPIRICAL CONNECTION BETWEEN Y AND THE GAUSSIAN DISTRIBUTION
The above theoretical conclusion states that the distribution of Y is infinitely close to Gaussian distribution when infinitely large; while, in practice, we have adopt finitely large m due the memory limit. Here we empirically check the closeness between Y and Gaussian distribution when using a manageable size of dimension in practice.
Without losing any generality, we analyze arbitrary dimension of Y. The distribution of the i-th dimension of Y, as denoted as random variable Y, is visualized in Fig. 2 by binning 200,000
sampled data points (called 'samples' later) into 51 groups. Fig.
shows the distribution difference between Y; and Y; when selecting from manageable internal [2, 4, 16, 32, 64, 128, 256]. Note that the difference becomes negligible when m is moderately large (e.g.. m 32). To quantitatively measure the closeness, Fig. Figure 3: Distance between Y; and Y;
shows the change of the distance (e.g., Wasserstein distance as defined in App. G between Y, and Y; with respect to increasing m. One could observe that the distance is converged to zero with the large m. This also empirically evidences the conclusion in Theorem More details see App.
3.3
NEW METRIC FOR UNIFORMITY
In this section, we propose to use the distance between the distribution of learned representations and its ideal Gaussian distribution (O, I,) uniformity metric. Specially, we collect set of data vectors from learned representations, i.e., {Zı, Z2, .... 2n), and adopt Iz-normalized vectors to calculate the mean and covariance matrix as follows:
f1
Z
Σ
Σ n
22 (2./|zzlll (z) 14),
Where μ E IR E X and m the dimension size of vectors. To facilitate the calculation of distribution distance, apply a Gaussian hypothesis to learned representations (u, E). Based on this assumption, we employ Wasserstein distance a well-known distribution distance, to calculate the distance between two distributions, which takes the minimum cost of turning one pile into the other when viewing each distribution as a unit amount earth/soil, see the definition in App. G
We also discuss using other distribution distances uniformity metries, such as Kullback-Leibiee Divergence and Bhattacharyya Distance Gaussian distribution. See more details in App.
5

---

Under review as a conference paper at ICLR 2023
Theorem 3. Wasserstein Distance (Olkin & Pukelsheim (1882) Suppose two random variables Z1 ~ (11,, En) and Zg ~ (H,, En) obey multivariate normal distributions, then l2- Wasserstein distance between Zn and Z2 is:
(2 Z2) Ml - A42 1 Tr(2; + -
Despite its complexity, illustrated Theorem 3
Wasserstein distance over Gaussian distributions is easy to calculate as We instantiate Equation with the distribution of learned representations and ideal distribution. Then, an uniformity metric via Wasserstein distance can be formulated as:
W2 -
1 +
2 -TT(E Vm
(5)
The smaller W2, indicates the larger uniformity of representations. Besides its usefulness in collapse analysis, our proposed uniformity metric can be also used as an additional loss for various existing self-supervised methods since it is differentiable during the backward pass. One difference is that the mean and covariance matrix in Equation 3 is calculated by batch data during the training phase.
ON UNIFORMITY METRICS
In this section, we first introduce the desirable properties (called 'Desiderata') of any well-defined uniformity metric in Sec. 4.1] Sec|4.2land Sec. 4.3 compare the proposed uniformity metric W2 with existing uniformity metric Lu theoretically and empirically respectively.
4.1 DESIDERATA UNIFORMITY
A uniformity metric is a function to map of learned representations (typically dense vectors) to uniformity indicator (typically a real number).
u { 1 + I2
(6)
D E [R"" a set learned vectors (D = 12,, ....... Zn H,, each vector is the feature representation of instance, E R.m In this section, we formally define five desiderata (i.e., desirable properties) for any uniformity metrics.
Intuitively, uniformity is invariant to the permutation of instances, as it cannot affect the distribution. Property 1. Instance Permutation Constraint (IPC)
U(T(D)) = UD))
a is an instance permutation operator that changes the order of representations.
The uniformity should be invariant when all representations re-scaled, since modern machine learning tends to use directions of learned representation vectors to capture the semantic infor- mation of instances. For example, most recent self-supervised representation learning approaches learn representations with l2 norm constraint (Zbontar et al.) 2021 Wang & Isola [020 Grill et al. 2020
Chen et al. 2020), restricting the output representations to the surface of unit hypersphere, i..
D
11, S2,..., J. and s; zz/||z,ll2 is on the surface of the unit hypersphere 1
Property 2. Instance Scaling Constraint (ISC)
u({A1zı, '2Zz..... 2, U(D), ER,
The uniformity invariant when instances are cloned, since the cloning operator does not change the original distribution density. Property 3. Instance Cloning Constraint (ICC)
U(DUD) U((),
U is the union of two sets that can achieve instance cloning, DUD {Z),-.. ............ Zn 21, Z,,
The uniformity decreases when cloning features for each instance, since the feature-level clone will bring some redundancy, leading dimensional collapse (Zbontar et al. 2021 Bardes al. 2022)

---

Under review as a conference paper at ICLR 2023
Property 4. Feature Cloning Constraint (FCC)
U(D + D) (D)),
(10)
@ an feature-level concatenation operator that can achieve feature cloning as DeD = (2) 12 22 22, .... Zn ① ). and where 2, + Fil, Zim; " Fim] ] IR 2m U(D @ D) = U(D) if and only 2 22 = 7/ 0
The uniformity decreases when adding constant features for each instance, since it introduces unin- formative features and results some collapsed dimensions.
Property 5. Feature Baby Constraint (FBC)
u(D ) UD(), EENt,
(11)
D 0 0 U(D e 0*) (D) if and only if 2
(2)
22 00 0
2, + 0² and z; - 0* [211 2:2, 2im 0, .....) 0J" E Rm 22
Zr 0
Note that these five properties are necessary but not sufficient for a well-designed uniformity metric. That is, well-designed uniformity metric should satisfy these properties; while only satisfying these properties does not sufficiently lead to an ideal uniformity metric.
4.2 EXAMINING DESIDERATA OF UNIFORMITY
We employ the desiderata in Sec. 4.1 as criterion to conduct theoretical analysis for two metries Cu
in Equation and - W/ in Equation 5 The conclusion is stated in the Claim [ and Claim 2
Claim 1. Our proposed metric (ie..
W22 satisfies all properties including Property 1
and
5 See App. E. for the detailed proof.
Claim the baseline metric (i.e.--uu) satisfies Property 7 and 2
but it violates Property 3 and
5 See App. E.2 for the detailed proof.
In terms of Property IPC and Property ISC, we can directly use their definition to demonstrate both two metrics satisfy the properties. To further check whether two metric could satisfy other three properties, see App. E for the detailed proof.
Particularly, the proposed metric -W2 satisfies FBC Property while the baseline metric - -Lu does not. This opens new angle to explain the advantage of our proposed metric -W2 from the dimensional collapse perspective. Specially, the larger k would bring the more serious dimensional collapse for 0 than 'D. However, - -Lu fails to identify the more serious dimensional collapse due (D + 0*) (D). On the contrary, our proposed metric sensitive the dimensional collapse as W (D + 0*) (D))..
4.3 EMPIRICAL ANALYSIS VIA SYNTHETIC DATA
Correlation between Lu and W3- We employ synthetic experiments to study uniformity metrics. In detail, we manually sample 50000 data vectors from different distri- butions, such as standard Gaussian distribution N/O, I), uniform Distribution (0, 1), the mixture of Gaussian, etc. Based these data vectors, estimate the uniformity of different distributions by two metrics. As shown in Fig. 4 standard Gaussian distribution achieves the minimum val- ues by both W2 and Lu., which indicates that standard Gaussian distribution could achieve larger uniformity than other distributions. This empirical result is consistent with Theorem that standard Gaussian distribution achieves the
4.5
-22
-00
Figure 4: Uniformity analysis distribu- tions via two metries. maximum uniformity.
On the Dimensional Collapse To synthesize data with various specified degrees of dimensional collapse, we concatenate the zero vectors (i.e., they are full dimensional collapse) with sampled data vectors from the standard Gaussian distribution (i.e., ideal uniformity without collapse). The
7

---

Under review as a conference paper at ICLR 2023
percentage of zero-value dimensions of the concatenated vectors is ry while that of non-zero vectors is 1 1/. As shown in Fig. ((a)aaa))) and Fig. [(b) W2 is capable capturing salient sensitivity to collapse
level, while Lu keeps almost change even in 80% collapse level, indicating Lu is insensitive to the dimensional collapse.
0.59
(a) Collapse analysis via Lu
(b) Collapse analysis via W3
Figure 5: Analysis on dimensional collanse degrees.
W2 sensitive collapse degrees than Cu-
Discmooe dogutn)
(a) Collapse analysis via Lut
(b) Collapse analysis via W22
Figure 6: Dimensional collapse w.r.t various dimensions. Lu fails to identify the dimensional collapse with large dimension, while W2 able to identify the dimensional collapse matter how great/small m is.
Interestingly, visualized in Fig. Cue becomes indistinguishable with different degrees of dimen- sion collapse (17 = 25%. 30%, and75 75%) when the dimension m becomes large (e.g.. m 2 2*). On the contrary, our proposed W2 is constant the dimension number under specific degree of dimension collapse; W/ only depends on the degree of dimension collapse and is independent of the dimension number. In summary. our proposed metric W2 is more reasonable metric to measure the uniformity than the existing one Lu from an empirical perspective.
5 EXPERIMENTS
In this section, we impose the proposed uniformity metric as an auxiliary loss term for various existing sule-supe methods, and conduct experiments CIFAR-10 and CIFAR-100 datasets to demonstrate effectiveness. Codes implemented in Pytorch will be released.
Models We conduct experiments series of self self-supervised representation learning models: (i) AlignUniform (Wang& & Isola 2020), whose loss objective consists of an aligmment objective and uniform objective. (ii) three contrastive methods, i.e., SimCLR (Chen al. 2020), MoCo (He et al. 2020). and NNCLR (Dwibedi et al., 2021). (iii) two asymmetric models, i.e., BYOL (Grill al.
2020 and SimSiam (Chen & He 2021). (iv) two methods via redundancy reduction,
Barlow (Zbontar et al. (2021) and Zero-CL (Zhang et al., (002bb). To study the behavior of proposed Wasserstein distance in the self-supervised representation learning, impose it as an auxiliary loss term to the following models: MoCo v2, BYO,, Barlow and Zero-CL. To facilitate better use of Wasserstein distance, we also design linear decay for weighting Wasserstein distance during the training phase, i.e., Omas I (Omaz (mmin/TT, where T. Qmer Ommnn ae are current epoch, maximum epochs, maximum weight, minimum weight, and current weight, respectively. More detailed experiments setting see in App.
Metries We evaluate the above methods from two perspectives: one is linear evaluation accuracy measured by Top-1 accuracy (Acc@) @ I) and Top-5 accuracy (Acc@5); another is representation capacity. According to (Arora et al. 2019 Wang & Isola 2020). alignment and uniformity are the two most important properties evaluate self-supervised representation learning. We use two metrics Lu and Wi, to measure the uniformity, and a metric A to measure the alignment between the positive pairs (Wang & Isola 2020). More details about the alignment metric in App. K
8

---

Under review as a conference paper at ICLR 2023
Table I: Main comparison on CIFAR-IO and CIFAR-100 datasets. Proj. and Pred. the hidden dimension in
projector and predictor. f and | mean gains and losses, respectively.
Motheds
Proj
Prod.
11c0:1
CIFARIT
1
CIFAR-1O0
Ws
SimCLR
256
X
89.85
99.78
L04
3.75
0.47
52.45
$3.97
1.05
3.75
0.50
NNCER
256
256
87.46
99.63
1.25
-3.12
0.38
54.90
$3.81
1.25
-3.15
0.43
SinSSam
256
256
H6.71
99.07
1.19
-3.33
0.29
56.19
54.34
1.21
.3.29
0.42
Aligallnifonn
256
X
90.37
99.76
0.94
0.51
65.08
90:15
0.95
-1882
MoCo x2
256
X
90,65
99.81
1,06
-375
0.51
60.27
$6.29
1.07
-3.60
0.46
MoCa x
256
Tamn
99.68
Tam
63.68
Tea
Two
-1.6
0.66
Jess
BYOL
256
255
39.53
99.71
1:21
2.99
0.31
61.56
33.81
1:20
2.87
0.33
BYOL + W3
256
256
90.31
99.77
0.38
To
-3.90
0.65
65.16
11.00
59.25
0.36
Taae
.391
0.69
Iaas
Twies
256
X
90.64
-3.91
0.75
Barkow + 19/
256
91.43
99.78
0.39
Ta
-3.92
0.76 Leu0
66.47
90.64
0.39 Ta
-3.1
0.79
Ja.ce
7en-CI.
256
X
91,35
99.74
90:97
-3.93
0.75
ZeooCL W5
256
91.42 Tous
99.82
0.34
-3.44
0.71
6M.55
Taxs
91.02
0.34
-3.4
0.76 Ja.ca
Main Results As shown in Tab. We could observe that by imposing W2 as additional loss it consistently improves the performance than that without the loss. Interestingly, although it slightly harms alignment, it usually results in improvement in uniformity and finally leads to better accuracy. This demonstrates the effectiveness of W2 as a uniformity metric. Note imposing an additional loss during training does not affect the training or inference efficiency; therefore, adding W2 as loss is beneficial without any tangible costs.
We test the Top-1 accuracy of these models on CIFAR-10 and CIFAR-100 via linear evaluation protocol (as described in App.D) when training them in different epochs. As shown in Fig. 9 in App. L By imposing W2 as an additional loss for these models, it converges faster than the raw models, especially for MoCo v2 and BYOL with serious collapse problem. Our experiments show that imposing the proposed uniformity metric as an auxiliary penalty loss could largely improve uniformity but damage alignment,
Convergence Analysis
more representation analysis in App. M
Dimensional Collapse Analysis To gain a better understanding of how the additional loss W2 benefits the alleviation of the dimensional collapse, we visualize singular value spectrum of the representations (Jing et al.] [222). As shown in Fig. 7 the spectrum contains the singular values of the covariance matrix of representations from CIFAR-100 dataset in sorted order and logarithmic scale. Most singular values collapse to zero in BYOL and MoCo v2 models (exclude BarlowTwins). Twins), indicating a large number of collapsed dimensions occur both models. By imposing W2 as an additional loss for these two models, the number of collapsed dimensions almost decrease to zero, indicating W2 can effectively address the issue of dimensional collapse.
3VVC.
EscanT Becachin
PVTE
(a) MoCo v2
(b) BYOL
(c) BarlowTwing
Figure 7: Dimensional collapse analysis on CIFAR-100 dataset.
6 CONCLUSION
In this paper, we theoretically and empirically demonstrate that the existing uniformity metric is insensitive the dimensional collapse, and focus on designing a new uniformity metric that could capture salient sensitivity to the dimensional collapse. To this end, we propose to use the Wasserstein distance between the distribution of learned representations and the ideal distribution as the metric of uniformity. Furthermore, we formulate five desirable constraints (desiderata) for ideal uniformity metrics, based on which we find that the proposed uniformity metric satisfies all desiderata while the existing one does not. Moreover, we conduct synthetic experiments to further demonstrate that the proposed uniformity metric is capable to deal with the dimensional collapse while the existing one is insensitive. Finally, we apply our proposed metric in the practical scenarios, and impose the proposed uniformity metric as an auxiliary loss term for various existing self-supervised methods, which consistently improves the downstream performance. One limitation of our work is that five desirable constraints (desiderata) are not sufficient for ideal uniformity metrics. In future work, we would make further efforts to seek more reasonable properties for uniformity metries.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varfane-iivarrane--vvraaann regularization for self-supervised learning. In ICLR, 2022.
A. Bhattacharyya. On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of the Calcutta Mathematical Society, 1943.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using siamese time delay neural network. In NeurlPS, 1994.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' eggo,, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML,, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Henry Cohn and Abhinav Kumar. Universally optimal distribution of points spheres. Journal of the American Mathematical Sociery, 2007.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, N. Sebe, and Elisa Ricci. Solo-lcamn A library of self-supervised methods for visual representation learning. JMLR, 2022.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: neighbr contrastive learning of visual representations. In ICCV, 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, E. Sangineto, and N. Sebe. Whitening for self self-supervised representation learning. /CML, 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In ArXiv, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Berarro Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. In ICLR, 2021.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In CVPR, 2006.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation self-supervised learning. In ICCV, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
10

---

Under review as a conference paper at ICLR 2023
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
David Lindley and Solomon Kullback. Information theory and statistics. Journal of the American Statistical Association, 54:825, 1959.
lyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
Ingram Olkin and Friedrich Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications, 48:257-263, 1982.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, 2018.
Dong Wang, Ning Ding, Pijian Li, and Haitao Zheng. Cline:
Contrastive learning with semantic negative examples for natural language understanding. In ACL, 2021a
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. ICML, 2020.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021b.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu. Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, 2021.
Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. CVPR, 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. ArXiv, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. In ICLR, 2022a.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-CL: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2022b.
Xiangyu Zhao, Raviteja Vemulapalli, P. A. Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In ICCV, 2021.
Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and rong Wen. Debiased contrastive learning of unsupervised sentence representations. In ACL, 2022.

---

Under review as a conference paper at ICLR 2023
A PROOF OF THE THEOREM
Proof. According to the property Gaussian distribution, the distribution of the variable Z N00 a II,) invariant arbitrary orthogonal transformation U:
Z = UZ (UO, o UU,,
OO (UO 0, 0, = UU = Im)
Therefore Ž is identically distributed with the random variable Z. We denote identically distributed
operation as Z $ Z. For the l2- lz-normalized variables:
Y Z/ [Z
Y ==2/|
2|1,
Y 14 Y.
Since |UZ|| 2 = V (UZ) (UZ) = VZ"U'UZ = Z
|Z 2.
Y UZ
UZ
UY,
| 2
|Z||
Therefore, Y is an identically distributed operation as UY, i.e. Y A UY after an arbitrary orthogonal transformation. To conclude that the random variable Y uniformly distributes the surface of the unit hypersphere 8m = {y E R" llyll2 = 1}, here we use the proof by contradiction.
Let us assume the opposite of the above conclusion: Y does not uniformly distribute on the surface of the unit hypersphere sm-1, other words, the density of each specified-sized area in Y is not identical for the unit hypersphere Smm-1, The random variable Y has continuous density Suppose that for rı, 12 Sm- ri f r2 and p(ri) > p(r2), there exists radius for any norn (also holds for other norms) such that on
D (r E Sm-1, ||| ri <)
D2 = {r sm-1, ||r r2ll2 < c),
we still have
Vr D1, Vs E D2, p(s).
Therefore, P(D))
P(D2). Since 14 UY, D2 can be obtained from Dj by a orthogonal which implies that P(Di) ) P(D2).
transformation f
Contradiction! Hence p(ri) = p(r2) for Vr,, r2 E 5 and ri * r2. Therefore, = 2//|Z/p 2 uniformly distributes the hypersphere gm-1
B MEAN AND COVARIANCE MATRIX Y
Theorem 4. For random variable Z ~ NN,,00 Immm), and Z RT, for the ly-normalized form Z/| Zl/2. its mean and covariance matrix can be formulated as follows.
== ==
m I
Proof.
Z = [=1, [m NO, o Im,).
and its probability density function (pdf) can be written
as:
fz(z) (2 (2T)mm/2l== 2 1/2 expl 2² (o'Im (2xo²)m// expf 2 i=1
1
Σ -2)
denote Y = [91, ym. Then the mean of i-th variable yi can be written as below:
B[yd
2: VE. 27 (2m02)m/2 expf
Σ i=1
22/02 dzm-
'Let W be orthogonal transformation such that Wr, r2. Dr could be obtained by transforming every points from D, using orthogonal transformation W. namely D; (Wr: E D.).
12

---

Under review as a conference paper at ICLR 2023
As V.- of is odd function, Elys] O, and we further conclude f Y O. We also derive
the covariance matrix of Y according to its definition as below:
Ely"]
Elyn yal
R yn yon
T3 [yy2y1
B
E y2yml
Σ = B YY - -
V1
E ym
E
Then E[v,Us] (vi + j) can formulated as follows:
E[yeUy] = 1
2
2 (2 exp( Σ
dzm,
i=1
vE?"** is odd function, E[yay)] 0 (Vi (j) In terms of diagonal elements in E. we employ the symmetry to conclude E[yřl conclude Ely/] m
As VE."
E
Based on this principle, we E Bm
via below equations:
v = =
o/ E
= 1,
Therefore, Σ
1 I m
C PROBABILITY DENSITY FUNCTION OF Y;
Theorem 5. For random variable Z ~ NN0,00 Im)) and Z R for the ly-normalized form
Y Z/|Z// the probability density function (pdf) of a variable Y; in the arbitrary dimension is:
T(m/2)
[vdy)) VTI((m 1)/2]
(1 v2) (mm
Proof. Z = 12. 22. Zm] No0. 0² Im), then Z4 N 00 E [1, We denote the
variable U Z./o NO 1), V Σez (2, /2) x² - 1). then and are independent with each other. For the variable T VV/(m-1) it obeys the Student's t-distribution with 1
degrees freedom, and its probability density function (pdf) is:
T(m/2)
12
fr(()=
(1
m22
= v (m [1]T[((m 1)/2)
m
For the variable Y;
Z.
2.
2.ja
then v
Di z
V2 2j
(Z.
T VV/(m-1) V1-Y? function (cdf) of T and that of Y, can be formulated follows:
and Y VT the relation between the cumulative distribution
[P( S v11))
V S
(y.) = P([6 v1))
P([Y, 53) + Y, vd))
yi >
P(L T
VT=4m-1 5 vi}
Wi S 0 >> 0
P( T
(1) P((0
T
$ v,1))
+m
P(I P(1TEtm- -T 2 T 0))
91 S
P([ S 20) + P( - $ 2
T2
03)
8/
P([ S vm-Ty: })
/ 50
VI-
P(T 50} + P((0
vm-ly })
9: >
VI-1
P(T < Vm 18 Fr
Vm 11/1
V1 V1
V1-
13

---

Under review as a conference paper at ICLR 2023
Therefore, the pdf of Y; can be derived follows:
d
d
V m 136
(v)) - dy
Py. (v6)
= Frl
dyp
v 1 1/²
- -rr( v m
Itls
d
vm= Iili
V1 v/² dy:
v v/3
r(m/2)
V(m (m 1)TI ((m-(()))))) 12/22 (1 - mm?
II
m 1(1
y. 3/2
T(m/2)
Val((m - 11/22 (1 y 1m/31/2
D PROOF OF THE THEOREM 2
Proof. For the variable Y; ~ NO, ). its pdf and k-th order raw moment can be formulated as:
(1--))111
(v)
m 2m
exp
my
2 }.
E|
k 2j.j 1, 3...
k: 2j
According to the Theorem
5 the pdf of Y; is:
T(m/2)
fx(1))) = VTI((m- 1)/2) (1 (1- (mm
For S y² < 1, the Taylor expansion of log(1 y²) can be written as:
06 2i
log(1 y²)
j
Then the Kullback-Leibler divergence between y, and Yú can be formulated as:
DKL(Y, =
fr. (v) (v) - log fr. (v:)ldy
my²
l'mm22)
3
Fe. [log
log
log(1 y) )dy
2x
V [TI( 1am--11 -1)/2)
2
log
m
log
[(m/2)
+
100
fr. fr((v]
myy
m
3
log( 1 - y²²1
2T
VTl'((1n
-1)/2)
2
2
-00
log
[[(m 1)/2)
+
fr(
my
+
171
3
06
y
2j
2
l'(m/2)
2
2
1=1
log
m
[(mm - 11)/2)
m
B(92)
+
m 3
B(Y22)/
//j
2
T(m/2)
2
2
j=1
m [(mm 1)/2)
1
m
3
1
3
5 3
log
V l(mm/2)
2
+
2
m
2m
3m3
0
m
According to the Stirling formula, we have l'x + a))
+ 1(I)x0 as H. + 00, therefore:
m [((m 1)/2) lim log
T( - 1)/2)
lim log m-+000
V T(m/2) m-
2 T((m 1)/2)( 1T 1//2
2
lim log m-
0
2 m
14

---

Under review as a conference paper at ICLR 2023
Then the Kullback-Leibler divergence between Y; and Y; converges
follows:
m2-000
lim DEL (Y,, Yi)
m 10-000
lim log
V
m
2
[((m 1)/2)
[(m/2)
2
+
m
2
+
2m
5
3m3
+ o
1 m
)
1
3
1
3
5
+ lim
+
+
0
)1
0
m00000
2
m
2m
3m
m
E EXAMINING THE DESIDERATA FOR TWO UNIFORMITY METRICS
E. PROOF FOR -W2
22 ON DESIDERATA
could be easily proved using the definition. We here to The first two properties (Property D and 2
examine the rest three properties by one for the proposed uniformity metric -
Proof. Firstly, we prove that our proposed metric - W2 could satisfy the Property 3 As DUD = [Z), 22, Zuy Z1, Z2 ..., Zn J. then its mean vector and covariance matrix can be formulated as follows:
n
2z |zill = ,b, E
Σ 2(z;
| ()) - -a) E,
2n
2n i=1
Then we have:
2 Tr((i112) = W2(D).
W2(DUD) & ||A +1+ +1+Tr(5)
Vm
Therefore, W22 -W((D), indicating that our proposed metric - We could satisfy the Property 3
Then, we prove that our proposed metric - W/ could satisfy the Property 2, [Z41, 2/2, ... Rims Zil, Z:2, ... Zim] set: D + D, its mean vector and covariance matrix can be formulated as follows:
Given 2,
[Zs1y R:2y on Zim] I" and 2;
E R2mm for the
(H//2) 1\/2))
2/2 Σ/2
Σ Σ/2
A =
Σ
Σ/2
1112/2
11/2/2 81/2/2 11/2/2)
As $1/2
TT((E) Tr(E) and Tr Tr($1/2) = Tr/E1/)),
Then we have,
2
Wh 8 D) &
+1+T(()
Tr(E112)
V2m
2
||a4112 +1+TrE))
V2m Tr(E1/2).
2 Tr(E1/2) Wc(DD,
|1m113 +r+T((E))
Vm
Therefore, WD ( < WD) indicating that our proposed metric - W3 could satisfy the
Property 4
Finally, we prove that our proposed metric W2 could satisfy the Property
Given 2;
[2s1: F ... I"m and 2; e 0
[Z01, #.2.... Zim, 0, O, ..., 0J E gu for the set: D 0
its mean vector and covariance matrix can be formulated as follows:
11 0*
$
Σ
ma
*
15

---

Under review as a conference paper at ICLR 2023
Therefore, Tr(Σ) Tr(E), and Tr Tr(E Tr(El/2):
W2 @ 0*) &
||Al 1+
Tr(E112)
Vm + k
|1 +1 +r(EE)
Tr(E1/2)
Vm + k
2 Tr(E4/2) = W2(D)
| 1+ Tr(E)
+
Vm
Therefore,
(2 @ 0%) < - W2 indicating that our proposed metric - W2 could satisfy the
Property
E.2
PROOF FOR LL ON DESIDERATA
The first two properties (Property D and P could be easily proved using the definition, We here to examine the rest three properties one by one for the existing uniformity metric - -Lu-
Proof. Firstly, we prove that the baseline metric - Cuu cannot satisfy the Property 3 the definition of Ly in Equation
According to
we have:
Cu UD) & log 2n 1)/2
[
11
j=1
log
(4
TR5T
+ n),
2n(2n 1)/2
1=2 j=1
We set G Σn
2j=1
11
T0,1
and then we have:
j==
G=
<|| (ndl
Hi 113 n(n 1)/2
1=2 j=1
1=2 j=1
G = n(n - 1)/2 if and only if = 2/2 = Zn
4G +
G
Lu(DUD) - (D) = log
log
2n(2n - 1)/2 1((1 - 1)/2
(4G n n)n(nn - 11//2 log (4G + n) 1)
loog
2n - -1)/2
4nG - 2G
4nG 1G n² n 2 log = 0.
=log
4nG - 2G
Lu (DUDD = (D( if and only n(n - 1)/2, which requires 21
=2,, (an
extreme case that all representations collapse to a constant point, as depicted in the Fig. D We
exclude this extreme case for consideration in the paper, and we have (D U D) < - Lu (D). Therefore, the baseline metric - Lu cannot satisfy the Property Then, prove that the baseline metric -Lu cannot satisfy the Property
4
Given z;
[2E1, Zi2,.., Zim]² FT, and Zj = [2j1, 2/2 2mm]" I" and we set 2, = 2, + 2; and à, = Zj + Zj. we have:
1 12
CD 1 D) A log n(n - 1)/2 i=2 1=1
As 2,
[211, 2,2, .... Zim 2,1, 2,2;
Zim T and 2,
[2y1, Zy2, Zimi Byi, 2j2;
Fjm T
then
|2 = /2 1l. || = /21lzz111 and (2,, 2,)} = 2(zz, 2,), we have:
2;
à
1/3 = 2 - 2
(2., 2,})
2 2 2 |z 2||
2(z,, 2i))
2:
Zj
2
2
|2,1
|2, |
|z,
16

---

Under review as a conference paper at ICLR 2023
Therefore, - LD (D) indicating that the baseline metric Lu cannot satisfy the Property 4
Finally, we prove that the bascline metric -Ly cannot satisfy the Property 5
Given z;
[ai1 F Fi2, Zim]", and 2j
[2y1, F22, Zjm] and we set 2; - 2, + and 2, = 2j @ 0%,
we have:
(D + 0*) e log
n 11/2
j=1
As 2, [Ze Fi2, .... Fim, 0, 0, ..., 0 and 2j [251, 222 .... Rjm 0,0, 0,0....
.... 0f then |2
-1z]l
| = [|z |l. and (2,, 2z)
- (2: kz), therefore:
2,
2
12
(2, 2n)
(2 k,)
Z;
2,
112.
|2
|2
|2
2 2zح
2;
| 2zg
Therefore, Property
LD @ 0%) - LD) indicating that the baseline metric
-Lu cannot satisfy the
F A TWO-DIMENSIONAL VISUALIZATION FOR Y AND Y
We also analyze the joint binning density and present 2D joint binning density of two arbitrary individual dimensions, Y; and Yj (i f) in (a), and Y; and Y, (i * j) in (b). More details about binning density see in App. H Even m is relatively small (i.e., 32), it looks that the density of two distributions close.
Bann 22 Donegg
111
1.0
77
(a) Density for two arbitrary dimensions of Y (b) Density for two arbitrary dimensions of Y
Figure 8: Visualization of two arbitrary dimensions for Y and Ý when m 32. See the binning
density in one-dimensional visualization over various dimensions in Fig.
G THE DEFINITION OF WASSERSTEIN DISTANCE
Definition 1. Wasserstein Distance or Earth-Mover Distance with norm is defined as below:
WP,,,,PP)) = inf Egr,yry [||x - 1/,, .ETT(P..P,))
(12)
where IIPP,,,, denotes the set of all joint distributions ((x, y) whose marginals are respectively P, and Pg. Intuitively, when viewing each distribution unit amount of carth/soil, Wasserstein Distance or Farth-Mover Distance takes the minimum cost transporting "mass" from x to y in order to transform the distribution P. into the distribution P.,
17

---

Under review as a conference paper at ICLR 2023
H DETAILS ON BINNING DENSITY
Details for ID Visualization The density of Y; and Y; visualized in Fig. 2 estimated by binning 200000 data samples into 51 groups. We observe that the density of Y, would be more overlapped with that of Y,. To further verify our observation, we instantiate P, and Pg in Equation 12 with the binning density of Y; and Y, and employ W1 (Fr, Fg) as the distribution distance between Y; and Y, We calculate Wi (IPr, Pq) ten times and average them as visualized in Fig.
Details for 2D Visualization The joint density of (Y, Yj) and (Y, Y,) (i + j). visualized in Fig. is estimated by 2000000 data samples into 51 x 51 groups in two-axis (m = 32).
I OTHER DISTRIBUTION DISTANCES OVER GAUSSIAN DISTRIBUTION
In this section, besides Wasserstein distance over Gaussian distribution, as shown in Theorem 3 we also discuss using other distribution distances as uniformity metrics, and make comparisons with Wasserstein distance. As provided Kullback-I.eibe Gaussian distribution in Theorem and in Theorem 7 both calculations require the covariance matrix is a full rank matrix, making them hard to be used to conduct dimensional collapse analysis. On the contrary, our proposed uniformity metric via Wasserstein distance is free from such requirement on the covariance matrix, making it easier to be widely used in practical scenarios.
Divergence and Bhattacharyya Distance over
Theorem 6. Kullback-Leible Divergence (Lindley & Kullback (1959)) Suppose two random vari- ables 2₁ ~ NNu, Σ1) and Z2 ~ -N (uz, Σ2) obey multivariate normal distributions, then Kullback- Leibler divergence between Z1 and Zz is:
DKL (Z, Z2) 2 ((1) 122 Eg (m1 12) + Tr E2 - In det Eg det E )
Theorem 7. Bhattacharyya Distance (Bhattacharyya (7443) Suppose two random variables Z₁ N(u1, Σn) and L2 ~ N (Hzz Σn) obey multivariate normal distributions, Σ (2)) + En), then bhattacharyya distance between Z1 and Zo is:
1
1 In (M1 T u2) +
det Σ
(1, DB DB(L,, Zz)
8
v det Σj det Σn²
J EXPERIMENTS SETTING IN THE EXPERIMENTS
Setting To make fair comparison, we conduct all experiments in Sec. on a single 1080 GPU. Also, we adopt the same network architecture for all models, i.c., ResNet-18 18 (He et al. 2016) as the encoder, a lhree-layer MI.P as the projector, and a three-layer MI.P as the projector, respectively. Besides, We use LARS optimizer (You et al.] (017)) with base learning rate 0.2, along with a cosine decay learning rate schedule (Loshchilov & Hutter 2017) for all models. We evaluate all models under linear evaluation protocol. In specific, models are pre-trained for 500 epochs and evaluated by adding linear classifier and training the classifier for 100 epochs while keeping the learned representations unchanged. We also deploy the same augmentation strategy for all models, which is the composition of series of data augmentation operations, such as color distortion, rotation, and cutout. Following (da Costa et al., 2022), we set temperature As for MoCo (He et al. 2020) and NNCLR (Dwibedi et al.) 2021) that require an extra queue to save negative samples, we set the queue size to 2n For the linear decay for weighting Wasserstein distance, detailed parameter settings are shown in T
0.2 for all contrastive methods.
Table 2: Parameter setting for various models in experiments.
Models
MoCo v2
BYOL
Barlow Twins
Zero-CL
amaz
1.0
0.2
30.0
30.0
Omin
1.0
0.2
0
30.0
18

---

Under review as a conference paper at ICLR 2023
K ALIGNMENT METRIC FOR SELF-SUPERVISED REPRESENTATION LEARNING
As one of the important indicators to evaluate representation capacity, the alignment metric measures the distance among semantically similar samples in the representation space, and smaller alignment generally brings better representation capacity. Wang et al (Wang & Isola 2020) propose a simpler approach by calculating the average distance between the positive pairs as alignment, and it can be formulated as follows:
A A Fi xp)m
z
[
(13)
||
|
Where (zf, z!) is a positive pair discussed in Sec
2.1
We set B = 2 in the experiments.
L CONVERGENCE ANALYSIS ON TOP-1 ACCURACY
Here show the change of Top-1 I accuracy through all the training epochs in Fig 9 During training, we take the model checkpoint after finishing each epoch to train linear classifier, and then evaluate the 1 accuracy on the unseen images of the test (in either CIFAR-IO or CIFAR-100) In both CIFAR- 10 and CIFAR-100 R-000, we could obverse that imposing the proposed uniformity metric as an auxiliary penalty loss could largely improve the Top-1 accuracy, especially in the early stage.
--. Micrr-
--- EICE KJ
0
200
80
1.00
(a) MoCo v2 in CIFAR-10
(b) BYOL in CIFAR-10
(c) Barlow Twins BarlowTwins
in CIFAR-10
C-pp- AQUUICT
Tpxils
Tyucks
(d) McCo v2 in CIFAR-100
(c) BYOL in CIFAR-100
(f) BarlowTwins in CIFAR-100
Figure Convergence analysis on Top-1 accuracy during training.
M ANALYSIS ON UNIFORMITY AND ALIGNMENT
Here we show the change of uniformity and alignment through all the training epochs in Fig. 10 and Fig. M respectively. During training, we take the model checkpoint after finishing each epoch to evaluate the uniformity (i.c., using the proposed metric W/2 and alignment (Wang & Isola 2020
on the unseen images of the test set (in either CIFAR- 10 or CIFAR-100 In both CIFAR- 10 and CIFAR- 100, we could obverse that imposing the proposed uniformity metric as auxiliary penalty loss could largely improve its uniformity. Consequently, it also lightly damage the alignment (the smaller, the better-aligned) since better uniformity usually leads to worse alignment by definition.
N THE EXPLANATION FOR PROPERTY 5
Here we explain why the Property 5 is an inequality instead of an equality by case study. Suppose set of data vectors (D) defined in Sec. is with the maximum uniformity. When more dimensions with zero-value are inserted to D, the set of new data vectors (D @ 0*) cannot achieve maximum uniformity any more, as they only occupy a small space on the surface of unit hypersphere. Therefore, the uniformity would decrease significantly with large k.
19

---

Under review as a conference paper at ICLR 2023
305
415
Opude
Dacl
(a) MoCo v2 in CIFAR-10
(b) BYOL in CIFAR-10
BarlowTwins in CIFAR-10
2004-100
Spode
Spocks
(d) MoCo v2 in CIFAR-100
(e) BYOL in CIFAR-100
(l) Barlow in CIFAR-100
Figure 10:
Visualization uniformity during training
Tpads
Spect
(a) MoCo in CIFAR-10
(b) BYOL CIFAR-10
BarlowTwins CIFAR-10
-4- 2916
Batichoan
(d) MoCo v2 in CIFAR- -10
(c) BYOL in CIFAR-100 100
(f) BarlowTwins in CIFAR-100 100
Figure 11: Visualization of alignment during training.
(a) Two-dimensional visual-(b) Three-dimensional visualiza-(c) Three-dimensional visualiza- ization with no collapsed di- tion with collapsed dimension tion with collapsed dimension mension
Figure 12: Case study for Property 5 and blue point are data vectors.
To further illustrate the inequality, we visualize sampled data vectors. In Fig. 12(a) we visualize
400 data vectors (D1) sampled from (0, I2), and they almost uniformly distribute on the S We
insert one dimension with zero-value to D,, and denote it as DD ② 0 as shown in Fig. 12(b) In
comparison with D2 where 400 data vectors are sampled from (O, Ia), as visualized in Fig. 12(c)
Di + 0 only occupy ring on the S², while D- almost uniformly distribute the S², Therefore.
2(2 ((1 1 @ O¹). Note that no matter how great/small m, the baseline uniformity metric (Wang & Isola 2020) and our proposed uniformity metric have equal maximum uniformity. i.e., W/ 0 and
Lu = 4.0. Therefore, the maximum uniformity over various dimensions m should be equal, or at
least close to, then have ((()) (D2) (DD, @ 0*). The Property 5 should be inequality, and can be used to identify the capacity on capturing sensitivity to the dimensional collapse.
20


Reviews:
[{"review_id": "t1VjdObcCL", "reviewer": "Reviewer_iVeQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Wasserstein distance between an isotropic zero mean Gaussian distribution and the distribution of the learned representations as a robust uniformity metric for self-supervised learning. Five desirable properties of uniformity are discussed and the proposed metric is compared against another recently introduced uniformity metric (Wang & Isola, 2020) in terms of how well the two metrics satisfy these properties. The proposed metric is also added to the loss function and several recent self-supervised algorithms have been retrained using this modified loss function. Results suggest that adding this new uniformity metric-based loss increases accuracy and improves convergence of the training. ", "summary_of_the_review": "The proposed uniformity metric can be considered as an important contribution of the work for the self-supervised learning literature. However, the fact that limitations and potential pitfalls were not discussed  raises important questions about whether the reported improvements are indeed due to the proposed metric or some other aspect of learning is being affected by the modified loss functions. ", "strength_and_weaknesses": "Strengths:\n\t- The emphasis on dimensional collapse and how it was overlooked by earlier work was well thought\n\t- The metric is very simple to implement because it only involves evaluating the Wasserstein distance of two Gaussian distributions\n\t- Extensive experiments have been performed reimplementing several of the recently introduced techniques with the modified loss function. \n\nWeaknesses:\n\t- The paper assumes that l2-normalized zero-mean isotropic Gaussian distribution follows a Gaussian-like distribution. The illustration in 1D is helpful, but it would have been more interesting to see the deviation trend as the dimensionality increases. \n\t- Five properties were introduced. It was not discussed whether these five properties were sufficient to prove ideal uniformity. \n\n---------------\nAuthors' responses somewhat address my critiques but not at a level to increase my rating. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "o6sgUAvDoJo", "reviewer": "Reviewer_9DpF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors are motivated by a recent observation in self-supervised learning---in order to prevent a model from dimensional collapse and subsequently poor data representation, we should encourage the representation to be uniform; see for example [1]. The authors argue that zero-mean isotropic Gaussian distribution has the ideal uniformity property, and based on this, they propose to use Wasserstein distance as a uniformity metric as a part of the loss function. It is shown that the Wasserstein distance is useful for avoiding dimensional collapse and improving representation.\n\n[1] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.\n", "summary_of_the_review": "The authors study uniformity in self-supervised learning and propose Wasserstein distance as a part of the loss function. I find the overall argument plausible. However, the major weaknesses---namely claiming obvious results as theoretical contributions, and weak empirical justications---severely lower the quality of this paper. Thus, I would not recommend acceptance.", "strength_and_weaknesses": "Here is a summary of strengths and weaknesses.\n\nPros:\n- The perspective of alignment and uniformity is very helpful for understanding data representation in self-supervised learning, and yet this perspective is not fully explored and some issues such that dimensional collapse remains. The authors suggest that one reason may be that common uniformity metrics do not satisfy Cloning Constraint and feature Baby Constraint properties. This argument seems plausible and provides a new way of thinking uniformity.\n\nCons:\n- Section 3 seems to be very obvious to many people. So it is not clear what the theoretical contribution of this paper is.\n- The empirical contribution is also fairly weak: the results in Table 2 are not overwhelmingly convincing, there is no visualization of data representation, there is a lack of in-depth analysis and comparison of different uniformity metrics, etc.\n- Estimation of covariance matrices from data, as shown in Eq. 3, is not well grounded. It is well-known in statistics that estimating covariance matrices in high dimensions is not reliable. Also, the Gaussian assumption may not be valid in the representation space.\n\n**Update:** After the revision, I feel that the empirical part is strengthened, so I slightly increased the score.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vhf1tOq-Es", "reviewer": "Reviewer_ZorS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the dimensionality collapse in self-supervised learning (SSL). Though the uniformity of the representation is regarded as an important feature for downstream tasks, the existing loss function in SimCLR is insensitive to dimensional collapse. With a theoretical guarantee, the authors propose a Wasserstein-based uniformity measure to avoid dimensionality collapse. Numerical experiments show that the proposed method efficiently works to extract useful features for downstream tasks. ", "summary_of_the_review": "The proposed method is based on some assumptions and approximations, and the theoretical analysis is insufficient. ", "strength_and_weaknesses": "Strength\n- The readability of the Introduction and related works is excellent. The idea of the proposed method is easy to follow. \n\nWeaknesses\n- In section 3, some theorems are presented. Since these theorems are elemental results, theorems should be moved to the appendix. Instead, the authors could discuss more important issues, such as the theoretical properties of the proposed Wasserstein-based uniformity measure.\n\n- The Wasserstein distance from the empirical distribution to the Gaussian distribution with mean zero and variance I/m is employed to measure the uniformity on the unit sphere. To derive the uniformity measure, some assumptions and approximations are introduced. Therefore, the theoretical properties of the proposed measure are not very clear. Even when the data is not uniformly distributed, the Wasserstein-based measure, i.e., the negative Wassersteindeitance, can take a large value.\n\n- The authors should concentrate on thorough theoretical analysis to guarantee the effectiveness of the proposed method. \n  - The measure -W_2 takes the first and second-order moment into account. How the higher order moment affects the W_2-based measure (4)? \n  - Is it possible to analyze the generalization error of downstream tasks when the proposed measure is used for feature learning? \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Zv2DE04Qm", "reviewer": "Reviewer_UX2Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a systematical analysis of the collapse degree of representations in self-supervised learning. To achieve a quantifiable analysis of the dimensional collapse, the work proposed to use the Wasserstein distance between the distribution of learned representations and the ideal distribution as the metric of uniformity. Based upon this, it introduces five desirable constraints for ideal uniformity metrics to make theoretical comparisons between the proposed metric and the existing one. Finally, the work shows that by imposing the proposed uniformity metric as an auxiliary loss term for various existing self-supervised methods, it consistently improves the downstream performance.\n\n", "summary_of_the_review": "Overall, this is a well-presented paper. However, the reviewer finds the result a bit incremental overall. \n", "strength_and_weaknesses": "# strength: the work studies an interesting problem in self-supervised learning. \nThe work proposed a new metric that better evaluates the quality of representations learned via self-supervised learning.\n\n# weakness: the reviewer finds the overall result a bit incremental, compared to the work Wang & Isola 2020\n1. The theorems in Section 3.1 & Section 3.2 are some known facts, \n2. more validation is needed on the assumption that a uniform distribution over the sphere is approximately the same as Gaussian distribution in high dimension\n3. the improvement of accuracy on downstream tasks in Table 2 is quite marginal.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject