Rethinking the Uniformity Metric in Self-Supervised Learning
Authors: Xianghong Fang
Jian Li
Qiang Sun
Benyou Wang
Venue: ICLR 2024 poster (2024)
Keywords: Effective uniformity metrics
dimensional collapse
Wasserstein distance
self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a new Wasserstein uniformity metric that could capture feature redundancy and dimensional collapse.

Abstract: Uniformity plays an important role in evaluating learned representations, providing insights into self-supervised learning. In our quest for effective uniformity metrics, we pinpoint four principled properties that such metrics should possess. Namely, an effective uniformity metric should remain invariant to instance permutations and sample replications while accurately capturing feature redundancy and dimensional collapse. Surprisingly, we find that the uniformity metric proposed by \citet{Wang2020UnderstandingCR} fails to satisfy the majority of these properties. Specifically, their metric is sensitive to sample replications, and can not account for feature redundancy and  dimensional collapse correctly. To overcome these limitations, we introduce a new uniformity metric based on the Wasserstein distance, which satisfies all the aforementioned properties. Integrating this new metric in existing self-supervised learning methods effectively mitigates dimensional collapse and consistently improves their performance on downstream tasks involving CIFAR-10 and CIFAR-100 datasets. Code is available at \url{https://github.com/statsle/WassersteinSSL}.

Published as a conference paper at ICLR 2024
RETHINKING THE UNIFORMITY METRIC IN SELF- SUPERVISED LEARNING
Xianghong Fang
Jian Li Tencent Al Lab
The Chinese University of Hong Kong, Shenzhen fangxi anghong2 1 com
11
com
Qiang Sun'
Benyou Wang"
University of Toronto & MBZUAI qsunstats a. com
The Chinese University of Hong Kong, Shenzhen & SRIBD wangbenyou@ edu.
ABSTRACT
Uniformity plays an important role in evaluating learned representations, pro- viding insights into self-supervised learning. In our quest for effective unifor- mity metrics, we pinpoint four principled properties that such metrics should possess. Namely, an effective uniformity metric should remain invariant to in- stance permutations and sample replications while accurately capturing feature redundancy and dimensional collapse. Surprisingly, we find that the unifor- mity metric proposed by Wang & Isola (2020) fails satisfy the majority of these properties. Specifically, their metric is sensitive to sample replications, and can not account for feature redundancy and dimensional collapse correctly. To overcome these limitations, we introduce new uniformity metric based on the Wasserstein distance, which satisfies all the aforementioned properties. Inte- grating this new metric in existing self- sef-ssupevviee learning methods effectively mitigates dimensional collapse and consistently improves their performance on downstream tasks involving CIFAR-IO and CIFAR-100 datasets. Code is available at httph::///// //tp:::///tttiii com/ statslaaalea...aaa..aa wtt.eaassseeeaasesseeeessst
INTRODUCTION
Self-supervised learning excels in acquiring invariant representations various augmentations (Chen et al., 2020; He et al., 2020; Caron et al., 2020; Grill et al., 2020; Zbontar et al., 2021). It has been outstandingly successful across wide range of domains, such as multimodality learning, object detection, and segmentation (Radford et al., 2021; Li et al., 2022; Xie et al.. 2021; Wang et al., 2021; Yang et al., 2021; Zhao al., 2021). To gain a deeper understanding of sele-super learning. thoroughly evaluating the learned representations is necessary (Wang & Isola, 2020; Gao et al., 2021; Tian et al., 2021; Jing et al., 2022).
Alignment, metric quantifying the similarities between positive pairs, holds significant importance in the evalua- tion of learned representations (Wang & Isola, 2020). It ensures that positive pairs are mapped to similar features, making them invariant to unnecessary details (Hadsell et al., 2006; Chen et al., 2020). However, relying solely alignment proves inadequate for effectively assessing the representations. This limitation becomes evident in the presence of extremely small alignment values in col- lapsing solutions, as observed in Siamese networks (Had- sell et al., 2006), where all outputs collapse a single point (Chen & He. 2021), as illustrated in Figure 1. In such cases, the learned representations exhibit optimal alignment but fail to provide meaningful information for any downstream tasks. This underscores the necessity of incorporating additional metrics when evaluating learned representations.
Constant Collapse
Dimensional Collapse
The left figure presents constant collapse, and the right figure visualizes di- mensional collapse.
Figure 1:
"iang Sun and Benyou Wang joint corresponding authors.

---

Published as a conference paper at ICLR 2024
To further evaluate the learned representations, Wang & Isola (2020) formally introduced uniformity metric based on the logarithm of the average pairwise Gaussian potential (Cohn & Kumar, 2007). distributed uniformly across the unit hypersphere,
Uniformity assesses how feature embeddings and higher uniformity indicates more information from the data is preserved. Since its introduction, uniformity has played pivotal role in understanding self-supervised learning and mitigating constant collapse (Arora et al., 2019; Wang & Isola, 2020; Gao et al., 2021). Nevertheless, the effectiveness of this partieular uniformity metric warrants further examination.
To delve deeper into the existing uniformity metric proposed by Wang & Isola (2020), we introduce four principled properties that an effective uniformity metric should possess. Guided by these properties, we conduct a theoretical analysis, unveiling key limitations of this metric, particularly its inability to capture feature redundancy and dimensional collapse (Hua et al., 2021). Dimensional collapse refers to the scenario where representations occupy a lower-dimensional subspace rather than the entire embedding space (Jing et al., 2022); see Figure 1. We reinforce our theoretical findings with empirical evidence, demonstrating, for instance, the existing metric's 's inability to differentiate between different degrees of dimensional collapse. Subsequently, we propose a novel uniformity metric based on the quadratic Wasserstein distance that satisfies all four properties, thereby surpassing the existing one. Finally, integrating the proposed uniformity metric as an auxiliary loss within existing self-supervised learning methods consistently enhances their performance in downstream tasks.
Our main contributions are summarized as follows. (i) We identify four principled properties that an effective uniformity metric should possess, providing new guidelines on designing such metrics. (ii) Surprisingly, we find that the existing uniformity metric (Wang & Isola, 2020) fails meet the majority of these properties. For example, it can not correctly capture dimensional collapse. (iii) We propose a new uniformity metric based on the Wasserstein distance that satisfies all four properties, addressing key limitations of the existing metric. (iv) Our proposed uniformity metric can seamlessly integrate as an auxiliary loss in various self-supervised learning methods, resulting improved performance in downstream tasks.
2
BACKGROUND
2.1
SELF-SUPERVISED REPRESENTATION LEARNING
Self-supervised learning leverages the idea that similar samples should have similar representa- tions that are invariant to unnecessary details (Wang & Isola, 2020). For instance, the Siamese network (Hadsell et al., 2006) takes as input positive pairs 1x,xx" often obtained by taking two augmented views of the same sample x. These positive pairs are then processed by an encoder network f consisting of a backbone (e.g., ResNet (He et al., 2016)) and a projection MLP head (Chen al., 2020), yielding representations (z" = f(x"), ' To enforce invariance, natural approach is to minimize the following alignment loss, defined as the expected distance between positive pairs:
LA = ( || 2 2
where (s.) the distribution of positive pairs.
However, optimizing the above alignment loss alone may lead to an undesired collapsing solution, where all representations collapse into a single point, as shown in Figure 1.
2.2 EXISTING SOLUTIONS TO CONSTANT COLLAPSE
To prevent constant collapse, existing solutions include contrastive learning, asymmetric model architecture, and redundancy reduction.
Contrastive Learning Contrastive learning offers potent solution mitigate constant collapse. The key idea is to leverage negative pairs. For example, SimCLR (Chen et al., 2020) introduced an in-batch negative sampling strategy that utilizes samples within a batch as negative samples. However, its effectivvnees is contingent the use of large batch size. To address this limitation,
For simplicity, we also refer to (z", x²) as positive pairs.
2

---

Published as a conference paper at ICLR 2024
MoCo (He et al., 2020) used memory bank, which stores additional representations negative samples. Recent research endeavors have also explored clustering based contrastive learning. which combines clustering objective with contrastive learning techniques (Li et al., 2021; Caron et al., 2020).
Asymmetric Model Architecture The use of asymmetric model architecture represents another strategy to combat constant collapse. One plausible explanation for its effectiveness is that such an asymmetric design encourages encoding more information (Grill et al., 2020). To maintain this asymmetry, B BYL (Grill et al., 2020) introduces the concept of using an additional predictor in one branch of the Siamese network while employing momentum updates and stop-gradient operators in the other branch. DINO (Caron et al., 2021), takes this asymmetry a step further by applying two encoders, distilling knowledge from the momentum encoder into the other one (Hinton et al., 2015). SimSiam (Chen & He, 2021) removes the momentum update from BYOL, and shows that the momentum update may not be essential in preventing constant collapse. However, Mirror- SimSiam (Zhang al., 2022a) swaps the stop-grt gradient operator to the other branch. Its failure challenges the assertion made in SimSiam (Chen & He, 2021) that the stop-gee nop-gradient operator the key component for preventing constant collapse. Tian et al. (2021) provides theoretical examination to elucidate why an asymmetric model architecture can effectively avoid constant collapse.
Redundancy Reduction The fundamental principle behind redundancy reduction to mitigate constant collapse is to maximize the information preserved by the representations. The key idea to decorrelate the learned representations, Barlow Twins (Zbontar al., 2021) aims achieve decorrelation by focusing on the cross-correlatio matrix, while VICReg (Bardes et al., 2022) focuses on the covariance matrix. Zero-CI. (Zhang et al., 2022b) takes hybrid approach, combining instance- wise and feature-wise whitening techniques.
2.3 THE EXISTING UNIFORMITY METRIC
While the aforementioned solutions effectively prevent constant collapse, they are not as effective in preventing dimensional collapse, wherein representations occupy a lower-dimensional subspace instead the entire space. This phenomenon has been observed in contrastive learning by visualizing the singular value spectra of representations (Jing et al., 2022; Tian et al., 2021).
To quantitatively measure the degree of collapse, Wang & Isola (2020) introduced a uniformity loss based on the logarithm of the average pairwise Gaussian potential. Given (normalized) feature representations {Z), Z2, ..., 2. their proposed empirical uniformity loss is:
Lu :0: log n(n
1
2,13
1)/2
j=1
where t 0 is fixed parameter, often set to 2. Then -Lu serves as the corresponding uniformity metric, with higher value indicating greater uniformity.
We demonstrate in this work that this metric is insensitive to dimensional collapse, both theoretically in Section 3.2 and empirically in Section 5.2.
WHAT MAKES AN EFFECTIVE UNIFORMITY METRIC?
In this section, we begin by presenting four fundamental properties that an effective uniformity metric should possess. Leveraging these properties as a lens, we then scrutinize the existing uniformity metric- -Lu, shedding light on limitations.
3.1 FOUR PROPERTIES FOR UNIFORMITY
A uniformity metric 41 Rm" + R is function that maps set of learned representations to scalar indicator of uniformity. In the following section, we introduce four principled properties that an effective uniformity metric should possess. Let D Z1. Zı..... Zn E RTn represent the learned representations. To avoid the trivial case, we assume that Z,...., Z, are not all equal, meaning that not all points collapse to single constant point.
3

---

Published as a conference paper at ICLR 2024
First, effective uniformity metric should be invariant to the permutation of instances, the distribution of representations should not be affected by permutations.
Property 1 (Instance Permutation Constraint (IPC)). An effective uniformity metric U should satisfy
U(=(D) UD())
where n is permulation over the inslances.
Second, effective uniformity metric should be invariant instance clones, instance cloning does not vary the distribution of representations.
Property 2 (Instance Cloning Constraint (ICC)). An effective uniformity metric U should satisfy
U(D A D) =(()),
where D 9 D (Z1, 22. An, 21, Z2, ........ Zn
Third, an effective uniformity metric should strictly decrease feature-level cloning for each instance occurs, as this duplication introduces redundancy, which corresponds to dimensional collapse (Zbontar al., 2021; Bardes al., 2022).
Property (Feature Cloning Constraint (FCC)). An effective uniformity metric U should satisfy
u(D D) U(D),
where DeD == {z1 BZ B22. Z, b2 and +z == (211, -. Zim: Ze1.
Zim E R
Fourth, an effective uniformity metric should strictly decrease with the addition of constant features for each instance, as this introduces uninformative and thus redundant features, which again corresponds to dimensional collapse.
Property 4 (Feature Baby Constraint (FBC)). An effective uniformity metric U should satisfy u(D 0*) < U(D), KEN+
(6)
where @ is defined in Property 3, that is, D 0* = {zz + 0 + 0 Zn 2 0%} and z; + 0 (2:1, 242,2,2.... 2.,, Zim: ..., 0)T E mm+
Intuitively, Properties and 2 ensure that the uniformity metric should remain insensitive to instance permutations and sample replications. respectively. Mcanwhile, Properties 3 and ensure that feature redundancy and dimensional collapse reduce the uniformity metric, as they make the distribution the representations less uniform. These four properties constitute intuitive yet principled characteristics of an effective uniformity metric.
3.2 EXAMINING THE UNIFORMITY METRIC -L1
We employ the four properties introduced earlier to analyze the uniformity metric Lu defined in Eqn. (2). The following theorem summarizes findings.
Theorem 1. The uniformity metric Luu satisfies Property 1, but violates Properties 2, and
The proof of the above theorem is provided in Appendix C. The violation Property 2 indicates that the uniformity metric LLu is sensitive sample replications, while the violations of Properties and 4 suggest that feature redundancy and dimensional collapse do not reduce the uniformity metric Lu, making this uniformity metric unable correctly reflect feature redundancy and dimensional collapse. Therefore, there a pressing need to develop a new uniformity metric.
4 A NEW UNIFORMITY METRIC
In this section, introduce uniformity metric to address the limitations of Lu-
4

---

Published as a conference paper at ICLR 2024
60
30
Kt. Divvr (K (x0.)
-21
36
Wasser Dist
-4
12
Dimension (loggm)
Dimension (logam)
(a) KL Divergence
(b) Wasserstein Distance
Figure 2: The KL divergence and Wasserstein distance between Y2 and Y, w.r.t. various dimensions.
4.1 THE UNIFORM SPHERICAL DISTRIBUTION AND AN APPROXIMATION
As pointed out by (Wang Isola, 2020), feature vectors should be roughly uniformly distributed on the unit hypersphere Smm-1, preserving as much information of the data as possible. Therefore, we adopt the uniform spherical distribution as our target distribution.
Our approach utilizes the quadratic Wasserstein distance, form of statistical distance, between the feature distribution and the target distribution the new uniformity loss. However, computing any statistical distances involving the uniform spherical distribution can be challenging. To address this, we first establish an asymptotic equivalence between the uniform spherical distribution and the isotropic Gaussian distribution. By adopting Gaussian distribution for the representations, then exploit the fact that the quadratic Wasserstein distance between two Gaussian distributions has closed form involving only the means and covariance matrices, leading to and simple uniformity loss. We need the following fact.
Fact 1. If o' then Y == Z/ || 2 is uniformly distributed on the unit hypersphere Sm-11
Because the average length of |Z|| is roughly o.mm (Chandrasekaran al., 2012), that is,
1 vm.
we expect that Z/(o ~ NO, Anmm/n /m) provides reasonable approximation to Z/||Z||, and thus the uniform spherical distribution. This is partially justified by the following theorem.
Theorem 2. Let Y; be the i-th coordinate of Y = Z/||Z[|2 E RR" where Z ~ (0, Then the quadratic Wasserstein distance between Y, and Y; N 1/m) converges to that is,
co,
lim W (Y, Y2) m0000
Theorem 2 suggests that (0, I../mm approximates the distribution of each coordinate of the uniform spherical distribution as m - c0. It can be proven by first employing the Talagrand T2 inequality (Van Handel, 2016) to upper bound the quadratic Wasserstein distance using the Kullback-eibler (KL) divergence, and then establishing that the Kullback-Leiller (KL) divergence converges to 0. The proof is provided in Appendix B.
We empirically compare the distributions of Y, and Y; across various dimensions m 2, 4, 8, 16, 32, 64, 128, 256. For each m, we sample 200,000 data points from both Y, and Y, bin them into 51 groups, and calculate the empirical KL divergence and Wasserstein distance. Figure plots both distances versus increasing dimensions. We observe that both distances converge to as m increases. Specifically, these results indicate that the distribution of Y, provides a reasonable approximation to that of Y; when m 2 21 = 16. Further comparisons between Y and Y can be found in Appendix D.
4.2 NEW METRIC FOR UNIFORMITY
In this section, discuss how to the quadratic Wasserstein distance between the distribution of learned representations and A (000, Im/m, in place the uniform spherical distribution Unif(S"m m our uniformity loss.
5

---

Published as a conference paper at ICLR 2024
To facilitate computation, adopt Gaussian hypothesis for the learned representations and assume they follow N (H, Σ). With this assumption, we employ the quadratic Wasserstein distance" to measure the distance between two distributions. We need the following well-known lemma (Olkin & Pukelsheim, 1982).
Lemma 1. Then the quadratic Wasserstein distance between Nu, Σ) and (0, 1/m) is
| +1+ +I+tr(2)
tr(Ea). m
The lemma above indicates that the quadratic Wasserstein distance can be easily computed using the population mean and covariance of the representations. In practice, we estimate the population mean and covariance by using the sample mean j and covariance matrix Σ, respectively. Specifically, the empirical quadratic Wasserstein distance serves the new empirical uniformity loss:
W2
| + +1
tr Et) m
2
(8)
Thus, -W2 can utilized as the new uniformity metric, with larger values indicating greater uniformity. Moreover, our new uniformity loss can be seamlessly integrated into various existing self-supervised learning methods to enhance their performance.
COMPARING TWO METRICS
5.1 THEORETICAL COMPARISON
We examine the proposed metric -W₂ in terms of the four properties introduced earlier. The following theorem summarizes our findings.
Theorem 3. The uniformity metric -W2 satisfies all four properties, that is, Properties 1-4.
The proof of the above theorem is similar to that of Theorem 1, and is provided in Appendix C.2. Table compares -Lu and W22. important to highlight that our new uniformity metric is invariant to instance permutations and sample replications, while effectively capturing feature redundancy and dimensional collapse.
example, consider D + 0 versus D. Here, a larger k indicates a more severe dimensional collapse. However, - Lys fails to identify this issue, as
Taking dimensional collapse
Table I: Comparing the two uniformity metrics.
Properties
IPC
ICC
FCC
FBC
Lu
X
(D + 0*) (D). stark contrast, our proposed metric can accurately detect this dimensional collapse, (D 0*) (D).
WW
/
5.2 EMPIRICAL. COMPARISONS VIA SYNTHETIC STUDIES
We perform synthetic experiments to investigate the two uniformity metrics. An empirical examination of the correlation between these metrics shows that data points following an isotropic Gaussian distribution exhibit better uniformity compared to those from other distributions; see Appendix E for detailed results. Additionally, we generate data vectors from this distribution to enable a thorough comparison between the two metries.
On Dimensional Collapse Degrees To generate data reflecting varying degrees of dimensional collapse, we sample data vectors from an isotropic Gaussian distribution, normalize them to have & norms?, and then zero out a proportion of the coordinates. As the proportion of zero-vale vero-value coordinates. denoted by 7) increases, dimensional collapse becomes more pronounced, while the proportion of
We discuss using other statistical distances as uniformity losses, such as the Kullhack-L divergence and Bhattacharyya distance, in Appendix A. In this paper, we always first normalize the representations have unit & norms.

---

Published as a conference paper at ICLR 2024
4.0
0.0
3.2
.0.9
=00
0.8
500
-12
500
m 1000
1000
0.0
-1.5
100
Collapse Level 77 (%)
Collapse Level (%)
(a) Sensitivity to collapse degrees for Cus
(b) Sensitivity to collapse degrees for W/2
Figure 3: Sensitivity to dimensional collapse degrees: -W2 is more sensitive than Le.
non-zero coordinates is 1 - In Figure 3(a) and Figure 3(b), we observe that - WW effectively captures different collapse degrees, whereas Lu remains almost unchanged even with 80% collapse (1) 80%), indicating that - Lu is insensitive to the degrees of dimensional collapse.
-0.4
2.4
0.8
-1.2
A 75%
n 75%
7 50%
-1.6
1) 50%
? 25%
75
0.0
2
10
-2.0
Dimension (logzm)
Dimension (logam)
(a) Effectiveness of -Lu when increasing
(b) Effectiveness of WW of when increasing m
Figure 4: Effectiveness of the metries when increasing dimension m: -Lu fails distinguish different dimensional collapse degrees for large m, while WW2 is always able to.
On Sensitiveness of Dimensions Figure 4 demonstrates that Lue can not distinguish between different degrees of dimensional collapse (1) 25%, 50% and 75%) the dimension m increases (e.g., m > 2 256). In contrast, - W2 only depends on the degree of dimensional collapse and is independent of the dimensions m.
To complement the theoretical comparisons between the two metrics discussed in Section 5.1, we also conduct empirical comparisons in terms of FCC and FBC. ICC comparisons are collected in Appendix E.
On Feature Cloning Constraint We investigate the impact of feature cloning by creating multiple feature clones of the dataset, such as D @ 'D and 'D ② D Φ DD corresponding to one and two times cloning, respectively. Figure 5(a) demonstrates that the value of -Lu increases the number of clones increases, which violates the strict decline in Eqn. (5). In contrast, in Figure 5(b), our proposed metric W2 decreases, satisfying the property.
On Feature Baby Constraint We proceed to analyze the effect of feature baby, where we insert k dimensional zero vectors into each instance of D. This modified dataset is denoted D 0 and we examine the impact of & on both metrics. Figure 6(a) shows that the value - L2 remains constant increases, violating the strict inequality constraint in Eqn. (6). In contrast, Figure 6(b) shows that our proposed metric WW₂ decreases, satisfying the constraint.
Summary of Synthetic Studies In summary, empirical results corroborate theoretical analysis, confirming that our proposed metric W2 outperforms the existing metric -Lu capturing feature redundancy and dimensional collapse.
7

---

Published as a conference paper at ICLR 2024
25
0
n 75%
n=
20
-4- n 50%
-.7
n
25% 155
n - 25%
-2.1
-.88
10
-35
0
10
Times of Feature Clone
Times of Feature Clone
(3)
Lur docs NOT satisfy FCC
(b)
W2 does satisfy FCC
Figure 5: FCC analysis.
4.0
75%
3.2
-0.3
50%
==
2.4
-0.9
75%
0.8
50%
-1.2
n= 25%
10
-1.5
Dimensions of Feature Baby (i k/m)
Dimensions of Feature Baby ( k/m)
Lu does NOT satisfy FBC
(b) WW2 does satisfy FBC
Figure 6: FBC analysis.
EXPERIMENTS
In this section, we integrate the proposed uniformity loss as auxiliary term into various existing self-supervised methods. We then conduct experiments on CIFAR-IO and CIFAR-100 datasets to demonstrate its effectivvness.
Models We conduct experiments on series of self-supervised representation learning models: (i) AlignUniform (Wang & Isola, 2020), which incorporates both alignment and uniformity losses in its objective function; (ii) three contrastive learning methods, namely SimCLR (Chen et al., 2020), MoCo (He et al., 2020), and NNCLR (Dwibedi al., 2021); (iii) two asymmetric models, BYO (Grill et al., 2020) and SimSiam (Chen & He, 2021): (iv) two methods based on redundancy reduction, BarlowTwins (Zbontar al., 2021) and Zero-CL CL (Zhang al., 2022b). To investigate the behavior of the proposed Wasserstein uniformity loss in self-supervised learning, we integrate as an auxiliary loss into the following models: MoCo v2, BYOL, BarlowTwins, and Zero-CI.. Additionally, propose using linear decay to weight the Wasserstein uniformity loss during training. This achieved by setting a; = Qmax 1, (Omax 0min)/T. where I, T, O'max Ominn and o; represent the current epoch, maximum epochs, maximum weight, minimum weight, and current weight, respectively. Further details on the experimental settings can be found in Appendix F.I.
Accuracy and representation capacity We assess the aforementioned methods using two distinct criteria: accuracy and representation quality/capacity. Accuracy is gauged through linear evaluation accuracy, quantified by Top-1 accuracy (Ace @ 1) and Top-5 accuracy (Ace@5). On the other hand, representation quality/capacity is evaluated using the uniformity losses Lu and W2, along with the alignment loss C.A.
As depicted in Table 2, incorporating W2 as an additional loss consistently yields superior performance compared models without this loss or those with Ly as the additional term. Intriguingly, although it marginally compromises alignment, it enhances uniformity and accuracy in downstream tasks. This underscores the effectiveness W2 as a uniformity loss. Notably, integrating the Wasserstein uniformity loss does not impede training or inference efficiency.
Main Results
Convergence Analysis We evaluate the Top-1 accuracy of these models on CIFAR- 10 and CIFAR- 100 using the linear evaluation protocol, as described in Appendix F.2, across different training
8

---

Published as a conference paper at ICLR 2024
Table 2: Main results on CIFAR-10 and CIFAR-100. Proj. and Pred. are the hidden dimensions in projector and predictor. and & indicates gains and losses, respectively.
Mathods
Proj
Prod
Acc
Acc
CIPAR-TO
14:
1
CA1
A0c
CIFAR.II TXI
SinCL& NNCLR
256
$2.85
99.78
1.04
-3.75
0.47
63.43
$0.97
1.05
-3.75
0.50
2556
256
$7.46
99.63
1.23
-3.12
0.38
54,90
83.81
1.23
-3.18
0:43
SumSiata
256
256
50.71
99.67
1.19
-3.33
0.39
56.10
54.34
1.21
-3.29
0.42
Aligntinifrm
256
X
90.37
99.76
0.51
65.08
90.15
0:95
.3.82
0.53
MoCo v2
256
90.65
99.81
1.06
-3.75
0.51
60.27
86.29
1.07
-3.60
0.46
McCo x2-+ Cu
256
90.98
10.30
99.67
0.98
1am
-3.22
0.53 Jone
61.21
Taoe
87.32
0.98
Ta.ce
-3.81
0.52
MoCo v2 We
256
X
0.33
0.63
63.08
53.48
0.28
-3.86
0.66
256
$9.53
99.71
55.81
2.87
HYUL + Lu
256
X
90.89
T0.56
99.75
1.09
10.12
-3.06
0.40
Jam
62,68
Jouse
S3.44
1,08
T0.12
-3.70
0.51
Le:35
RYOL + 212
256
256
90.11
99.77
0.38
-3.90
0.65
1a3
65.16
80.25
0.36
Tass
-3.91
0.69
HariowT
256
91.16
99.30
0.22
-5.91
0.75
68.19
90.64
0.23
-3.91
0.75
256
X
91.38
Tooi
Ja
68.41
90.99
Two0
-3.91
0.76
BurlowTwins W)
255
91.43 1033
99.78
0.19
-3.92
0,76 Ja
68.47
Tooss
90.64
0.19
T0.04
-3.91
0,79
256
X
91.35
99.74
90.97
-3.3
0.75
Zero-CL- CL Lu
256
91.28
Jaun
99.74
0.15
-3.94
0.72
Jo
08.44 Louse
90.91
0.115
-3.93
0.74
Thuos
7ao-CL+
256
X
91.42
Toum
99.82
0.14
Tooi
0.71
Jo
Touec
91.02
-3.94
0.76 1e.0s
epochs. Figure 15 illustrates the results. By incorporating W2 as an additional loss for these models, we observe faster convergence compared to the raw models, particularly for MoCo v2 and BOL. which exhibit significant collapse issues. Our experiments demonstrate that imposing the proposed Wasserstein uniformity metric as an auxiliary penalty loss greatly enhances uniformity but may compromise alignment. We further analyze uniformity and alignment throughout all training epochs in Appendix F.3.
Log of Singater Vahne
BYOL
Lag of Singule Valaes
Log of Singstor Vahees
McCu
Sendian
-20
KALLA
Algaliasmm
-23
MeCe
MeCa dar
BYOL-4 Cy-
MeCo x2 10,
BYOL-W 16
60
125
160
2003
203
290
60
30
130
100
230
340 110
40
12)
150
350
120
252
Socted digttvalee Index
eigeovvlie index
Socted elgeavathe indes
(a) Singular Value Spectra
(b) MoCo v2
(c) BYOL
Figure 7: Dimensional collapse analysis on CIFAR-100 dataset.
Dimensional Collapse Analysis We visualize the singular value spectra of the learned representa- tions (Jing et al., 2022) for various models. These spectra contain the singular values of the covariance matrix of representations from CIFAR-100 dataset, sorted in logarithmic scale order. As shown in Figure 7(a), most singular values collapse to zeros in most models, indicating a large number of collapsed coordinates most models. To further understand how the additional loss W22 helps prevent dimensional collapse, we add W2 as an additional loss for Moco v2 and BYOL, the numbers of collapsed coordinates decrease to zeros in both cases; see Figure 7(b) and Figure 7(c). This verifies that our proposed uniformity loss W2 can effectively address the dimensional collapse issue for Moco v2 and BYOL. In contrast, Lu can not effectively prevent dimensional collapse.
7 CONCLUSION
In this paper, we have identified four principled properties that effective uniformity metric should possess. Namely, an effective uniformity metric should remain invariant to instance permutations and sample replications while accurately capturing feature redundancy and dimensional collapse. Surprisingly, the popular uniformity metric proposed by Wang & Isola (2020) fails to meet the majority of these properties, unveiling its limitations. Empirical investigations corroborate our theoretical findings. To overcome these limitations, we introduce new uniformity metric that satisfies all four properties. Particularly, this new metric demonstrates remarkable abilities to capture feature redundancy and dimensional collapse. Integrating it as an auxiliary loss in various self- supervised learning methods effectively mitigates dimensional collapse and consistently improves their performance downstream tasks. Nonetheless, it is worth noting that the four identified properties may not encompass a comprehensive characterization of an ideal uniformity metric, warranting further exploration.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Benyou Wang was partially supported by the Shenzhen Science and Technology Program (JCYJ202208 103001002), Shenzhen Doctoral Startup Funding (1BS2002200000933066)) and Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608). Qiang Sun was partially supported in part by the Natural Sciences and Engineering Research Council of Canada under Grant RGPIN-201 8-06484 and Data Sciences Institute Catalyst Grant.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. thcoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivarrancc-ovaraaae regularization for self-supervised learning. In ICLR, 2022.
A. Bhattacharyya. On measure of divergence between two statistical populations defined by their probability distributions. Bulletin of the Calcutta Mathematical Society, 1943.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Venkat Chandrasekaran, Benjamin Recht, Pablo A Parrilo, and Alan S Willsky. The convex geometry of linear inverse problems. Foundations of Computational mathematics, 12:805-849, 2012.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Henry Cohn and Abhinav Kumar. Universally optimal distribution of points on spheres. Journal of the American Mathematical Society, 2007.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, N. Sebe, and Elisa Ricci. Solo-learn: A library of self-supervised methods for visual representation learning. JMLR, 2022.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In ICCV, 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. ArXiv, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach self-supervised learning. In NeurIPS, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Geoffrey E. Hinton. Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1 503.02531, 2015.
10

---

Published as a conference paper at ICLR 2024
Tianyu Hua, Wenxiao Wang. Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In ICCV, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR, 2021.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive innguage-image pre-training paradigm. In ICLR, 2022.
David Lindley and Solomon Kullback. Information theory and statistics. Journal of the American Statistical Association, 54:825, 1959.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017.
Ingram Olkin and Friedrich Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications, 48:257-263, 1982.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In /CML, 2021.
Ramon Van Handel. Probability in high dimension. Lecture Notes (Princeton University), 2016.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICML, 2020.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, 2021.
Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. In CVPR, 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. ArXiv, 2017.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. In ICLR, 2022a.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-CL: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2022b.
Xiangyu Zhao, Raviteja Vemulapalli, P. A. Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In ICCV, 2021.
11

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Statistical distances Gaussian distributions
12
B Proof of Theorem 2
13
B.I Proofs for supporting lemmas
14
C Examining the four properties for two uniformity metrics
15
C.I. Proof of Thcorem I: Examining the four properties for - -Lu
15
C.2 Proof of Theorem 3: Examining the four properties for - W22
16
D Further comparisons between and Y
17
E Additional synthetic studies
17
E.1 Correlation between -Lu and WW2
17
E.2 On Instance Cloning Constraint
18
E.3 Understanding Property 4: Why does it relate to dimensional collapse?
19
E,4
Understanding W2: Large means may lead to collapse
19
F Experiment settings and convergence analysis
20
F.I
Experiment settings
20
F2
Convergence analysis for Top-1 accuracy
20
F3
Convergence analysis for uniformity and alignment
21
A STATISTICAL DISTANCES OVER GAUSSIAN DISTRIBUTIONS
We first introduce the Wasserstein distance or the earth mover distance. Definition 1. The Wasserstein distance earth-mover distance with p norm is defined below:
W WP P.) = inf (EII(P..P.)
((,,w)>> [|| "/
where IT,,,,, Py) denotes the set of all joint distributions (xx, y) whose marginals are respectively P, and Iy.. Intuitively, when viewing each distribution as a unit amount of earth/soil, the Wasserstein distance earth-mover distance takes the minimum cost of transporting "mass" from y to transform the distribution IP, into the distribution P,,, This distance is also called the quadratic Wasserstein distance when p
In this paper, mainly exploit the quadratic Wasserstein distance over Gaussian distributions. Besides this distance, we also discuss other distribution distances as uniformity metrics and make comparisons with the Wasserstein distance. Specifically, the Kullback-Leibler divergence and the Bhattacharyya distance over Gaussian distributions are provided in Lemma and Lemma 3 respec- tively. Both distances require full-rank covariance matrices, making them impropriate conduct dimensional collapse analysis. In contrast, our quadratic Wasserstein distance-based uniformity metric free of such a requirement.
Lemma 2 (Kulbbck-Leeble divergence (Lindley & Kullback, 1959)), Suppose two random variables Zi ~ N(1,, En) and Z2 ~ N Σn) obey multivariate normal distributions, then Kullback- Leibler divergence between Z1 and Z2 is:
1 DKL (Z, Zz) (m1 222) E
1 (m 22) tr EE,',, Σ, I) In det Σ₂
det Σ,
12

---

Published as a conference paper at ICLR 2024
Lemma 3 (Bhattacharyya Distance (Bhattacharyya, 1943)). Suppose random variables Z₁ N(p1 Σn) and Zy ~ (uz Σg) obey multivariate normal distributions, E [Σn + Σ2), then bhattacharyya distance between Z1 and Zg is:
1 Db(Z1, Z2) 8 [ E (m1 p2) + In det E det Σn
1
det Σ
B PROOF THEOREM 2
We first need the following lemma, whose proof is collected in the end this section.
Lemma 4. Let ~ (mm)) and Y Z/ 2. Then the probability density function of Y,, the i-th coordinate of Y is:
T(m/2) (1 7m)mm 33/2, V E [-1,1]] (v.) VTI 1)/2)
We are ready to prove Theorem 2.
Proof of Theorem 2. According to the Lemma 4, the pdf of Y; and Y, are:
T(m/2) (1 02)mm-33) 31/2 (x( = v m (y(y) (TT - 1)/2)
myy²
exp(
2 }.
2n
Then the Kullback-Leible divergence between Y, and Y,
S fr. (w) log (v)
DYL(Y.1((.)
[ fy. (w)llog VTI((m 11)/2)
T(m/2)
+ m 3 3
log(1 - -2²) - -oog V 2T
m
myy²
dyy
2
2
= log V m l'(mm 1)/2) + [ fv((
2 T(m/2)
m
log(1 - y²) + may²
2 1dy.
2
Letting = y2, we have v and dy = - du.. Thus,
m 3 3
my
A = Jv. (v)
log(1 - y +
dy
2
2
l(m/2)
3
log(1 122 + my² Jdy
2
y²)
VTT( 1)/2)
2
T(m/2)
(1 - p1)
m- m
log(1 11) + m
2 14| dj
VTl((m- 1)/2)
2
l(m/22
m 3 3
(1
m-3
2
log 1 -d)du
VTT((m 1)/2)
T(m/2)
(1 4) 4 dp.
V - 1)/2)
J
13

---

Published as a conference paper at ICLR 2024
By using the property of Beta distribution, and the inequality that i- log(1 11) S -M4, we have
T(m/2)
m 3
S (1 (1)
A VTT(mm- 1)
1 log(1 - pdd
2
l(m/2)
m - 3
{
(1
11)
m=3
2 p dp
VTT((m- 1)/2)
2
J0
T(m/2)
m
3
3
m
B
and
VTT((m 1)/2)
2
2
2
A2; V (T - -1)
T(m/2)
111
1
p.)
m
1 du
2 J
((m/2)
m
3
B
VTI((m- 1)/2) 2
2
2
Then, for A, we have
1(m/2)
- 3
3
1
((m/2)
3 - 1
A A1
A2 s
B
B
v ( - -1)/2)
2
2'
2
V 1)/2)
3
T(m/2)
3
m -
3
T(m/2) ((3/2)T( 1)/2)
B(
2 V ( - -1)/2)
2'
2
2
-1)/2)
I'mm 2)/2)
3 ((3/2)1((m/2)
3
( 11/22[((m22)
3 l'(m/2)
2 v V 2)/2)
2 TTT (m 2)/2) 4T(mm 2)
Using the Stirling formula, we have [(x aa) + I(x)x= as x + xo and thus
r(m/2)
lim (Y3() = lim log
lim A m l'((m - 1)/2) m-000
m-000
m=+00
2 1 - 11)/2(( -
3 l(m/2)
lim log m-+00
V
m
T((m - 11)22)
lim m0000
4 4T((m + 2)/2)
2
m
3 [(m/2) lim log m-000
m
3
lim m0000
log
V
2
4 (mmm2))
4mn
0.
We further use T3 inequality (Van Handel, 2016, Theorem 4.31) to derive the quadratic Wasserstein metric (Van Handel, 2016, Definition 4.29) as:
2 2 (Y, m
lim W2 8.) m=
lim m0-000
V
B.1 PROOFS FOR SUPPORTING LEMMAS
Proof of Lemma 4. Let Z Zz,Z..... 2m] ~ O( then Z, NN o²), Vi E [1, m].
Let U = Zi/o~ ~ (O, 1), V Enp( (2/00 x² 1), then U and V are independent with follows the Student's t-distribution with 1
each other. The random variable /V/mm-1)
degrees of freedom, and its probability density function (pdf) is:
l2mm/2) V(m 1m - 1) (1+ +
82
(r(()
1 )mm/2
For random variable Y,, we have
Z,
Z,
2
U
Y, =
ZZ
22
Em Z3
t
((zy/o)2+ + t (2//0))
VV
14

---

Published as a conference paper at ICLR 2024
and then T 1)
vm-1V; IY; VI-Y?
Y,
T
Therefore, the cumulative distribution
vT=+m-1
function (edf) is:
P((Y, S v1))
(y) P([ S (13)
yi
P(LY, S 0}) + P(10 Y; S y1 >
P( T
4m /11
y4 S 0 > 0
P(( VT*+m-I 5 + P((0 < -T=+m-I
S vb11
P(IT+4m-1 2 -T,
T2
31 S
P(I S 0} + P(( T T-
T
S V. N 0})
9: > 0
P(I S -m-lmk) }}
91 9:50
V1-y,
P([ S 50} + P((O T
ym-ly b/
})
y1 >
V1-1,
Vm Ty: 11) )) Fr(
Vm 11/i
P((T &
V1 V1-
v1 V/
The probability density function of Y; can then be derived as:
d
d
Vm- 13
(v:
(y))
Fr(
dy
dy v Vi
.r(( m - Iy/i d ym Iyli
V1 vi dyj v1 1/5
(mm/22)
v(m 1)aT((())))) 1)/2) (1 m/
I(1 v/² -3/2
T(m/2)
y22 (mm (m-33/22
VTl((mm- - 1)/2)
C EXAMINING THE FOUR PROPERTIES FOR TWO UNIFORMITY METRICS
C.I PROOF OF THEOREM 1: EXAMINING THE FOUR PROPERTIES FOR CLu
Property can be easily verified for - Lu and thus we skip the verification. We only examine the other three properties for the uniformity metric -Lu.
First, we prove that -Lu does not satisfy Property 2. Due to the definition of Lu in Eqn. (2), we have
Cu(D W DD) := log 1
2,
Σ
2 113
2n(2n 1)/2
1=1
log 2n 2n(2n 1)/2
1
[| - 2
i= 1=1
(10)
Letting G 5n E/1 -1,,,, we have
i-1
G
f|| 2 2,
- = =n(n - 1)/2,
1=2 1=1
2=2
15

---

Published as a conference paper at ICLR 2024
and G n(n 1)/2 if and only 21 Z2
Zn. Thus
Lu w D) - (D) = log 4G +
G
2n(2n 1)/2 - log
1e(1 1)/2
(4G + n)n(n - 1)/2 log (4G + n - 1)
= log
2nC((n 11/2
4nG -G
4nG 4G + n²²
- n 2 log 1 = 0.
= log 4nG 22
The above equality holds if and only if = n(n 1)/2, which requires 2 22 = ... Zn trivial case when all representations collapse one constant point. We have excluded this trivial case, and thus (D A D) < (D). Therefore, the uniformity metric Lu does not satisfy Property
Second, we prove that - Lu does not satisfy Property 3. Letting 2 z4 @ 2, and 233 23 L 2g. we have
Lu log n(n 1)/2 r||;; 2,
By the definitions of 2; and Zj. have ||a,||2 v2 2(2,, 2z)- Thus
/2|121122 2, 12ا122 v2| 12, and (2,,
22 2, = 2 112 + 2112,11 4(z,, zy) 2/|2, 1,1 ž Iz; 2
Therefore, CL (D D)= 2 (D)) indicating that the uniformity metric Lu does not satisfy the Property 3.
Third, we prove that the existing metric - Lur does not satisfy the Property 4. Letting ă, @ 0 and 2, = + 0*, we have
(D 0*) = log n(n 1)/2 1=2 j=1
1
[| 11R.-2,1 2, 13
By the definitions of 2, and Zj, we have ||ã:|12 = ||| 1|, | I|2 = 2 112. (2,, 21) = (2,, 2z), thus
and
i2 2, 1 1ق 2(2, 2,) = ||2,1 +
||2 2(z Za} ||z, Z, l12.
Therefore, (D - 0.) = (D), indicating that the uniformity metric - Lu does not satisfy Property 4.
C.2 PROOF OF THEOREM 3: EXAMINING THE FOUR PROPERTIES FOR -W2
Property can be easily verified for - W2, and thus the proof is skipped. We only examine the rest three properties for the proposed uniformity metric - We.
First, we prove that our proposed metric W2 satisfies Property 2. Let jt and Σ be defined above, for D D D iz1, Z2,
Zan Z1, Z2;
Zn , the mean and covariance estimators are
1 2n
2z.
Í Σ
2(z, 2(z.
MJJ (z.
2n 1=1
1=1
which agree with those for D. Then we have
2 tr($112) W2(D).
(2 | + 1
Vm
Therefore, our proposed metric - W2 satisfies Property 2.
Second, we prove that - W/2 satisfies Property 3. Let 2, = 24 ① 2, E R22m and covariacce estimators are:
For @ D, the mean
1
Σ =
Σ Σ
11
Σ
Σ
16

---

Published as a conference paper at ICLR 2024
We easily have
1 11/2/V2 //2/12 1//2/v8 /1/2/22/
tr(2) -2rrEE) and tr(S4/2) (2
Thus
W2 DD
1 +1+ (r(2)
tr (112))
22m
2V2 tr(E1/2)
211 +1+2tr($)
22
|A +1+
tr(( = W2(D).
mm
Therefore, W2 + D) < (D), indicating that our proposed metric W2 could satisfy the Property 3.
Third, we prove that our proposed metric W2 satisfies Property 4. Let z, @ 0 E R with an overload of notation. For + 0%, the sample mean and covariance estimators are
i 0*
Σ
Σ
m
0 fxm
0
where ğ and Σ are defined previously. Therefore, we have tr tr(2), tr((2/1) = tr(E1/)), and thus
2
W2 + 0*)
1
1 tr
tr 11/2)
Vm -
2
|10112 + 11 tr(())
tr($1122)
vm + k
2
|1012 + 1+ tr
(
tr (/12)) = W2(().
m
Therefore,
WW 0%) Wb((D). indicating that our proposed metric WW2 satisfies the Property 4.
D FURTHER COMPARISONS BETWEEN Y AND Ý
This section further compares the distributions of Y and Ý
We visually compare the distributions of Y, and Y;. To estimate the distributions of Y, and Ye, we bin 200,000 sampled data points into 51 groups. Figure compares the binning densities of Y; and Y when m E {2, 4, 8, 16, 32, 64, 128, 256}. We can observe that two distributions are highly overlapped when is moderately large, e.g.,
> 16.
By binning 2,000,000 data points into 51 x 51 groups in two-axis, we also analyze the joint binning densities and present 2D joint binning densities of (Y,, Y2) (i * j) in Figure 9()) and (Y, K,) (i (i)) j) Figure 9b)). Even if is relatively small (i.c., 32), the densities of the two distributions are close.
E ADDITIONAL SYNTHETIC STUDIES
E.1 CORRELATION BETWEEN -Lu AND W
17

---

Published as a conference paper at ICLR 2024
Doaty O R100
RHD
(a) 772 = 2
(b) m 4
(c) m7 = 8
(d) m 16
Yumon
Yuriic
Vanulc
Vantlfc
(e) m 32
(I) m
64
(g) mr 128
(h) mr 256
Figure 8: Comparing the binning densities of Y, and Y, with various dimensions.
Stasing Danety
Damily
(a) Density for Y; and Y;
(b) Density for V and Y
Figure 9: Visualization of two arbitrary dimensions for Y and Y when = 32.
We employ synthetic experiments to study the uniformity metrics across different distributions. Specifically, we sample 50,000 data vectors (m = 256) from different distributions, such as the isotropic Gaussian distribution (O I), the uniform distribution on the hyperrectangle 1]. and the mixture of Gaussians, etc. Then we nor- malize these data vectors, and estimate the uniformity of different distributions by two metrics. As shown in Fig. 10, isotropic Gaussian distribution achieves the maximum val- ues for both W2 and -Ly, which indicates that isotropic Gaussian distribution achieves larger uniformity than other distributions. This empirical result is consistent with Fact that the isotropic Gaussian distribution (approximately) achieves the maximum uniformity.
Mote
Steded
Chappart
Eqpactia
0.9
Figure 10: Uniformity analysis for vari- ous distributions by two metrics.
E.2 ON INSTANCE CLONING CONSTRAINT
In this section, we compare the two metrics in terms of Property 2 (ICC). Specifically, we randomly sample 1,000 data vectors from the isotropic Gaussian distribu- tion (m 32) and then mask 50% of their coordinates with zeros, forming new dataset 'D with an overload of notation. To investigate the impact of instance cloning, we create multiple clones of the dataset, such D y D and D J Du w D, which correspond to one and two times cloning, respectively. We evaluate the two metrics these datasets. Figure 11 shows that the value of -L slightly decreases as the number of clones increases, in-
-W3
0.75
-1.76
3.5
00.78
.0.99
Tince of Iintaane Clane
Figure 11: ICC analysis.
18

---

Published as a conference paper at ICLR 2024
(a) Two-dine Two-dimensional visu- alization with no collapsed dimension Figure 12: A case study for Property and blue points are data vectors.
(b) three-dimensional visualiza-
(c) Three- dimensional visualiza-
tion with one collapsed dimen- tion with no collapsed dimension
sion
dicating that Luu violates the equality in Equation 4. In contrast, proposed metric - WW2 remains constant, satisfying the equality.
E.3 UNDERSTANDING PROPERTY 4: WHY DOES RELATE TO DIMENSIONAL. COLLAPSE?
This section delves into Property through case studies. Let us begin with thought experiment. Consider a dataset with instances uniformly distributed on the unit hypersphere, thereby possessing (almost) maximal uniformity. When additional coordinates with zeros inserted to each instance of D, forming a new dataset @ 0 can no longer maintain maximal uniformity. This is because, the new dataset only occupies small area of the unit hypersphere. Consequently, k increases, the uniformity of the dataset would decrease significantly.
Let visualize this thought experiment using synthetic studies. In Figure 12(a), present 400 data vectors (D1) sampled from NO,, I2), which are also nearly uniformly distributed on S By inserting one zero-coordinate each instance of P1, we obtain a new dataset Dj @ 0 as depicted in Figure 12(b). We also construct another dataset 'D2 consisting of 400 data vectors sampled from (0, Is), visualized Figure 12(c). Notably, Dj @ 0² forms a ring on S while D2 is almost uniformly distributed over S², Naturally, (((2)) > U(D; @ 0²). (D) = UD2))* then ((D)) = M(22) > U(D, e 0²). This partially confirms the validity of Property 4.
Additionally, increasing the value of & in Property 4 exacerbates the degree of dimensional collapse. To lustrate, consider dataset D sampled from multi- variate Gaussian distribution (O Im/m), exhibiting a collapse degree close 0%. However, upon inserting m-dimensional zero-value vectors to each instance of D. denoted as D 0th half of the dimensions collapse.
Log Singglaa Values
-5
k- 0*
-10
k=1*m
-5
k=
km 3m3
k 4*
Consequently, the collapse degree increases to 50%. Fig- ure 13 visually represents the collapse of D @ 0 using the singular value spectra of the representations. It is evident that a larger & results in a more pronounced dimensional collapse. In summary, Property 4 corresponds to dimensional collapse.
-25
100
Percenlage of Sorted eigenvalue index (%)
Figure 13: Singular value spectrum of D
0
E.4 UNDERSTANDING W3: LARGE MEANS MAY LEAD TO COLLAPSE
In this section, we explore our uniformity loss W3. This loss embodies two primary constraints. Firstly, it promotes the covariance matrix to be isotropic (specifically In/m). Secondly, it enforces the mean to be zero. The latter constraint on the mean is crucial. To illustrate, we present a case study demonstrating that deviating the mean from zero compromises uniformity, even if the covariance matrix is precisely I,, and thus isotropic. Means deviating from zero may result dimensional collapse and even constant collapse.
'Intuitively, maximal uniformity should stay constant regardless of dimensions; otherwise the corresponding uniformity metric exhibit preference for larger or smaller dimensions.
19

---

Published as a conference paper at ICLR 2024
(2)
(b) (0.5 1,
(c) M(1 1,12)
(d) A/2 1,
(e) ((4 1,
(l) N 1,1,)
(g) N16 1,
(h) W/32 12)
Figure 14: Visualizing & normalized Gaussian vectors with different means.
Table 3: Parameter settings for various models in the experiments.
Models
MoCo v2
BYOL
Barlow Twins
Zero-CL
Qmax
1.0
0.2
30.0
30.0
Omin
1.0
0.2
0
30.0
Assuming X E IR2 follows a Gaussian distribution (0, I2), let Y X k: such that Y ~ Alk 1,), where 1 E R represents a vector of all ones. We vary le from 0 32 and visualize the ég-normalized Y's in Figure 14 (by generating multiple independent copies). It is clear that an excessively large means will cause representations to collapse to a single point, even the covariance matrix is isotropic.
F EXPERIMENT SETTINGS AND CONVERGENCE ANALYSIS
F.1 EXPERIMENT SETTINGS
To ensure fair comparisons, all experiments in Section 6 are conducted on a single 1080 GPU. Additionally, maintain consistency in network architecture across all models, utilizing ResNet- 18 (He et al., 2016) as the backbone and three-layer MI.P as the projector. The LARS optimizer (You al., 2017) is employed with a base learning rate of 0.2, accompanied by cosine decay learning rate schedule (Loshchilov & Hutter, 2017) for all models. Evaluation follows a linear evaluation protocol, where models are pre-trained for 500 epochs. Evaluation involves adding linear classifier and training the classifier for 100 epochs while preserving the learned representations. The same augmentation strategy deployed across all models, encompassing various operations such as color distortion, rotation, and cutout. Following da Costa et al. (2022), we set the temperature / = 0.2 for all contrastive learning methods. For MoCo (He et al., 2020) and NNCLR (Dwibedi al., 2021), which require additional queue to store negative samples, we set the queue size to 2 Regarding the linear decay for weighting the quadratic Wasserstein distance, refer to Table for the parameter settings.
F.2 CONVERGENCE ANALYSIS FOR TOP-1 ACCURACY
Here we illustrate the convergence of Top-1 accuracy across all training epochs in Fig 15. Throughout the training, capture the model checkpoint at the end of each epoch to train linear classifier. We subsequently evaluate the Top-1 accuracy on unseen images from the test set (either CIFAR-10 or CIFAR-100).
20

---

Published as a conference paper at ICLR 2024
For both CIFAR-10 10 and CIFAR-I00, 100,, observe that integrating the proposed uniformity metric as an auxiliary loss significantly enhances the Top-1 accuracy, particularly in the initial stages of training.
F.3 CONVERGENCE ANALYSIS FOR UNIFORMITY ALIGNMENT
This section presents the convergence of the uniformity metric and alignment loss across all training epochs in Figure 16 and Figure 17, respectively. Throughout the training, we record the model checkpoint at the end of each epoch to evaluate the uniformity using the proposed metric W2 and alignment (Wang& & Isola, 2020) on unseen images from the test set (either CIFAR-10 or CIFAR-100).
For both CIFAR-10 and CIFAR-100 100, we observe that integrating the proposed uniformity metric as an auxiliary loss significantly improves uniformity. However, it also slightly compromises alignment (where smaller alignment loss indicates better alignment). It should be noted that improved uniformity often leads to worse alignment.
BYOL- W5
Baite Vy Bacioolutee
Accunng
MaCo
Accuncy
IVVEE
Top- Accurayy
Top
Epocts
Epochs
Epocia
(a) MoCo v2 on CIFAR-10 10
(b) BYOL on CIFAR-10 10
(c) BarlowTwing on CIFAR-10
BYOL- IVDL.
Baziee
pop- Acomay
Top- Acouazy
106
200
303
450
202
130
272
306
400
102
200
1000
300
Epocts
Epochs
Epocis
(d) MoCo v2 on CIFAR-100
(c) BYOL on CIFAR- 100
(f) BarlowTwins on CIFAR-100
Figure 15: Convergence analysis for Top-1 accuracy during training.
Ma
BYOL. BYOL
0.3
M2Crr12
40
BalcnTwio
BatosTTwaa
41.5
20
366
300
182
202
300
416
550
-1.
100
203
302
430
30
Epochs
Epocks
Epocts
(a) MoCo v2 on CIFAR-10
(b) BYOL on CIFAR-10
(c) Barlow on CIFAR-10
MarCa
DVOO
4.6
0.9
-29
RadoxTTitc:
BalooTTwio
100
150
-3.5
183
302
216
-15
250
70
Egochc
Epochs
Epoch
(d) MoCo v2 on CIFAR-100
(c) BYOL on CIFAR-100 100
(f) BarlBarlwwwa Twins on CIFAR-100
Figure 16: Visualizing uniformity during training
21

---

Published as a conference paper at ICLR 2024
0.2
0.5
0.6
8.4
0.6
0.4
0.4
03
0.2
BecoeTwite
Epodts
Epochs
Epaclo
(a) MoCo v2 on CIFAR-10
(b) BYOL CIFAR-10
(c) Barlow Twins on CIFAR-10
1.0
1.0
1.0
02
McCu 12 11g
BYOL + 11%
Batlee Evits Wb
Molia
BYOL
Batton Totee
166
200
300
-003
502
330
430
500
160
300
360
360
Epodtt
Epuchs
Epoclo
(d) MoCo v2 on CIFAR-100
(e) BYOL CIFAR-100
(I) BarlowTwins on CIFAR-100
Figure 17: Visualizing alignment during training.
22


Reviews:
[{"review_id": "qeUY9DAvpd", "reviewer": "Reviewer_RABe", "summary": "The authors propose criteria for a uniformity loss during representation learning. This criteria is met through the use of the quadratic Wasserstein distance between learned representations and a uniform Gaussian distribution as a loss function, promoting uniformity without reduction in rank of the learned representation. Empirically this is shown to improve performance on the CIFAR-10/100 for a variety of models.", "questions": "Are there any results for differing representation dimensions?\n\n(Not a negative comment, just hoping to get some insight into the behavior of the loss) Is there any intuitive reason as to why the top-5 accuracy decreases for MoCo and BarlowTwins on CIFAR-10 using the proposed loss even though the top-1 accuracy increases?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written, clearly stating the desired criteria and showing the proposed approach meets these criteria. The overall motivation behind the desired properties for a representation learning loss seem reasonable with a combination of theoretical justification and intuitive explanation/visualization provided.\n\nEmpirically, the results show an improvement by adding the proposed loss function. The authors do a nice job of evaluating the impact of this additional loss term for a variety of approaches.", "weaknesses": "The main weakness of this paper is that experiments are only done for the CIFAR-10/100 datasets. Given the proposed approach is claiming that dimensional collapse represents a fundamental issue with representation learning, the impact/relevance of this claim would be significantly strengthened by showing dimensional collapse is a fundamental issue and not necessarily a product of the CIFAR datasets.\n\nOn a related note it would be interesting to see the impact of the proposed metric on differing representation dimensions. In particular would the performance improvements still be observed if the representation dimension was doubled or tripled given that property 5 penalizes constant dimensions, intuitively making it seem that this would require careful selection of the representation dimension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0zfLwAShu", "reviewer": "Reviewer_KkL8", "summary": "This paper explains the shortcomings of the existing uniformity metric and the potential dimensional collapse then the authors propose new metrics of uniformity. Numerous experiments have proved its effectiveness.", "questions": "Why the new metric has a relatively small performance increase on zero-CL?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The uniformity metrics proposed in this paper are more comprehensive and their validity has been verified experimentally.", "weaknesses": "Can the authors further elaborate on the importance of ICC, FCC and FBC to better understand the difference between the proposed metric and previous metrics?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmzLeyOnWa", "reviewer": "Reviewer_a1G3", "summary": "In this paper, the author revisits the alignment and uniformity property in Self-Supervised Learning and shows that the metric in [1] lacks the ability to measure dimensional collapse, which is a phenomenon where the representations learned through self-supervised learning span a low-dimensional subspace instead of being distributed uniformly in the representation space. Therefore, the author proposes a new metric utilizing the Wasserstein distance between the distribution of the representation space and the normalized isotropic Gaussian distribution.\n\n[1] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.", "questions": "Please refer to Weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper includes a lot of empirical analysis to demonstrate the effectiveness of the proposed metric.\n2. The paper is well-written and easy to follow.", "weaknesses": "1. The font size of the figures is too small.\n2. It should be made more clear why the original metric doesn't correspond to the proposed 5 properties and how these properties are related to dimensional collapse.\n3. It should be made more clear why dimensional collapse is a undesired property in self-supervised learning.\n4. Since $\\mathcal{W}_2$ is also related to the covariance matrix of representation, what is the relationship between the proposed method and Barlow Twins/VICReg?\n5. Why is the KL divergence of $Y$ and $\\hat{Y}$ not presented for comparison as in Figure 3?\n6. As in [2], the phenomenon of dimensional collapse happens in the embedding space. However, in [2], when the projector is presented, the representation space doesn't collapse. Therefore, why do the singular values of the representation in Figure 8 collapse?\n[2] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3p6nEsVpeG", "reviewer": "Reviewer_u9aL", "summary": "This study develops a new metric assessing the robustness of learnt representations by measuring collapse degree of them. The authors first point out the weakness of previous metrics proposed by Wang & Isola (2020) in its insensitivity with dimensional collapse. In addition, five characteristics are determined as the criteria for the ideal metric. The new metric (*Wasserstein Distance* between normalized learnt representations’ distribution with zero-mean isotropic Gaussian distribution) is introduced satisfying all these criteria, and illustrates its sensitivity towards dimensional collapse. Empirically, this metric help boosting existing frameworks’ performance in downstream tasks.\n\n\nReference:\n\nWang, T., & Isola, P. (2020, November). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In *International Conference on Machine Learning* (pp. 9929-9939).", "questions": "- Can the authors empirically justify if the proposed loss outperform existing loss when being incorporated with other frameworks?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Contribution:\n    - This study stems from a good motivation: The lack of current researches explicitly dealing with representation collapse - here dimensional collapse.\n    - 5 listed criteria for the ideal ‘uniformity’ metric are logically/mathematically sensible. By pointing out the failure of previous work’s metric (Wang & Isola, 2020), the authors clarify the need for a new metric.\n    - The experiments (e.g. Table 2) compare various vanilla approaches with their variance with the proposed metric. These are representative frameworks following three existing directions in dealing with constant collapse. This suggest the potential robustness of proposed metrics when being incorporated to a wide range of models.\n\n- Presentation:\n    - The intuitive flow makes it easy catch on what the authors want to deliver.\n    - The authors provide trackable mathematical notations and derivations.", "weaknesses": "### \n\n- While the 5 criteria are sensible, they can be considered as **necessary conditions** for a good metric. However, no assessment made to ensure they are **sufficient** to construct an ideal one.\n- The key contribution of the work - new ‘uniformity’, currently rely on the assumption that the learnt representations follow a Gaussian distribution, which, in turn, can fall apart and the metric can cause undesired effect.\n- For experiments, only what related to collapse analysis there exists the comparison between Wang & Isola’s ‘uniformity’ metric with the proposed one. We can conclude nothing on the better performance of the proposed WD-based loss when fused with existing frameworks. It seems not convincing for developing a new loss if the existing loss still works better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)