LDReg: Local Dimensionality Regularized Self-Supervised Learning
Authors: Hanxun Huang
Ricardo J. G. B. Campello
Sarah Monazam Erfani
Xingjun Ma
Michael E. Houle
James Bailey
Venue: ICLR 2024 poster (2024)
Keywords: Self-supervised learning
Local Intrinsic Dimensionality
Dimension Collapse
Contrastive Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: LDReg improves the representation quality of self-supervised learning method by regularizing dimensionality

Abstract: Representations learned via self-supervised learning (SSL) can be susceptible to dimensional collapse, where the learned representation subspace is of extremely low dimensionality and thus fails to represent the full data distribution and modalities.
Dimensional collapse ––– also known as the "underfilling" phenomenon ––– is one of the major causes of degraded performance on downstream tasks. Previous work has investigated the dimensional collapse problem of SSL at a global level. In this paper, we demonstrate that representations can span over high dimensional space globally, but collapse locally. To address this, we propose a method called *local dimensionality regularization (LDReg)*. Our formulation is based on the derivation of the Fisher-Rao metric to compare and optimize local distance distributions at an asymptotically small radius for each data point. By increasing the local intrinsic dimensionality, we demonstrate through a range of experiments that LDReg improves the representation quality of SSL. The results also show that LDReg can regularize dimensionality at both local and global levels.

Published as a conference paper at ICLR 2024
LDREG: LOCAL DIMENSIONALITY REGULARIZED SELF-SUPERVISER LEARNING
Hanxum Huang! Ricardo J. G. B. Campello 2 Michael E. Houle James Bailey'
Sarah Erfani! Xingjun Ma²
SScooo of Computing and Information Systems, The University of Melbourne, Australia *Department of Mathematics and Computer Science, University of Southern Denmark, Denmark SSchool of Computer Science, Fudan University, China 'Department of Computer Science, New Jersey Institute of Technology, USA
ABSTRACT
Representations learned via self-supervised learning (SSL) can be susceptible dimensional collapse, where the learned representation subspace is of extremely low dimensionality and thus fails to represent the full data distribution and modali- phennommnn
ties. Dimensional collapse is one of the major causes of degraded performance on downstream tasks. Pre- vious work has investigated the dimensional collapse problem of SSL at a global level. In this paper, we demonstrate that representations can span over high dimen- sional space globally, but collapse locally. To address this, we propose method called local dimensionality regularization (LDReg). Our formulation is based on the derivation of the Fisher-Rao metric to compare and optimize local distance distributions at an asymptotically small radius for each data point. By increasing the local intrinsic dimensionality, demonstrate through range of experiments that LDReg improves the representation quality of SSL. The results also show that LDReg can regularize dimensionality at both local and global levels.
also known as the "underfillin"" phenomenon
INTRODUCTION
Self-supervised learning (SSL) is now approaching the same level of performance as supervised
learning on numerous tasks (Chen
2020a by He et al. 2020, Grill et 2020 Chen He
2021 Caron et al.] 2021 Zbontar et al. 2021 Chen et al. 2021: Bardes et al. 2022 Zhang et al.
SSL focuses on the construction effective representations without reliance on labels. Qual- ity measures for such representations are crucial to assess and regularize the learning process. A key aspect of representation quality is to avoid dimensional collapse and its more severe form, mode collapse, where the representation converges to trivial vector (Jing et al., 2022). Dimensional col- lapse refers to the phenomenon whereby many of the features are highly correlated and thus span only lower-dimensional subspace. Existing works have connected dimensional collapse with low quality of learned representations (He & Ozay 2022 Li et al.) 2022 Garrido et al. [2023a] Dubois al., 2022). Both contrastive and non-contrastive learning can susceptible dimensional col- lapse TTan et al. 2021 Jing et al. 2022 Zhang et al.) 2022). which can be mitigated by regularizing dimensionality global property, such as learning decorrelated features (Hua et al. 202T) or min-
2022
imizing the off- dif-diagonal terms of the covariance matrix (Zbontar et al. 2021 Bardes et al. 2022).
In this paper, we examine an alternative approach the problem of dimensional collapse, by in- vestigating the local properties of the representation. Rather than directly optimizing the global dimensionality of the entire training dataset (in terms of correlation measures), we propose to regu- larize the local intrinsic dimensionality (LID) (Houle 2017a] b) at each training sample. We provide an intuitive illustration of the idea of LID in Figure Given representation vector (anchor point) and its surrounding neighbors, if representations collapse to a low-dimensional space, it would re- sult in a lower sample LID for the anchor point (Figure la In SSL, each anchor point should be dissimilar from all other points and should have higher sample LID (Figure lbp. Based on LID, we reveal an interesting observation that: representations can span a high dimensional space glob- ally, but collapse locally. As shown in the top subfigures in Figure Tc the data points could span

---

Published as a conference paper at ICLR 2024
over different local dimensional subspaces (LIDs) while having roughly the same global intrinsic dimension (GID). This suggests that dimensional collapse should not only be examined as a global property but also locally. Note that Figure Ic illustrates synthetic case of local dimensional col- lapse. Later we will empirically show that representations converging to a locally low-dimensional subspace can have reduced quality and that higher LID is desirable for SSL..
Anchor Point -
Anchor Point LD=5
Local timensional tapiapse
m Gi+2a
14 4.14
0.5
1.0
(a) Local collapse
(b) local collapse
(c) Local dimensional collapse
Figure 1: Illustrations with 2D synthetic data. (a-b) The LID value of the anchor point (red star) when there (or is no) local collapse. (c) Fisher-Rao (FR) metric and mean LID (mL.ID estimates. FR measures the distance between two LID distributions, and is computed based on theoretical results. mLID the geometric mean of sample-wise LID scores. High FR distances and low mLID scores indicate greater dimensional collapse. Global intrinsic dimension (GID) is estimated using the DanCO algorithm (Ceruti et al. 2014
To address local dimensional collapse, we propose the Local Dimensionality Regularizer (LDReg), which regularizes the representations toward desired local intrinsic dimensionality to avoid col- lapse, as shown at the bottom subfigure of Figure Tc. Our approach leverages the LID Representation Theorem (Houle (017a), which has established that the distance distribution of nearest neighbors in an asymptotically small radius around given sample guaranteed to have a parametric form. For LDReg to be able to influence the learned representations toward a distributionally higher LID, we require a way to compare distributions that is sensitive to differences LID. This motivates us to develop new theory enable measurement of the 'distance' between local distance distribu- tions, as well as identify the mean of a set of local distance distributions. We derive a theoretically hell-founded Fisher-Rao metric (FR), which considers statistical manifold for assessing the dis- tance between two local distance distributions the asymptotic limit. As shown in Figure Tc FR corresponds well with different degrees of dimensional collapse. More details regarding Figure Ic can be found in Appendix A
The theory we develop here also leads two new insights: i) LID values are better compared using the logarithmic scale rather than the linear scale; ii) For aggregating LID values, the geometric mean more natural choice than the arithmetic or harmonic means. These insights have consequences for formulating our local dimensionality regularization objective, as well as broader implications for comparing and reporting LID values in other contexts.
To summarize, the main contributions of this paper are:
A new approach, LDReg, for mitigating dimensional collapse in SSL via the regularization of local intrinsic dimensionality characteristics.
Theory to support the formulation of LID regularization, insights into how dimensionalities should be compared and aggregated, and generic dimensionality regularization technique that can potentially be used in other types of learning tasks.
Consistent empirical results demonstrating the benefit of LDReg in improving multiple state-of-the-ard SSL methods (including SimCLR, SimCLR- Tuned, BYOL, and MAE), and its effectiveness in addressing both local and global dimensional collapse.
2
RELATED WORK
Self-Supervised Learning (SSL). SSL aims to automatically learn high-quality representations without label supervision. Existing SSL methods be categorized into two types: generative
2

---

Published as a conference paper at ICLR 2024
methods and contrastive methods. In generative methods, the model learns representations through a reconstruction of the input (Hinton & Zemel 1993). Inspired by masked language modeling (Ken- ton & Toutanova, 2019), recent works have successfully extended this paradigm to the reconstruc- tion of masked images (Bao et al., 2022 Xie et al. 2022), such as Masked AutoEncoder (MAF) (He al.] 2022). It has been theoretically y proven that these methods are special form of con- trastive learning that implicitly aligns positive pairs (Zhang et al. (2022). Contrastive methods can further be divided into 1) sample-contrastive, 2) dimension-connrrasive (Garrido et al. 2023b), and 3) asymmetrical models. SimCLR (Chen et al. 2020a) and other cumple-contrastin methods (He) al., 2020 Chen et al. 2020a b Yeh et al. 2022) are based on InfoNCE loss (Oord et al. 2018)
The sample-contrastive approach has been extended by using nearest-neighbor methods (Dwibedi
jtt al. 2021 Ge et al. 2023), clustering-based methods (Caron et al., 2018 2020 Pang et al. 2022).
methods (Zhontar cct al. 2021 Bardes et al., 2022) regularize the off-diagonal terms of the covariance matrix of the embedding. Asymmetrical models use an asymmetric architecture, such as additional predictor
and improved augmentation strategies (Wang et al. (003). Dimension-contraatee
(Chen & He (021). 1 self-distillation (Caron et al. 2021), or a slow-moving average branch as in BYOL (Grill et al. 2020).
Dimensional collapse in SSL. Dimensional collapse occurs during the SSL process where the learned embedding vectors and representations span only lower-dimensional subspace (Hua al.) 2021 Jing et al. 2022 He & Ozay] 2022 [i et al. 2022). Generative methods such as MAH (He)
ct al. 2022) have been shown be susceptible dimensional collapse (Zhang et al. (022). Sample- contrastive methods such as SimCLR have also been observed to suffer from dimensional collapse (Jing et al. 2022). Other studies suggest that while stronger augmentation and larger projectors are beneficial to the performance (Garrido et al. 2023b). they may cause dimensional collapse in the projector space (Cosentino et al.J (022). It has been theoretically proven that asymmetrical model methods can alleviate dimensional collapse, and the effective rank (Roy & Vetterli (007) is use- ful measure of the degree of global collapse (Zhuo et al. 2023). Effective rank also helpful in assessing the representation quality (Garrido et al., 2023a). By decorrelating features, dimension- contrastive methods (Zbontar et al.) 2021] Zhang et al. 2021; Ermolov et al.) [222]] [ardes et al.) 2022) can also avoid dimensional collapse. In this work, we foeus on the local dimensionality of the representation (encoder) space, which largely determines the performance of downstream tasks.
Local Intrinsic Dimensionality. Unlike global intrinsic dimensionality metrics (Pettis al. 1979
Bruske & Sommer (998), local intrinsic dimensionality (LID) measures the intrinsic dimension in
the vicinity of a particular query point (Levina & Bickel 2004 Houle 2017a). It has been used as
measure for similarity search (Houle et al. 2012), for characterizing adversarial subspaces (Ma et al., 2018ap, for detecting backdoor attacks (Dolatabadi et al.) 2022). and in the understanding of deep learning (Ma et al. [011b)] Gong et Ansuuii
Ansuini et al. 2019 Pope et al. (202I). In Appendix B
we provide comparison between the effective rank and the LID to help understand local vs. global dimensionality. Our work in this paper shows that LID is not only useful as descriptive measure, but can also be used as part of a powerful regularizer for SSL.
BACKGROUND AND TERMINOLOGY
We first introduce the necessary background for the distributional theory underpinning LID. The dimensionality of the local data submanifold in the vicinity of a reference sample is revealed by the growth characteristics of the cumulative distribution function of the local distance distribution.
Let F be real-valued function that non-zero over some open interval containing r E IR, R 00.
Definition ((Houle 2017a)). The intrinsic dimensionality of F' at is defined as follows, whenever the limit exists:
In (F((()))))))))) IntrDimp(r) A lim
c+0 In((1+c)r/rr
Theorem ( (Hule 2017a)). F is continuously differentiable at r, then
LIDx(r) A F') IntrDime(r).
F(r)
Although the preceding definitions apply more generally, we will be particularly interested func- tions F' that satisfy the conditions of a cumulative distribution function (CDF). Let x be location of
3

---

Published as a conference paper at ICLR 2024
interest within data domain S for which the distance measure d 8 S - Rzo has been defined. To any generated sample s E S, we associate the distance d(x, s); in this way, a global distribution that produces the sample can be said to induce the random value d(x,s) s) from local distribution of distances taken with respect to x. The CDF F(r) of the local distance distribution is simply the probability of the sample distance lying within a threshold r ー that is, F(r) 2 Pr[d(x, s) S r].
To characterize the local intrinsic dimensionality in the vicinity of location x, we consider the limit of ( as the distance r tends to O. Regardless of whether F satisfies the conditions of a CDF, we denote this limit by
LID
- lim LID,(r). --0+
Henceforth, when we refer to the local intrinsic dimensionality (LID) of a function F, or of a point whose induced distance distribution has F as its CDF, we will take 'LID' to mean the quantity LID
In general, L f.r is not necessarily integer. Unlike the manifold model of local data distributions where the dimensionality of the manifold is always an integer, and deviation from the manifold considered 'error' the LID model reflects the entire local distributional characteristics with- out distinguishing error. However, the estimation the LID at x often gives an indication of the dimension of the local manifold containing that would best fit the distribution.
ASYMPTOTIC FORM OF FISHER RAO METRIC FOR LID DISTRIBUTIONS
We now provide the necessary theoretical justifications for our LDReg regularizer which will be later developed in Section Intuitively, LDReg should regularize the LID of the local distribution of the training samples towards a higher value, determined as the LID some target distribution. This can help to avoid dimensional collapse by increasing the dimensionality of the representation space and producing representations that are more uniform in their local dimensional characteristics. To achieve this, we will need an asymptotic notion of distributional distance that applies to lower tail distributions. In this section, we introduce an asymptotic variant of the Fisher-Rae Rao distance that can be used identify the center (mean) of a collection of tail distributions.
4.1 FISHER-RAY DISTANCE METRIC
The Fisher-Raa distance is based on the embedding of the distributions on a Riemannian manifold, where it corresponds to the length of the geodesic along the manifold between the two distribu- tions. The metric is usually impossible to compute analytically, except for special cases (such as certain varieties of Gaussians). However, in the asymptotic limit analytically tractable for smooth growth functions.
will show that it is
Definition 2. Given a non-empty set X and a family of probability density functions o(x(0) pa- rameterized by O on X, the space M = {o(x(()]0 Rd) forms a Riemannian manifold. The Fisher-Rao Riemannian metric on M is function of 0 and induces geodesics, i.e., curves with minimum length on M. The Fisher-Rao distance between two models 02 and 0₂ is the arc-length of the geodesic that connects these two points.
In our context, we will focus on univariate lower tail distributions with a single parameter 0 corre- sponding the LID of the CDF. In this context, the Fisher-Rao distance will turn out to have an elegant analytical form. We will make use of the Fisher information Z, which is the variance of the gradient of the log-liielioood function (also known as the Fisher score). For distributions over [0, w) with single parameter 6, this is defined as
Io) - 10 (1 F (r
Lemma 1. Consider the family of tail distributions [0, w) parameterized by 8, whose CDFs are smooth growth functions of the form
Hujp(r) =
4

---

Published as a conference paper at ICLR 2024
The Fisher-Ke Rao distance deR between Hulo, and Hwlo; is
02
dyR (Hojo, Hm)
In
01
The Fisher information Tw for smooth growth functions of the form Hw is:
(0) / (20 30
(o (r)
0
The proof Lemma] 1 can be found in Appendix
4.2 ASYMPTOTIC FISHER-RAO METRIC
We now extend the notion of the Fisher-Rao metric distance distributions whose CDFs (condi- tioned to the lower tail [0, w/) have the more general form of growth function. The LID Represen- tation Theorem (Theorem in Appendix C.1 tells us that any such CDF (r) can be decomposed into the product of a canonical form H LIDE (r) with an auxiliary factor ( w):
Fu (r) HuILLD; (r) Ap(r
(") W
r
LID'
exp ( LID LIDr(t) dt
From Corollary 3.1 (Appendix C.1 the auxiliary factor (r w) tends to and tend to 0, provided that r stays within a constant factor of w. Asymptotically, then, For can be seen to tend to H. Huje the tail length tends to zero, for LID',. More precisely, for any constant 2
For (T)
lim
lim wreersew
AF(r, w) =1.
Hu|LID;
Thus, although the CDF F. does not in general admit a finite parameterization suitable for the direct definition of Fisher-Rai distance, asymptotically it tends to distribution that does: H LIDE.
Using Lemma I we define an asymptotic form of Fisher-RRe Rao distance between distance distributions.
Definition 3. Given two smooth- growth distance distributions with CDFs F and G, their asymptotic Rasher-Rao distance is given by
LID
dAFR(F,C) G) & lim drR(Hw) H HWILIng) =
w-+0+
LID
4.3 IMPLICATIONS
Assume that LID;, 2 and that Gw = is the one-dimensional uniform distri- bution over the interval [0, wi] (with LID' therefore equal to 1). We then have
Remark 1.1.
dAFR (Fw, (1 LID exp (dAFR (F,, U1,
In LID'
We can therefore interpret the local intrinsic dimensionality of distribution F' conditioned to the interval [0, w| (with LID; 2 1) as the exponential of the distance between distribution F and the uniform distribution in the limit as w + 0.
There is also a close relationship between our asymptotic Fisher-Rao distance metric and a mathe- matically special measure of relative difference.
Remark 1.2. One can interpret the quantity In (IID%/LID;) as a relative difference between LID and LID",. Furthermore, it the only measure of relative difference that both symmetric, additive, and normed Törnqyist et al. 1985
The asymptotic Fisher-Rao distance indicates that the absolute difference between the LID values of two distance distributions is not good measure of asymptotic dissimilarity. For example,
5

---

Published as a conference paper at ICLR 2024
pair of distributions with LID and LID are much less similar under the asymptotic Fisher-Raa Rao metric than pair of distributions with LID",> - = 20 and LID = 22.
We can also the asymptotic Fisher-Rao metric compute the 'centroid' Fréchet mear of set of distance distributions, as well as the associated Fréchet variance.
Definition Given a set of distance distribution CDFs {F), F,,,F,,. ...., FN.., }. the empirical Fréchet mean of F is defined as
N
arg
arg min Hwis
Σ
(dAFR(Hujo, Fi))²
N
2=1
The Fréchet variance of F then defined
of
N Σ (dAPR(uF,F'))
1 N
(ILIIp In LI
The Fréchet variance can be interpreted as the variance of the local intrinsic dimensionalitics of the distributions in FF taken in logarithmic scale.
The Fréchet mean has a well-known close connection to the geometric mean, when the distance is expressed difference of logarithmic values. For our setting, state this relationship in the following theorem, the proof of which can be found in Appendix C.3]
Theorem
Let p.y be the empirical Fréchet mean of a set of distance distribution CDFs
FF
{F), {FI,F2... F?...,,, .....,,N using the asymptotic Fisher-Rao metric dAFR- Then LID
exp DE the geometric mean of (LID'r...... LID FxN]
Corollary 2.1. Given the CDFs FFF,
the quantity N ΣE In is:
1. The averuge asymptotic Fisher-Rac distance of members of to the one-dimensional uni- form distribution (if for all i we have LIDp, 2 1).
2. The logarithun the local intrinsic dimension of the Fréchet mean of F.
3. The logarithm of the geometric mean of the local intrinsic dimensions of the members of F.
The proof of Assertion is in Appendix Assertions and follow from Theorem
It is natural to consider whether other measures of distributional divergence could be used in place of the asymptotic Fisher-Rao metric in the derivation of the Fréchet mean. Bailey et al. (2022) have shown several other divergences involving the LID of distance distributions - most notably that of the Kullback-Leible (KI.) divergence. We can in fact show that the asymptotic Fisher-Rao metric is preferable (in theory) to the asymptotic KI. distance, and the geometric mean is preferable to the arithmetic mean and harmonic mean when aggregating the LID values of distance distributions. For the details, we refer readers to Theorem 4 in Appendix C.5
In summary, Theorems and 4 (Appendix C.5) show that the asymptotic Fisher-Rao metric is prefer- able in measuring the distribution divergence of LIDs. These theorems provide theoretical justifica- tion for LDReg, which will be described the following section.
LID REGULARIZATION FOR SELF-SUPERVISED LEARNING
In this section, we formally introduce our proposed LDReg method. For an input image x and an encoder f(.), the representation of x can be obtained as z fxx.. Depending on the SSL method, a projector g(.), a predictor h(-). and a decoder t(.) can be used to obtain the embedding vector or the reconstructed image from z. LDReg is generic regularization
representations obtained by the encoder, and as such it can be applied to a variety of SSL methods (more details are in Appendix
We denote the objective function of an SSL method by LSLL
Also known the Karcher mean, the Riemannian barycenter and the Riemannian center of

---

Published as a conference paper at ICLR 2024
Local Dimensionality Regularization (LDReg). Following theorems derived in Section assume that the representational dimension is d, and that we are given the representation of a sample x, Suppose that the distance distribution induced by F x; is F (r)... To avoid dimensional collapse, we consider maximizing the distributional distance between F (r) and a uniform distance distribution 21, (r) (with L.D 1): for each sample, could regularize local representation that has a local intrinsic dimensionality much greater than I (and thus closer to the representational dimension > 1). We could regularize by maximizing the sum of squared asymptotic FR distances (L2-style regularization), or of absolute FR distances (L1 (tyle regularization).
In accordance Corollary 2.1 in Section 4.2 we apply L1 reggulaarizatin to minimize the negative log of the geometric mean of the ID values. Assuming that LIDjg, pr. is desired to be 2 1,
1 Σ lim (F max N
min
Σ In LID
(1)
where N is the batch size. Following Theorem 2 in Section 4.2 maximize the Fréchet variance under prior of p. = 1:
we apply Le-reullarization to
N
max N Σ lim (F)) (r,))) = min 1 N
Σ
LID
2 FL
)²
(2)
Our preference for the geometric mean over the arithmetic mean for L1- and Lg-reuularization is justified by Theorem in Appendix C.5 We refer readers to Appendix D for a discussion of other regularization formulations.
We use the Method of Moments (Amsaleg et al. 2018 as our estimator of LID, due to its simplicity. Since only the encoder is kept for downstream tasks, we estimate the LID values based the encoder representations (z = f(x)). Specifically, we calculate the pairwise Euclidean distance between the encoder representations of a batch of samples to estimate the LID'F, for each sample x; in the batch: LID's = Me
μ where denotes the number of nearest neighbors of z, wp is the distance to the k-th nearest neighbor, and μı is the average distance to all k nearest neighbors.
The overall optimization objective is defined as minimization of either of the following losses:
N
N
2 )²}
LL = L
SSL.
8 N LID' or LL2 = SSL. B
L12=
Σ N
II pL
(3)
where 3 is hyperparameter balancing the loss and regularization terms. More details of how ap- ply LDRegg on different SSL methods can be found in Appendix [E] the pseudocode is in Appendix
6 EXPERIMENTS
We evaluate the performance of LDReg in terms of representation quality, such as training a linear classifier top of frozen representations. We use SimCLR (Chen et al. (Garrido et al., 2023b), BOO (Grill et al. 2020), and MAE (He et al., 2022) as baselines. We perform our evaluation with Re (He et al. 20166 (for SimCLR, SimCLR- and BYOL) and ViT-B (Dosovitskiy et al. 2021) (for SimCLR and MAE) on ImageNet (Deng et al. 2009). As default, we batch size 2048, 100 epochs of pretraining for SimCLR, SimCLR- Tuned and YOL, and 200 epochs for MAE, and hyperparameters chosen in accordance with each baseline's recommended values. We evaluate transfer learning performance by performing linear evaluations on other datasets, including Food-101 (Bossard et al. 2014)) CIFAR (Krizhevsky & Hinton 2009). Birdsnap (Berg et al. (014)) Stanford Cars (Krause et al. [2013), and DTD (Cimpoi et | 2014]
2020a). SimCLR-Tuned
For finetuning, we use RCNN (Girshick et al., 2014) to evaluate on downstream tasks using the COCO dataset (Lin et | 2014). Detailed experimental setups are provided
Appendix F For LDReg regularization, we use k 64 as the default neighborhood size. For ResNet-50, we set 3 0.01 for SimCLR and SimCLR Tuned, 3 = 0.005 for BYOL. For ViT-B, we set 3 = 0.001 for SimCLR, and / 5 10 for MAE. Since LL and LL2 perform similarly (see Appendix G.1 here mainly report the results of CT.1- An experiment showing how local collapse triggers mode collapse is provided in Appendix G.2 while ablation study of hyperparameters is in Appendix G.3
7

---

Published as a conference paper at ICLR 2024
6.1 LDREG REGULARIZATION INCREASES LOCAL AND GLOBAL INTRINSIC DIMENSIONS
Intrinsic dimensionality has previously been used to understand deep neural networks in a supervised learning context (Ma et al. 20088) 8b Gong et al. 2019 Ansuini et al. 2019). For SSL, Figure Tigure2alshow 2asshows that the geometric mean of LID tends to increase over the course of training. For contrastive methods (SimCLR and BYOL), the mean of LID slightly decreases at the later training stages (dash lines). With LDReg, the mean of LID increases for all baseline methods and alleviates the decreasing trend the later stages (solid lines), most notably on BYO..
700
1111 Orignd
[ELI Orghnal
LETAS
300
Guorente
100
6. 37.3
200
300 110L
8
8
15
D10L
(a) LIDs in SSL (b) Color jitter strength
(c) Effective rank (d) Geometric mean of LID
Figure 2: (a) Geometric mean of LID values over training epochs. (b) Geometric mean of LID
values with varying color jitter strength in the augmentations for SimCLR. The linear evaluation result is reported in the legend. (a-b) LID is computed on the training set. (c-d) The effective rank and LID are computed for samples in the validation set. The solid and transparent bars represent the baseline method with and without LDReg regularization, respectively. MAE uses ViT-B as the encoder, and others use ResNet-50.
In Figure 2b we adjusted the color jitter strength of the SimCLR augmentation policy and observed that the mean LID of the representation space positively correlates with the strength. This indicates that stronger augmentations tend to trigger more variations of the image and thus lead to represen- tations of higher LID. This provides insights on why data augmentation is important for SSL Grill ct al. (2020): Von Kügelgen et al. (202I) and why can help avoid dimensional collapse (Wagner et al.
2022 Huang et al.) (203)).
The effective rank (Roy & Vetterli 2007) a metric evaluate dimensionality as a global property and can also be used as a metric for representation quality (Garrdoo et al., (023a). Figure 2c shows that BYOL less susceptible to dimensional collapse. SimCLR-Tune uses the same augmentation as BYO (which is stronger than that of SimCLR), yet still converges to a lower dimensional space as compared with BYOL. This indicates that for SimCLR and its variants, stronger augmentation is not sufficient to prevent global dimensional collapse. Generative method MAE is known to be prone to dimensional collapse (Zhang et al. 2022). Unsurprisingly, it has the lowest effective rank in Figure 2c Note that the extremely low effective rank of MAE is also related its low repre- sentation dimension which is 768 in VIT-B (other methods shown here used ResNet-50 which has a representation dimension of 2048).
We also analyze the geometric mean of the LID values in Figure 2d It shows that, compared to other methods, BYOL has much lower mean LID value. This implies that although BYOL. does not collapse globally, it converges to a much lower dimension locally. We refer readers to Appendix G.2 for an analysis of how local collapse (extremely low LID) could trigger a complete mode collapse, thereby degrading the representation quality. Finally, the use of our proposed LDReg regularization can effectively void dimensional collapse and produce both increased global and local dimensionalities (as shown in Figures 2c 2d with '+ LDReg').
6.2 EVALUATIONS
We evaluate the representation quality learned by different methods via linear evaluation, transfer learning, and fine-tuning downstream tasks. As shown in Table 1 LDReg consistently improves the linear evaluation performance for methods that are known to be susceptible to dimensional col- lapse, including cumple-contrastive method SimCLR and generative method MAE. It also improves BYOL, which is susceptible to local dimensional collapse as shown in Figure 2d Tables 2 and B further demonstrate that LDReg can also improve the performance of transfer learning, finetuning
8

---

Published as a conference paper at ICLR 2024
Table 1: The linear evaluation results (aceuracy (%)) of different methods with and without LDReg. The effective rank is calculated on the ImageNet validation set. The best results are boldfaced.
Geometric mean
Model
Epochs
Method
Regularization
Linear Evaluation
Effective Rank
of LID
64.3
470.2
18.8
SimCLR
IDReg
64.8
529.6
20.0
SimCTR
67.2
525.8
24.9
ResNet-50
100
(Tuned)
LDReg
67.5
561.7
26.1
67.6
583.8
15.9
BYOL
LDReg
68,5
594.0
22.3
SimCLR
72.9
283.7
13.3
LDReg
73.0
326.1
13.7
ViT-B
200
57.0
86.4
25.8
MΛE
LDReg
57.6
134.1
29.8
Table 2: The transfer learning results in terms of linear probing accuracy (%), using ResNet--0 as the encoder. The best results are boldfaced.
Method
Regularization
Batch Size
Epochs
ImageNet
Food-T01
CIFAR-TÚ
CIFAR- (00
Birdsnap
Cars
DTD
64.3
71.2
32.0
36.7
2048
100
SimCLR
LDReg
64.8
69.1
89.2
70.6
33.4
37.3
67.7
69.0
71.1
90.1
71.6
37.5
35.3
70.7
4096
1000
I.RRgg
69.8
73.3
91.8
75.1
38.7
41.6
70.8
Table 3: The performance of the pre-trained models (ResNet-50) on object detection and instance segmentation tasks, when fine-tuned on COCO. The bounding- (APib) and mask (APrik) average precision are reported with the best results are boldfaced.
Object Detection
Segmentation
Method
Regularization
Epochs
Batch Size
Aphb
AP
AP
Apmk
AP
Ap
35.24
55.05
37.88
31.30
51.70
32.82
SimCLR
LDReg
35.26
55.10
37.78
31.38
51.88
32.90
100
2048
36.30
55.64
38.82
32.17
52.53
34.30
BYOL
LDReg
36.82
56.47
39.62
32.47
53.15
34.60
36.48
56.22
39.28
32.12
52.70
34.02
SimCLR
1000
4096
LDReg
37.15
57.20
39.82
32.82
53.81
34.74
on object detection and segmentation datasets. Moreover, longer pretraining with LDReg can bring more significant performance improvement. These results indicate that using LDReg regularize local dimensionality can consistently improve the representation quality.
Table also indicates that the effective rank is good indicator of representation quality for the same type of SSL methods and the same model architecture. However, the correlation becomes less consistent when compared across different methods. For example, SimCLR LDReg and SimCLR- Tuned have similar effective ranks (~ 525), yet perform quite differently on ImageNet (with an accuracy difference of 2.4%). Nevertheless, applying our LDReg regularization can improve both types of SSL methods.
CONCLUSION
In this paper, we have highlighted that dimensional collapse in self-supervised learning (SSL) could occur locally, in the vicinity of any training point. Based on novel derivation of an asymptotic variant of the Fisher-Rao metric, we presented a local dimensionality regularization method LDReg alleviate dimensional collapse from both global and local perspectives. Our theoretical analysis implies that reporting and averaging intrinsic dimensionality (ID) should be done at logarithmic (rather than linear) scale, using the geometric mean (but not the arithmetic or harmonic mean). Fol- lowing these theoretical insights, LDReg regularizes the representation space of SSL to have nonuni- form local searest-neightee distance distributions, maximizing the logarithm of the geometric mean of the sample-win wise LIDs. We empirically demonstrated the effectiveness of LDReg in improving the representation quality and final performance of SSL. We believe LDReg can potentially applied generic regularization technique help other SSL methods.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
Details of all hyperparameters and experimental settings are given in Appendix F LDReg and LID estimation can be found in Appendix |
Pseudocode for summary the implementation is avail- able in !ppendix We provide source code for reproducing the experiments in this paper, which can be accessed here: https///gthub.ccm. http:/ghhbb..om////nmm////DDee We also discuss computational limitations of LID estimation in Appendix
ACKNOWLEDGMENTS
Xingjun Ma is in part supported by the National Key R&D Program of China (Grant No. 202 IZD01 12804) and the Science and Technology Commission of Shanghai Municipality (Grant No. 22511 I 106102). Sarah Erfani is in part supported by Australian Research Council (ARC) Dis- covery Early Career Researcher Award (DECRA) DE220100680 This research was supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative.
REFERENCES
Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stéphane Girard, Michael E Houle, Ken-ichi Kawarabayashi, and Michael Nett. satreme-valuuetheoree estimation of local intrinsic dimen- sionality. Data Mining and Knowledge Discovery, 2018.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In NeurlPS, 2019.
James Bailey, Michael Houle, and Xingjun Ma. Local intrinsic dimensionality, entropy and sta- tistical divergences. Entropy, 24(9):1220, 2022.
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transform- ers. In ICLR, 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variane-iivarinnc-covvaann regulariza- tion for self-supervised learning. In ICLR, 2022.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In CVPR, 2014.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-10 mood-101-minng 101-mining discriminative compo- nents with random forests. In ECCV, 2014.
Jörg Bruske and Gerald Sommer. Intrinsic dimensionality estimation with optimally topology pre- serving maps. TPAMI, 1998.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Kevin M Carter, Raviv Raich, and AO Hero. Learning on statistical manifolds for clustering and visualization. In Allerton Conference on Communication. Control, and Computing, 2007.
Claudio Ceruti, Simone Bassis, Alessandro Rozza, Gabriele Lombardi, Elena Casiraghi, and Paola
Campadelli. Danco: An intrinsic dimensionality estimator exploiting angle and norm concentra- tion. Pattern recognition, 47(8), 2014.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI. 2020a.
10

---

Published as a conference paper at ICLR 2024
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. NeurIPS, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In CVPR, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Romain Cosentino, Anirvan Sengupta, Salman Avestimehr, Mahdi Soltanolkotabi, Antonio Ortega, Ted Willke, and Mariano Tepper. Toward a geometrical understanding of self-supervised con- trastive learning. arXiv preprint arXiv:2205.06926, 2022.
Marco Del Giudice. Effective dimensionality: tutorial. Multivariate behavioral research, 56(3):
527-542, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. Collider: A robust training
framework for backdoor data. In ACCV, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy $ Liang. Improving self-supervised learning by characterizing idealized representations. NeurIPS, 2022.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In ICML, 2021.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In ICMI., 2023a.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In ICLR, 2023b.
Chongjian Ge, Jiangliu Wang, Zhan Tong, Shoufa Chen, Yibing Song, and Ping Luo. Soft neighbors are positive supporters in contrastive visual representation learning. In ICLR, 2023.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In CVPR, 2014.
Sixue Gong, Vishnu Naresh Boddeti, and Anil K Jain. On the intrinsic dimensionality of image representations. CVPR, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaro Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020.
Bobby He and Mete Ozay. Exploring the gap between collapsed & whitened features in self- supervised learning. In ICML, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
11

---

Published as a conference paper at ICLR 2024
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. NeurIPS, 1993.
Michael E Houle. Local intrinsic dimensionality I: an extreme-value-thherriii foundation for simi- larity applications. In SISAP, 2017a.
Michael E Houle. Local intrinsic dimensionality II: multivariate analysis and distributional support. In SISAP, 2017b.
Michael E Houle, Xiguo Ma, Michael Nett, and Vincent Oria. Dimensional testing for multi-step similarity search. In ICDM, 2012.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation self-supervised learning. In ICCV, 2021.
Weiran Huang, Mingyang Yi, Xuyang Zhao, and Zihao Jiang. Towards the generalization of con- trastive self-supervised learning. In ICLR, 2023.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive supersuue learning. In ICLR, 2022.
Jacob Devlin Ming-We Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine- grained cars. 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. NeurIPS, 2004.
Alexander C Li, Alexci A Efros, and Deepak Pathak. Understanding collapse in non-contrastive siamese representation learning. ECCV, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 (00.:00000.0383,, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Xingjun Ma. Bo Li, Yisen Wang, Sarah M. Erfani. Sudanthi Wijewickrema, Grant Schoenebeck, Michael E. Houle, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018a.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijew- ickrema, and James Bailey. Dimensionality- learning with noisy labels. In ICML, 2018b.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03777 2018.
Bo Pang, Yifan Zhang, Yaoyi Li. Jia Cai, and Cewu Lu. Unsupervised visual representation learning by synchronous momentum grouping. In ECCV, 2022.
Karl W Pettis, Thomas A Bailey, Anil K Jain, and Richard C Dubes. An intrinsic dimensionality estimator from near-neighhor information. TPAMI, 1979.
12

---

Published as a conference paper at ICLR 2024
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In ICLR, 2021.
Simone Romano, Oussama Chelly, Vinh Nguyen, James Bailey, and Michael E Houle. Measuring dependency via intrinsic dimensionality. In ICPR, 2016.
Olivier Roy and Martin Vetterli. The effective rank: measure of effective dimensionality. In European signal processing conference, 2007.
Stephen Taylor. Clustering financial return distributions using the fisher information metric. Entropy, 21(2):110, 2019.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML, 2021.
Leo Tömqqist, Pentti Vartia, and Yrjö Vartia. How should relative changes be measured? The American Statistician, 39(1):43-44) 1985.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. NeurIPS, 2021.
Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirmeister, Samuel Müller, and Frank Hutter. On the importance of hyperparameters and data augmentation for self-supervieed learning. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward ICML, 2022.
Zhaoqing Wang, Ziyu Chen, Yaqian Li, Yandong Guo, Jun Yu, Mingming Gong, and Tongliang Liu.
Mosaic representation learning for self-supervised visual pre-training. In ICLR, 2023.
Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Chun-Hsiao Yeh, Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In ECCV, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 03888, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In JCML, 2021.
Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurlPS, 2022.
Shaofeng Zhang. Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for negative-fiee free symmetric contrastive learning. In ICLR, 2021.
Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards unified theoretical understanding of non-contrastive learning via rank differential mechanism. ICLR, 2023.
13

---

Published as a conference paper at ICLR 2024
A ACHIEVING DESIRED LOCAL DIMENSIONALITY WITH LDREG
In this section, we provide details regarding how Figure is obtained. Following our theory, LDReg obtain representations that have desired local dimensionality. We use linear layer and randomly generated synthetic data points 2D following the uniform distribution. The linear layer transforms these points into representation space. Following Definition 3 might specify the desired local intrinsic dimensions as LID The dimension of the representations is LID To achieve the desired local dimensionality, minimize the following objective:
N
LID
min
I
N
LID
This objective corresponds to minimizing the asymptotic Fisher-Rao distance between the distribu- tion of the representations and a target distribution of fixed local dimensionality. In Figure 3 we plotted the results with target dimensions LID' equal [1.0, 1.2, 1.4, 1.6, 1.8, 2.0]. The results show that the estimated LID is very close to the desired values. This indicates that LDReg can also specific value. From this point of view, LDReg
be used to regularize the representations potentially be applied to other learning tasks as generic representation regularization technique.
mLID=1.1 GID=1.9
mUD==13 GID=1.9
mLID=1.5 GID=2.0
10
15
15
10
-05
46.
(a) LID = 1.0
(b) LIDE = 1.2
(c) LIDE = 1.4
mLID=1.7 GID=1.9
m.ID=1.9 GID=2.0
LID=2.1 GD==2.0
10
-05-
-100
(d) LID 1.6
(c) LID = 1.8
(f) LID 2.0
Figure 3: Each caption of the subfigures shows the desired local dimensionality and each title of the subfigures shows the estimated LID and global intrinsic dimensionality (GID). GID is estimated using the DanCO approach (Ceruti et al. 2014). mLID is the geometric mean of estimated sample LIDS..
In the context SSL, to avoid dimensional collapse, the resulting representation should span (fill) the entire space, shown in Figure 3f The L1- and L--regllarization terms used for SSL aim to maximize the asymptotic Fisher-Rao Rao distance between local distance distribution F(r) and a uniform distance distribution U1 (r) (which has LID equal 1, shown in Figure 3ap. In other words, the final representations should be 'further' from that of Figure Ba and 'closer' to that of Figure
14

---

Published as a conference paper at ICLR 2024
B EFFECTIVE RANK VERSUS LOCAL INTRINSIC DIMENSIONALITY
Zhuo et al. (2023) proposed the effective rank (Roy & Vetterli 2007) as a metric to evaluate the degree of (global) dimensional collapse. Given the feature correlation matrix, the effective rank cor- responds to the exponential of the entropy of the normalized eigenvalues. It is invariant to scaling and takes real values, not just integers. It has maximum value equal to the representation dimen- sion. Intuitively, the effective rank assesses the degree to which the data 'fills' the representation space, terms of covariance properties. One might also define local effective rank, which would assess the degree to which the representation space surrounding a particular anchor sample is filled.
In contrast to the effective rank, the local intrinsic dimension (LID) is a local measure. Roughly speaking, particular anchor sample, the LID assesses the growth rate of the distance distribution of nearest neighbors. is thus focused on distance properties (distances between samples), rather than covariance properties between features. Comparisons between intrinsic dimensionality and effective rank, emphasizing their differences, have been discussed in Del Giudice (2021).
C PROOFS
C.I BACKGROUND
Theorem 3 (LID) Representation Theorem (Houle, 2017a)). Let F :R + T. be real-valued function, and assume that L,ID,p exists. Let r and w be values for which r/w and F(r) F(w) are both positive. If F is non-zero and continuously differentiable everywhere in the interval [min(r, w), max(r, W. then
LID; LID; F(r) F(w)
(_) w
where AF(r,w) & exp LID, LID
AF(r,
t
whenever the integral exists.
Corollary 3.1 ((oule 2017a)) Let e and w be real constants such that c 2 1 and w > O. Let
F R R be real-valued function satisfying the conditions of Theorem over the interval
(0, cw... Then
lim W
AF(r, w) =1
w/cSrScu cw
F(0) = O, and F is non-decreasing and continuously differentiable over some interval [0, w] for >> then F is referred as smooth growth function. If F is CDF, we the notation F,, to refer to F conditioned over [0, w]: that is, (rr(r) = F(r)/F(w).
C.2 PROOF OF LEMMA 1
Proof. We leverage result from TTylor 2019), which shows that the Fisher-Rao metric be- tween two one-dimensional distributions with a single parameter can be expressed as d(l,, 02) = 1, u(f) d0), where u(0)² I(0), and I(0) is the Fisher information with respect the single parameter 0. In our context, 0 corresponds to the local intrinsic dimensionality. For the Fisher information for functions restricted to the form Haje. we will therefore use the quantity
I (0) = (0d
100 (
In H Fippltt) H Hrojo(r) dr.
30
15

---

Published as a conference paper at ICLR 2024
We derive an expression for the Fisher-Rao distance. From Taylor 2019), and noting that Fisher information is greater than or equal to zero,
dFR w H Hml0,)
VIw (0)
(0) do
82
w
a
0
(7)
8---1
))
r
8-1
In
dr
d0
00
82
w
r
0-1
In
dr
do
With the substitution v obtain
82
drR HoR Hwll2)
In
de
d#
0
80
av + 2(U In v v) + (u v 2 + 20)]
de
81
02
d0
In
01
0
C.3 PROOF OF THEOREM 2
Proof. We find the distribution Hwj that minimizes the expression
N
fG = arg min
Σ
(dAFR(HH Fi))
N
8
F=1
by taking the partial derivative with respect to 0, and solving for the value & 0g for which the partial derivative is zero.
N
In
0
00
LID
8=88
N
(In² + ln² LID
In 0 In LID
0
00
8=8a
2 ln 0G
In LID
pL
0
OG
lG
N Nlnfc
= Σ In LID
1 N
0G
exp
In LID Pi
In similar fashion, the second partial derivative be shown be strictly positive 0 0j. Since the original expression is continuous and non-negative over all 0 E (0), 0 global minimum.
16

---

Published as a conference paper at ICLR 2024
C.4
PROOF OF COLLARY 2.1
Proof. Assertion 1 follows from the fact that
N
LID
dAFR ,111 =
N
N
In LID p
(if Fi 2 1).
C.5 KULLBACK-LEIBLES DIVERGENCE AS DISTRIBUTIONAL DIVERGENCE
It is natural to consider whether other measures of distributional divergence could be used in place of the asymptotic Fisher-Rao metric, shown several other divergences and distances which, when conditioned to vanishing lower tail, tend to expressions involving the local intrinsic dimensionalities of distance distributions - most notably that of the Kullback-Leibler (KI.) divergence. Here, we define asymptotic distributional distance from the square root of the asymptotic KI. divergence considered in (Bailey et al.) 2022).
derivation of the Fréchet mean. [aileyy et al. (2022) have
Lemma (Bailey et al. 2022). Given two smooth- smooth-growth distance distributions with CDFs F and G, their asymptotic KI. distance is given by
LIDE LID
LIDE 1, LID F
dAKL. (F, G)
lim VDK (Pw, Gw) w-+0+
In
where
DKL(Fu, Gw) & f. (2(())
(()() dt
(t)
is the KL divergence from F to G when conditioned to the lower tail [0, w).
When aggregating the LID values of distance distributions, it is worth considering how well the arithmetic mean (equivalent the information dimension (Romano et al.) 2016)) and the harmonic mean might serve as alternatives to the geometric mean. Theorem shows that the arithmetic and harmonic means of distributional LIDs are obtained when the asymptotic Fisher-Rao metric is placed by the asymptotic KL distance, in the derivation of the Fréchet mean.
Theorem Given a set of distributions computing the Fréchet mean μr Hu:p-
(F), F2, FN), }, consider the metric used in
1. Using the asymptotic Fisher-Rao metric dAFR (Hwlo F+t) as in Definition givess equal the geometric mean of (LID" F1 ........ F....,, LID LIDpy'
2. Replacing dAFR (H o, F$) by the asymptotic KL distance (Huoo F+) gives equal to the arithmetic mean of (L. LID FN }.
3. Replacing (AFR((Hujo F²) ) by the (reverse) asymptotic KL distance dAKL( Hwjo) gives 0 equal to the harmonic mean of (LID",.... LID px).
Proof. Assertion 1 has been shown in Theorem
For Assertion 2, we find the value of & for which the following expression minimized:
0^ arg min
I N
Σ
(dAKe 2²
Using Lemma 2 and observing that LID'Tmpo O.
LID
LID px
(dAKI, F²))²
lim DKI,H F²) -0
In
17

---

Published as a conference paper at ICLR 2024
As in the proof of Theorem zero and solving for = Ag.
2 the minimization is accomplished by setting the partial derivative to
N
LIDP
LID p
In
=
00
N 1=1
0
0
8=HA
LID p
1
+
N
0
A
A
N
0^ = N
LIDE
For Assertion 3, we similarly find the value for which the following expression is minimized:
N
0H =
arg min
Σ
(dAKL(F', H e))²
Once again, take the partial derivative with respect to 6, zero, and solve for 0 0H:
0
0
In
00
LID
LID
0=00
1
0
N
LID
pri
0H
N
0μ
LID"
Note that 0g and 0H can be verified as minima by computing the second partial derivatives with respect 0.
The square root of the KL divergence is only 'weak approximation' of the Fisher-Rat Rao metric statistical manifolds, and this approximation is known to degrade as distributions diverge (Carter et al. 2007). Moreover, the KL divergence is also nonsymmetric. Theorem 4 therefore indicates that the asymptotic Fisher-Rao metric is preferable (in theory) to the asymptotic KL. distance, and the geometrie mean is preferable to the arithmetic mean and harmonic mean when aggregating the LID values of distance distributions.
D OTHER REGULARIZATION FORMULATIONS
We elaborate on our choices of regularization term.
Remark 4.1. The proposed regularization equivalent to maximizing the (log of the) geometric mean of the IDs of the samples.
Theorem provided arguments for why use of geometric mean is preferable to other means for the purpose of computing the Frechet a regularization corresponding to the arithmetic mean of the LIDs of the samples would be less preferable. i.e. max NV Σi LID Pt. Note that
of set of distributions. We can similarly consider why
I N
N Σ
max Σ N
LID
FN
max
N
exp
lim dFR(F (0-50
Observe that this arithmetic mean regularization, due to the exponential transformation, would apply high weighting to samples with very large distances from the uniform distribution (that is, samples with large ID). other words, such a regularization objective could be optimized by making the ID of small number of samples extremely large.
18

---

Published as a conference paper at ICLR 2024
E LDREG AND SSL METHODS
In this section, we provide more details on how to apply LDReg on different SSL methods. Since LDReg is applied to the representation obtained by the encoder, the varying combinations of projec- tor, predictor, decoder, and optimizing objective used by SSL methods does not directly affect how LDReg is applied. As a result, LDReg can be regarded as general regularization for SSL.
SimCLR. For input images x of batch size N, an encoder f()), and a projector g(-). the representa- tions are obtained by = fx),, and the embeddings are e (=)). Given batch of 2.V augmented inputs, the NT-Xent loss used by SimCLR (Chen et al. (020a) for positive pair of inputs (X,, xy) is:
exp(sim(e,,,e))7
C CNTXent In
Lmzj exp(sim((e em)/T)
2N
where T is the temperature, and the final loss is computed across all positive pairs.
For applying LDReg with LL1 term on SimCLR, we optimize the following objective:
2N
LL = NTXent - 2N
LLD
where the LID for each sample is estimated using the method of moments: LID":: Fg. where Pa the averaged distance to the nearest neighbors of 2;, and wn. is the distance the k-th nearest neighbor of 2;.
Me wo,
BYOL. BYOL (Grill et al. 2020) uses an additional predictor h(-) to obtain predictions p h((e,, momentum encoder (exponential moving average of the weights of the online model) where e' is obtained, and the loss function is the scaled cosine similarity between positive pairs, defined as:
e'
c CBYOL 2
Pi
with the final loss computed symmetrically across all positive pairs.
For applying LDReg with LL1 term on BYOL, optimize the following objective:
2N
L ====
= BYOL
Σ
InLID PL.
2N
The LID for each sample estimated in the same way as applying LDReg on SimCLR. For BYOL, use representations obtained by both the online and momentum branches as the reference set.
MAE. MAE (He et al. 2022) uses decoder that aims to reconstruct the input image. Unlike contrastive approach, it does not rely two different augmented views of the same image. MAE uses an encoder f(-) to obtain the representation x f(x') for a masked image x', and the decoder I(-) aims to reconstruct the original image by taking representation as input. Specifically, MAE optimizes the following objective:
C.
MAE (x((f(x()),,
For applying LDReg with CL1 term on MAE, we optimize the following objective:
Lr11 MAE
Σ In LID
N
The LID for each sample is estimated in the same way as applying LDReg on SimCLR.
F EXPERIMENTAL SETTINGS
For each baseline method, we follow their original settings, except in the case BYOL, where changed the parameter for the exponential moving average from 0.996 0.99, which performs
19

---

Published as a conference paper at ICLR 2024
better when the number of epochs is set to 100. Detailed hyperparameter settings can be found in Tables IT We use 100 epochs of pretraining and a batch size of 2048 as defaults. For LDReg regularization, we use k = 128 as the default neighborhood size. For ResNet-50, we use 8 0.01 for SimCLR and SimCLR Tuned, / = 0.005 for BYOL. For ViT-B, we use 3 = 0.001 for SimCLR, and [3 x 10-6 for MAE. We perform linear evaluations following existing works (Chen al. 2020a Grill et al. 2020; He et al., 2022 Garrido et al. 2202bb). For linear evaluations, we use batch size 4096 on ImageNet other settings are shown in Table
Following SimCLR (Chen et al. 2020a) and BYOL (rrill et al. 2020). we evaluate transfer learning performance by performing linear evaluations on other datasets, including Food- 1odd-111 (Bossard al., 2014). CIFAR (Krizhevsky & Hinton 2009). Birdsnap (Berg et al. [2044), Stanford Cars (Krause al. 2013), and DTD (Cimpoi et al., 22144. Due to computational constraints, for the transfer learning experiments, we did not perform full hyperparameter tuning for each model and dataset. The reproduced results of baseline methods are slightly lower than the reported results by/Chen al.] (020a): Grill et al. (2020). For all datasets, we use 30 epochs, weight decay to 0.0005, learning rate 0.01, batch size 256. and SGD with Nesterov momentum as optimizer. These settings are based on the VISSL library
We evaluate the finetuning performance with downstream tasks using the COCO dataset (train2017 and va12017) (Lin et al. 2014). We use ResNet-50 with RCNN-C4 (Girshick) et al. 2014) with batch size 16 and hase learning rate 0.02. We use the popular framework detectron2 and our configurations follow the MOCO-v1 official implementation exactly.
We conducted our experiments on Nvidia A100 GPUs with PyTorch implementation, with each experiment distributed across 4 GPUs. We used automatic mixed precision due its memory effi- ciency. The estimated runtime is 40 hours for pretraining and linear evaluations. As can be seen from the pseudocode in Appendix D] the additional computation mainly depends on the calculation and sorting of pairwise distances. As shown in Table 4 we observed no significant additional computa- tional costs for LDReg. Open source code available here: https::/github..com http:/gwwwb...ooo/aaamm/n/////// HanDunH/LDngegg
Table Wall-clock comparisons for pretraining with Distributed Data-Parallel training. Each exper- iment uses 4 GPUs distributed over different nodes. Results are based on 100 epochs of pretraining. Communication overheads could have a slight effect on the results.
Method
Wall-clock time
SimCLR
27.8 hours
SimCLR + LDReg
27.1 hours
Table 5: Pretraining setting for SimCLR (Chen et al. 2020a
Base learning rate
0.075
Learning rate scaling Learning rate decay Weight Decay Optimizer
0.075 x v Batch Size
Cosine Loshchilov & Hutter 2016) without restart
1,0 x 10
LARS YYou et al. 0.1
2017
Temperature for NTXent Projector
2048-128
Data Augmentations. For each baseline and LDReg version, the same augmentation in
existing works. Augmentation policy for SimCLR (Chen et al. (020a) is in Table [00 for SimCLR-
Tuned (Garrido al. 2023b) and B BYO (Grill et al. 2020) is Table 11
20

---

Published as a conference paper at ICLR 2024
Table 6: Pretraining setting for SimCLR-Tuned Garrido al. 2023b).
Base learning rate
0.5
Learning rate scaling
0.5 X Batch Size
256
Learning rate decay Weight Decay Optimizer Temperature for CNTNent
Cosine (oohchhlovv Hutter 2016) without restart
1.0 x 10
LARS (You et al.
2017
0.15
Projector
8192-8192-512
Table 7: Pretraining setting for BYOL Grrill al. 2020
Base learning rate
0.4 0.4 x BatchSize
Learning rate scaling
25
Learning rate decay
Cosine (Loshchilov & Hutter 2016
without restart
Weight Decay Optimizer
1.5 x 10
LARS You et 2017)
T for moving average Projector
0).99
4096-256
Predictor
4096-256
Table 8: Pretraining setting for MAE (He al. 2022
Base learning rate
1.5 x 10-1
Learning rate scaling
1.5 x 10 x BatchSize
255
Learning rate decay Weight Decay Optimizer
Cosine Loshchilov & Hutter 2016) without restart
0.05
AdamW (Loshchilov Hutter 2019
B1 for the optimizer
0.9
B2 for the optimizer
0.95
Table 9: Linear evaluation setting for ImageNet.
Epochs
90
Base learning rate
0.1
Learning rate scaling
0.1 x BatchSize
256
Minimal learning rate
1.0
1
Learning rate decay Weight Decay
Cosine (Loshchilov & Hutter 2016) without restart
0
Optimizer
LARS You al. 2017)
Table 10: Image augmentation policy for SimCLR
(Chen al.
2020a
Parameter
View
View 2
Random crop probability
1.0
1.0
Horizontal flip probability
0.5
0.5
Color jittering probability
0.8
0.8
Color jittering strength (s)
1.0
1.0
Brightness adjustment intensity
0.8 x
0.8
Contrast adjustment max intensity
0.8 x
0.8 x
Saturation adjustment max intensity
0.8 x
0.8 x
Hue adjustment max intensity
0.2
0.2
Grayscale probability
0.2
0.2
Gaussian blurring probability
0.5
0.5
21

---

Published as a conference paper at ICLR 2024
Table 11: Image augmentation policy for BYOL
(Grill al.
2020 and SimCLR-Tuned (Garrido
et al.
2023bb.
Parameter
View
View
Random crop probability
1.0
1.0
Horizontal flip probability
0.5
0.5
Color jittering probability
0.8
0.8
Brightness adjustment max intensity
0.4
0.4
Contrast adjustment max intensity
0.4
0.4
Saturation adjustment max intensity
0.2
0.2
Hue adjustment max intensity Grayscale probability
0.1
0.1
0.2
0.2
Gaussian blurring probability
1.0
0.1
Solarization probability.
0.0
0.2
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 COMPARING LOSS TERMS
can be observed from Table 12 that there are no significant differences between the regularization terms LL and L12 for improving the performance of SSL.
Table 12: Comparing the results of linear evaluations of regularization terms of LDReg. All models are trained on ImageNet for 100 epochs. The results are reported as linear probing accuracy (%).
Method
Regularization
k=64
k=128
LL
64.8
64.6
SimCLR
L
64.4
64.5
G.2 LOCAL COLLAPSE TRIGGERING COMPLETE COLLAPSE
In this section, we demonstrate that local collapse could trigger the worst-case mode collapse, where the output representation trivial vector. LDReg is a general regularization tool that regularizes representation to achieve target LID. One also use LDReg to achieve lower LID and, in the extreme case, local collapse. Specifically, we optimize the following objective function:
N
min
In LID
N
This objective regularizes the geometric mean of LID (of the representations) to 1. We use / to control the strength of the regularization term and denote Min LID.
Table 13:
Comparing the results linear evaluations of regularization terms of LDReg, MinLID and baseline. All models are trained ImageNet for 100 epochs using ResNet-50 encoder. The results are reported as linear probing accuracy (%) on ImageNet.
Method
Regularization
Linear Acc
Effective Rank Geometric mean LID
LDReg
0.01
64.8
529.6
20.0
64.3
470.2
18.8
Min LID
0.01
64.2
150.7
16.0
SimCLR
Min LID
0.1
63.1
15.0
3.8
Min LID
1.0
46.4
1.0
1.6
Min LID
10.0
Complete collapse
it can be observed that using Min LID with stronger (larger 8) will regularize the representation to have extremely low effective rank and eventually result in complete collapse, even if SimCLR explicitly uses negative pairs to prevent this. Figure 4 shows the visualizations
As shown in Table 13
22

---

Published as a conference paper at ICLR 2024
of learned representations with different regularizations. Additionally, the performance of linear evaluations degrades as dimensionality decreases. This result indicates that low LID undesirable for SSL.
(a) LDReg 3=0.01
(b) No regularization
(c) Min LID 3=0.1
(d) Min LID 3=1.0
Figure 4: t-SNE visualizations of the representations learned by different pretraining. Results are based on ResNet-50 with SimCLR with ImageNet validation set. Only the first 10 classes are lected for visualizations.
G.3 ABLATION STUDY
We examine the effects of varying 3 and for LDReg using SimCLR as the baseline. can be observed in Figure 5a and [5b that linear evaluation performance is relatively stable across different values of & and B. For effective rank, Figure shows that greater strength of LDReg regularization (larger (3) actually decreases the effective rank. This is surprising, since LID is local measure, and effective rank is a global measure. The differences between effective rank and LID are outlined in Appendix BB] Figuur(5d]shor shows that smaller value for k is more beneficial for LDReg. Smaller k is indeed more preferable, as it helps to preserve the locality assumptions upon which LID estimation depends.
510
66.00
06.50
05.15
Stm.1s
520
575
510
SInCLL
06.25
450
3.25
65.00
5200
06.15
820
66.00
06.50 30
350
66.2
66.25
0.00
66.00
310
0.55
(a) Linear Acc
(b) Linear Acc
(c) Effective rank
(d) Effective rank
Figure 5: (a-b) Linear evaluation results and (c-d) effective ranks with varying 3 and k. All models are trained on ImageNet for 100 epochs. The results are reported as linear probing accuracy (%) on ImageNet.
Table 14 shows that LDReg can consistently improve the baseline with different batch sizes. We reduce the h at the same rate N is reduced, e.g. 16 and 32 for batch sizes 512 and 1024, spectively, It can be observed that LDReg can consistently improve the baseline in different batch sizes.
Table 14: Comparing the results of linear evaluations with different batch sizes N. All models are trained on ImageNet for 100 epochs. The results reported linear probing accuracy (%).
Method
Regularization
N=512
N=1024
N=2048
63.6
64.2
64.3
SimCLR
LDReg
64.1
64.7
64.8
23

---

Published as a conference paper at ICLR 2024
G.4 ADDITIONAL LINEAR EVALUATION RESULTS
In this subsection, we further verify the effectiveness of LDReg with decorrelating feature methods such VICReg (Bardes et al., 2022) and Barlow Twins (Zbontar al.. 2021). We also evaluate with another SOTA sample-contrastive method MoCo (He et al. 2020). For applying LDReg on VICReg, we use
0.025 and = 64. For Barlow Twins, we use B 1.0 and k 64. For MoCo, we use B = 0.05 and k = 128. All other settings are kept the same as each baseline's originally reported hyperparameters. Results can found Table
Table 15: The linear evaluation results (accuracy (%)) of different methods with and without LDReg. The effective rank is calculated on the ImageNet validation set. The best results are boldfaced.
Geometric mean
Model
Epochs
Method
Regularization
Lineur Evaluation
Effective Rank
LfD
68.7
595.0
MoCo
LDReg
69.6
651.8
22.3
66.7
546.7
21.5
ResNet-50
100
VICReg
LDReg
66.9
602.4
22.5
63.3
602.1
20.8
Barlow Twins
LDReg
65.6
754.0
24.1
VICReg and Barlow Twins are SSL. methods rather than regularizers. For example, compared to MoCo, VICReg and Barlow Twins different projector architectures and loss functions. It's not fair to compare LDReg across different types of SSL. methods. For example, MoCo with I.Regg has linear evaluation of 69.6, yet MoCo alone can achieve 68.7, while VICReg under the same setting is only 66.7. 7. However, if we apply our regularizer to these methods, their performance can all be improved, shown in the table 15
To fairly compare the regularizers, we use the covariance (denoted as Cov) and variance (denoted as Var) alternative regularizers to replace LDReg and apply to BYOL. Note that they are pseudo- global dimension regularizers, as we cannot use the entire training set to calculate the covariance, calculated on mini-batch. We also performed hyperparameter search for Cov and Var (/3 for the strength of the regularization). We used the same regularization formula VICReg (Bardes) ct al. 2022) for Cov and Var ar as the following:
C(Z)
2zs 2) - =E);
Zis
(4)
Cov(Z) = c(Z)= = 1 E
(5)
if1
where is the representation dimensions.
1 Var(Z) = 7 max 7 S(2²,e) where Si.) is the standard deviation.
(6)
We apply the regularization on representations learned by the encoder, the same as in LDReg. The results can be found in the TT6 All results based 100 epoch pretraining with BYOL and ResNet-50. All settings are exactly the same as LDReg except for the regularization term.
Table 16: The linear evaluation results on comparing different regularization terms. The effective rank is calculated on the ImageNet validation set. The best results are boldfaced.
Method
Regularizer
3
Linear Evaluation
Effective Rank
Geometric mean of LID
None
67.6
583.8
15.9
Cov
0.01
67.6
583.5
15.9
Cov
0.1
67.5
593.5
15.8
BYOL
Cov+Var
0.01
67.8
539.2
15.5
Cov+Var
0.1
67.7
798.4
16.8
LDReg
0.005
68.5
594,0
22.3
Although the covariance and variance regularizer increase the global dimension, it does not improve the local dimension. It also has rather minor effect on the linear evaluation accuracy, whereas LDReg improves by almost 1 %. This further confirms the effectiveness of LDReg.
24

---

Published as a conference paper at ICLR 2024
H LIMITATIONS AND FUTURE WORK
The main theory of our work based on the local intrinsic dimensionality model and well-founded Fisher-Rao metric. Computation of LDReg and the Fisher-Rao metric all require the ability to accu- rately estimate the local intrinsic dimension. We have used the method of moments in this paper for LID estimation, due to its simplicity and attractiveness for incorporation within gradient descent framework. Other estimation methods could be used instead; however, all estimation methods for LID are known to degrade in performance as the dimensionality increases. Moreover, our estimation of LID based on nearest neighbor sets computed from within a minibatch. This choice made due to feasibility of computation, but entails reduction in accuracy compared to using nearest neighbors computed from the whole dataset. In future work, one might explore other estimation methods and tradeoffs between estimation accuracy and computation time.
Based on existing works, LDReg assumes that higher dimensionality is desirable for SSL. L.Regg relies hyperparameter to adjust the strength of the regularization term. The theory devel- oped in this work allows LDReg to achieve any desired dimensionality. However, the optimal di- mensionalities for SSL dependent on the dataset and loss function. Knowledge of the optimal dimensionality (if it could be determined) can be integrated into LDReg for best performance.
I IMPLEMENTATION DETAILS
For implementation with PyTorch, Garrido et al. (2023b) have discussed popular open-source imple- mentations of SimCLR (compatible with DDP using ga gather) that slightly inaccurate gradients. The implementation in VICReg (Bardes et al. 2022) codebase F is correct and should be used. We find that this slightly affects the performance when reproducing SimCLR's results. This also affects LDReg for estimating LIDs with DDP. For all of our experiments, we use the same implementation as Garrido et al. (2023b); Bardes al. 20222.
Estimating LID needs to compute the pairwise distance, in Py PyTorch, the cdist function by default uses matrix multiplication approach. For Nvidia Ampere newer GPUs, TensorFloat- tensor cores should be disabled due to precision loss in the matrix multiplication. This preci- sion loss can significantly affect the LID estimations.
J PSEUDOCODE
Algorithm 1: Method of moments for LID estimation using pytorch pseudocode.
data: representations reference: reference points
k: the number of nearest neighbours
def lid_mom_est (data, reference, k)
= torch. cdist (data, reference, p=2) # Pairwise distance a, idx = torch. sort (r, dim=1)
m = torch. (a[:, 1:k], dim=1) # mu
lids m (a(:, k] m) # al;, k] is the w_k
return lids
25

---

Published as a conference paper at ICLR 2024
Algorithm 2: LDReg using pytorch pseudocode.
f: representations
k: the number of nearest neighbours
beta: the hyperparameter $\betaş
loss: SSL loss (such NTXent)
reg_type: "11" or "12" (Ll or L2 loss)
lids = lid_mom_ mom__stt (data=f, reference=f. detach (), k=k)
if reg_type
"11":
lid_reg = - torch. abs (torch. log (lids)) )
elif reg_type
"12":
lid_reg = torch. sqrt (torch.squall square (torch. log (lids))) )
total_loss loss beta lid_reg
total_loss = total_loss. mean (dim=0)
26


Reviews:
[{"review_id": "8UWzdIxyhs", "reviewer": "Reviewer_oLx3", "summary": "The authors propose to increase the accuracy of deep models by regularizing the local intrinsic dimensionality (LID) of features. They observe that without the proposed regularization, the dimensionality collapses locally, even if the global dimension remains constant.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "(1) The paper is well presented and well supported by theory. \n(2) The method seems intuitive.", "weaknesses": "(1) It's hard to reference an equation without equation numbers. In your overall optimization objective, it is unclear how $LID^*_F$ is calculated. I see in section 3 that this quantity is the result of a limit of some other quantity $LID_F$, which then depends on differentiating a function. It would be hard for me to implement this loss function just from reading this paper. How are all of these values calculated?\n\n(2) Follow up from (1): It is unclear how you estimate a \"local\" dimensionality from a mini-batch of samples. The mini-batch is sampled over the entire dataset, so none of them lie in the local neighborhood of other samples within the batch. I don't think this is addressed.\n\n(3) How does the proposed LIDs regularizer compare to regularizing global dimensionality by decorrelating features. The authors site a few works at the end of the first paragraph of the intro, but do not compare against them. For instance [Barlow twins] and [VICE-Reg] are popular ways of doing this. \n\n(4) Follow up from (3): Regularizing global dimensionality makes sense to me, but regularizing local dimensionality does not. e.g. looking at Figure 1(c), the authors show a few examples of local dimensionality collapse compared with one example of high local dimensionality. To me, it looks like the examples with low local dimensionality (i.e. low LID, but constant GID) exhibit more structure and therefore could be better features. How do you expect to learn good features when you regularize the distribution to be a random gaussian both globally and locally?\n\n(5) I don't see any results showing a correlation between test accuracy and LID. I would expect to see a plot showing that when LID increases, so does accuracy. Perhaps I missed it, if so please point me there. Furthermore, I would expect GID and LID to be correlated; so I would expect some result that shows improving LID with constant GID improves accuracy. Perhaps the LID and GID scores could be added to Table 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DLFzE0JPP3", "reviewer": "Reviewer_LKEn", "summary": "[I made a mistake in the form.] I found out that I have accidentally checked the \"First Time Reviewer\" question, but in fact, I'm not. It seems that I cannot undo it now, so I'm instead writing it here.\n\nThis paper proposes a new regularization technique for self-supervised learning. There are many recent works on preserving the internal diversity of self-supervised representation, and one approach is to preserve the effective dimensionality of the representation. Unlike existing work that tries to preserve global dimensionality, this paper argues that local dimensionality might still collapse. To resolve this issue, the paper proposes the local intrinsic dimensionality (LID). Using the Fisher-Rao metric on LID, the proposed method adds a regularization term that makes the distribution of the representation far from the most simplistic distribution (with dimensionality one). Experiments show that the proposed method indeed improves the performance.", "questions": "Please see the above weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation behind the proposed method (local dimensionality collapse) makes sense, and it is demonstrated empirically.\n\n- The proposed regularization is elaborately designed based on well-founded theories (LID representation, Fisher-Rao metric).\n\n- Experiments show that the local dimensionality indeed improves. The performance improvement itself is somewhat incremental except for a few cases. However, considering the recent self-supervised learning works, this is understandable.", "weaknesses": "- There is no comparison to other similar methods for improving self-supervised learning. In particular, there is no comparison to ones with global dimensionality regularization (which has close relationships with the proposed method), even though they were mentioned in the paper.\n\n- The proposed method requires the calculation of distance distributions during training. This can be somewhat heavy, depending on the actual settings. Ideally, self-supervised learning is meant to be performed on large-scale data, so this point can be even more burdensome. How big is the actual computational burden? I said that the incremental performance improvement is understandable, but it might not be really beneficial if the computational burden is quite high, considering there are also other recent alternatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3GoV0xeoB0", "reviewer": "Reviewer_VyQK", "summary": "This paper presents a novel variant of the Fisher-Rao metric, and then proposes a local dimensionality regularization (LDReg) to alleviate the dimension collapse in self-supervised learning (SSL). Moreover, it is verified that geometric mean is suitable to manipulate the intrinsic dimension, and therein LDReg maximizes the logarithm of the geometric mean of the sample-wise LID to have nonuniform local nearest neighbor distance distributions. Empirical evaluations are provided to demonstrate the effectiveness of the proposed approach in some degree.", "questions": "- Since that the results in Table 1 and 3, the improvements are relatively weak. Are these improvements significant? Or it is from other minor modification or some random fluctuation?  \n\n- What about the performance of the LID based regularizer with respect to the parameter $k$? Is the parameter $k$ affected by the batch size (or the density of the samples)? \n\n- Since that $LID_F*$ defined as the limit of $LID_F(r)$ when $r\\rightarrow 0$. How about the approximation quality of such a local intrisic dimension estimator? In another words, is it a good estimator of the local intrisinc dimension? Is any theoretical or empirical evidence to show the quality of the so-defined LID to estimate the local intrinsic dimension of the data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ It is interesting to take the perspective of the local intrinsic dimension to remedy the dimensionality collapse. \n+ Applying the geometric mean of the local intrinsic dimensionality (LID) as a regularizer for SSL is novel.", "weaknesses": "- The empirical evaluations look insignificant. In Table 1 and 3, the performance improvements are relatively weak. The reviewer is curious that these improvements are significant or not?  \n\n- The reviewer is curious about the sensitivity to the locality parameter. What about the performance of the LID based regularizer with respect to the locality parameter $k$? Is it sensitive to the parameter $k$? There is another parameter $N$. Is it the number of samples of the dataset, or is it the batch size? If the later case, what about the sensitivity of the performance with respect to the batch size (or the density of the samples)? \n\n\n- $F$ is assumed to be differentialble at $r$. However, if $F(r)$ is defined as the prob of the sample distance lying within a threshold $r$, how can we define the differential at $r$ for $F(r)$? Moreover, $LID_F*$ defined as the limit of $LID_F(r)$ when $r\\rightarrow 0$, is refered as LID. How about the approximation quality of such a local intrisic dimension estimator? In another words, is it a good estimator of the local intrisinc dimension? Is any theoretical or empirical evidence to show the quality of the so-defined LID to estimate the local intrinsic dimension of the data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjMYO3seLe", "reviewer": "Reviewer_85fu", "summary": "This work introduces a novel approach called LDReg for addressing SSL problems. LDReg incorporates the concept of local intrinsic dimensionality (LID), as measured in (Houle, 2017a), and utilizes it as a penalty term in various SSL models.\nThe experimental results presented in this paper demonstrate the effectiveness of LDReg in improving SSL performance. The conducted experiments on benchmark datasets highlight the positive impact of incorporating LID as a regularization term.", "questions": "1.This paper appears to have been organized in a hurry, as evidenced by the missing parenthesis in Definition 1. On page 19, there are two versions of $w_k$ ($w^k$).\n\n2.On page 4, it is unclear what ‘Pr’ denotes. Please provide an explanation or definition for this term.\n\n3.In addition to the $\\bf{LID}$ defined using $\\bf{IntrDim}$ in (Houle, 2017a) and the theoretical analysis using Fisher-Rao metric to provide justification, please clearly highlight the additional contributions of this work.\n\n4.While the experimental results presented on different datasets show improvements compared to other SSL models without LDReg, the authors did not analyze the reasons behind the under-performance results. Additionally, it is unclear how dimensional collapse was observed in the experiment, as the results alone cannot directly reflect this improvement.\n\n5.Technically, the regularization used is $\\mu_k / (\\mu_k - w_k)$, which prefers that nearest neighbors are not in the same surface of the sphere ball of the sample. Other works that aim to improve dimensional collapse are referred to as compared methods in the experiments.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dimensional collapse problem in SSL has received significant attention in recent years, with several works addressing this issue. This work offers a fresh perspective on the problem and introduces a generalized regularization approach that can be applied to existing SSL models.\n\n2. An advantage of this work is its convenient implementation, allowing for improved performance in downstream tasks without the need to modify the underlying architecture. By providing an additional viewpoint on the dimension collapse problem, this work contributes to the ongoing efforts in tackling this challenge.", "weaknesses": "1.While the performance of SSL models with LDReg is impressive, the novelty of this work appears to be limited as the proposed LID is inspired by or defined similarly to IntrDim proposed in (Houle, 2017a). Without proper attribution to (Houle, 2017a), the novelty of this work becomes questionable.\n\n2.It is recommended that the authors verify whether Houle et al. have indeed proposed Theorem 1, which defines LID. If not, it would not be reasonable to cite (Houle, 2017a) in Theorem 1 and define LID based on it. On the other hand, if Houle et al. did define LID in (Houle, 2017a), the novelty of this work would be further discounted.\n\n3.Since the only contribution of this work seems to be the definition of LID, which is directly borrowed from (Houle, 2017a), it is suggested to provide a more detailed explanation in Section 3 to clarify the relationship between the proposed LID and the work of (Houle, 2017a).\n\n4.The experimental results lack in-depth analysis. Several significant observations, as mentioned in the Q4, have been overlooked. It is recommended to address these observations and provide a thorough analysis of the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)