LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
Authors: Vimal Thilak
Chen Huang
Omid Saremi
Laurent Dinh
Hanlin Goh
Preetum Nakkiran
Joshua M. Susskind
Etai Littwin
Venue: ICLR 2024 spotlight (2024)
Keywords: Self Supervised Learning
Joint Embedding Architectures
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A SOTA method for predicting linear probing performance in joint embedding architectures

Abstract: Joint embedding (JE) architectures have emerged as a promising avenue for ac-
quiring transferable data representations. A key obstacle to using JE methods,
however, is the inherent challenge of evaluating learned representations without
access to a downstream task, and an annotated dataset. Without efficient and re-
liable evaluation, it is difficult to iterate on architectural and training choices for
JE methods. In this paper, we introduce LiDAR (Linear Discriminant Analysis
Rank), a metric designed to measure the quality of representations within JE archi-
tectures. Our metric addresses several shortcomings of recent approaches based
on feature covariance rank by discriminating between informative and uninforma-
tive features. In essence, LiDAR quantifies the rank of the Linear Discriminant
Analysis (LDA) matrix associated with the surrogate SSL task—a measure that
intuitively captures the information content as it pertains to solving the SSL task.
We empirically demonstrate that LiDAR significantly surpasses naive rank based
approaches in its predictive power of optimal hyperparameters. Our proposed cri-
terion presents a more robust and intuitive means of assessing the quality of rep-
resentations within JE architectures, which we hope facilitates broader adoption
of these powerful techniques in various domains.

Published as a conference paper at ICLR 2024
LiDAR: SENSING LINEAR PROBING PERFORMANCE IN JOINT EMBEDDING SSL ARCHITECTURES
Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Josh Susskind, Etai Littwin Apple Correspondence to: fvthilak,
com
ABSTRACT
Joint embedding (JE) architectures have emerged as a promising avenue for ac- quiring transferable data representations. A key obstacle to using JE methods, however, is the inherent challenge of evaluating learned representations without access downstream task, and annotated dataset. Without efficient and re- liable evaluation, it is difficult to iterate on architectural and training choices for JE methods. In this paper, we introduce LiDAR (Linear Discriminant Analy- sis Rank), metric designed to measure the quality of representations within JE architectures. Our metric addresses several shortcomings of recent approaches based on feature covariance rank by discriminating between informative and un- informative features. In essence, LiDAR quantifies the rank of Linear Discrim- inant Analysis (LDA) matrix associated with the surrogate SSL task-a measure that intuitively captures the information content as pertains to solving the SSL task. We empirically demonstrate that LiDAR significantly surpasses naive rank based approaches in its predictive power of optimal hyperparameters. Our pro- posed criterion presents ity of representations within JE architectures, which we hope facilitates broader adoption of these powerful techniques in various domains.
robust and intuitive means of assessing the qual-
"Measure what is measurable, and make measurable what is not so.."
Galileo Galilei
INTRODUCTION
In recent years, self-supervised learning (SSL) has emerged as pivotal technique for pretraining representations on extensive, unlabeled datasets, thus effectively alleviating the often burdensome
labeling requirements (Chen et al.) 2020 Assran et al., 2023 Chen & He 2020 Caron al. 102T
Bardes et al., 2021]] Caron et al. 2018 2020: Baevski et al. 2022 Zbontar et al.) 2021] He et al,
2021 HaoChen et al. 2021| Grill et al. 2020). However, despite the remarkable progress made in SSL, assessing the quality of representations acquired through this method remains an open problem. This challenge is further amplified when considering Joint Embedding
(JE) architectures, which notoriously suffer from uninterpretable loss curves that offer little clue to assess the progression of training. The conventional and widely adopted approach involves evaluating model performance by employing these representations in downstream tasks. Nevertheless, when the objective is to learn versatile representations applicable across diverse domains, this method demands substantial investments of time and resources to comprehensively evaluate performance across multitude of tasks and datasets. Alternatively, limiting assessments to only a few datasets and tasks undermines confidence in the evaluation process.
As a result, fundamental question arises: Can we effectively evaluate the quality of learned repre- sentations without relying explicit downstream task evaluations? Addressing this inquiry neces- sitates the precise definition of "quality" the context of representations. Subsequently, we must explore statistical estimators capable of quantifying this quality without depending downstream evaluations. Beyond its theoretical implications, such metric holds significant practical value, as aids in model selection and the development of novel SSL algorithms.

---

Published as a conference paper at ICLR 2024
Recent works (Garrido et al. 2022 Agrawal al. 2022 Lu et al. 2023 Kalibhat al. 2022) have introduced metrics that measure the quality of representations that correlate well with downstream tasks. Among these methods, the metrics introduced in (Garrido et al. 2022 Agrawal et al. 2022) are relevant to our work as they rely on statistics of the empirical feature covriance matrix. Notably, the recently introduced RankMe (Garrido et al. 2022) method shows that the rank of the feature covariance correlates surprisingly well with downstream performance, demonstrating SOTA results in label free hyperparameter selection. In this paper, we build upon RankMe by proposing simple modification to the feature covariance matrix on which the rank is calculated, which significantly and consistently improves its predictive power of downstream performance. Our method, which we refer to as LiDAR, is motivated by simple observation: the covariance spectrum can be easily and arbitrarily manipulated, and this manipulation is often encouraged by implicit or explicit regulariza- tions in the SSI. objectives. Consequently, representation with full rank covariance matrix may result from spurious factors rather than representing rich semantic features. As a result, even though methods such as RankMe demonstrate impressive results, we show that non-trivial additional gains can be had rather effortlessly. We summarize our contributions below:
1. We introduce LiDAR, method for assessing representation quality of JE SSL objectives, and theoretically motivate it. LiDAR uses the SSL objective in its definition, providing more intuitive and robust metric for assessing SSL representations.
2. We conduct a comprehensive set of experiments spanning multiple JE architectures, both transformer and resnet based. These include contrastive and regularized methods, as well as newer models such as I-JEPA and data2vec, which leverage masking techniques. We demonstrate that the LiDAR metric correlates significantly and consistently higher with downstream linear probing performance than RankMe as measured by both the Spearman Rank and Kendall rank correlation coefficient or Kendall's
3. We show that LiDAR demonstrates consistently strong performance in hyperparameter se- lection, outperforming RankMe. We further demonstrate this capability in randomized tri- als, where multiple hyperparameters are varied simultaneously.
2 PRELIMINARIES
2.1 SELF SUPERVISED LEARNING
The main goal of self supervised pre-training is to learn a general purpose data representation f(x) that is transferable to large variety of downstream tasks. Informally, the degree of transferability of f can be measured by how easy it is to learn downstream task given f. In practice, a set of downstream tasks (T
used to assess the transferability of f by training additional readout networks Øy on top of f. That is, a successful pre-training scheme involves finding / such that there exists "simple" functions dy. such that Φ solves T₃ reasonable degree, and Φ5 are "easy" to learn In the popular linear probing protocol, linear readout functions dj is used to assess the quality of f. For the remainder of the paper we restrict our investigations to linear probing.
2.2 JOINT EMBEDDING ARCHITECTURES
common paradigm in the current practice of SSL the multi-view setting. In its simple form, pair of inputs are processed by a JE architecture, where each input is encoded separately by (possibly shared) encoder. The SSL objective is then tasked with learning a representation such that "compatible" pairs that share semantic information are easily predictive of the each other, perhaps with the help of a latent variable. Various methods in the literature fall under this general category, which mostly differ by how compatible inputs are sampled, and how they avoid representational collapse. We can formally define a JE architecture with an encoder F [L((x)
X' Rdd and a projector function (f) : Rdd - R.P We adopt the terminology in (Garrido et al., 2022). and refer to the output of the encoder f as a representation, and the output of the composed encoder
We use the terms "simple" and "casy learn" here loosely as requiring few samples "The encoder function need not shared between views, however for simplicity and without loss of gen- crality we assume it is Some SSL methods do not projector function, in which we can assume it is the identity function
2

---

Published as a conference paper at ICLR 2024
and projector tb o as an embedding. A common theme among most JE SSL objectives is that they seek to learn embedding functions that produce similar embeddings for compatible views, which are typically generated by various forms of data augmentation. However, more recent JE architecture somewhat depart from this paradigm by using input masking, and introducing latent variables in the projector function v.. For example, in I-JEPA (Assran al., 2023) and data2vec [Baevski et al.] 2022), I and z represent partially masked and unmasked inputs respectively, and ((f;z 2) is tasked with predicting the parts of the representation f() given spatial locations provided by 2. For simplicity and without loss of generality, we remove the explicit notation 2 in the definition of v. Note that, absent any input reconstruction loss, JE architectures potentially promote more abstract representations by filtering fine grained pixel level information. However, without an input reconstruction term, the loss used is often uninformative of the actual metric of interest, which is the zero or few shot transfer of the learned representation to downstream tasks. Compounding this challenge, extended training durations can significantly deteriorate representation quality, even when employing a set of hyperparameters with proven performance, illustrated in Figure In such cases, finding an appropriate early stopping criterion is critical.
Dimensional Collapse JE architectures are prone to various forms of representation dimension collapse, which can manifest in different flavors (Ziyin al. 2022 Jing et al. 2021; Hua et al.) 2021). In contrastive methods, dimensional collapse occurs when learning results in an excessively low dimensional representations in way that hinders downstream performance. Regularized meth- ods, when insufficiently regularized either implicitly or explicitly, can theoretically suffer complete collapse, where the learned representation trivializes to constant. However, beyond such rough categorization of the training process to partial or complete collapse, recent literature alludes to a more nuanced observation that, in some settings, the feature covariance eigenspectrum can be used to accurately gauge the quality of the learned representation as it pertains to downstream perfor- mance.
2.2.1 EMBEDDINGS AND FRATURE COVARIANCE EIGENSPECTRUM
Recent developments in the evaluation of JE-SSL representations have introduced methods that leverage information derived from the spectrum of either the representations or the embeddings (RankMe, Garrido et al. (2022) or from the spectrum of the covariance matrix, a-Req, Agrawal et al. 2022 ). Garrido et al. (2022) who propose a new measure named RankMe and demonstrate that this measure, when applied across different hyperparameter configurations, exhibits a strong correlation with downstream performance across various settings and tasks. In order to define RankMe, we start with dataset {ar drawn iid from some input distribution D, and an embedding function e(x) E Rd that produces an embedding matrix Z with dimensions (n, d) whose singular values are denoted as σ = 01, 02 02,..mmin(aadd These singular values reveal insights into the nature of the mapping function f(x). RankMe (Garrido et al. 2022) introduced a soft measure of effective rank expressed as exp(- E/= pi log Pt). where P.-
| c. and represents small constant.
In a related work, a-Req (Agrawal et al. 2022 uses insights from infinite dimensional spaces to argue that the eigenspectrum of representations, i.e., eigenspectrum of the feature covariance should decay an ideal rate of >; ~ O(i-1) where >; is the 7th eigenvalue of the covariance matrix. However, it's essential to note that any condition imposed on the eigenspectrum alone may not be sufficient to ascertain the representational power of the mapping function f. This limitation arises from the fact that a simple linear mapping, such as f(x) = Wz with weight matrix W, can manipulate the covariance matrix's spectrum arbitrarily. Additionally, even random mapping could exhibit a high effective rank without necessarily translating into significant downstream performance improvements. Notably, (Li et al. 2022) showed that the loss value and covariance spectrum can be used in tandem to predict performance. However, their method requires training a classifier on offline data, making it highly inefficient as an unsupervised method. LiDAR follows a similar intuition with more efficient and effective formulation: A representation with high effective rank, when coupled with a low objective loss, points to successful training. To balance the two terms we leverage discriminative method from classical statistics, repurposed to SSL settings.
3

---

Published as a conference paper at ICLR 2024
Probe accuracy
60
400
LIDAR (student)
200
40
tgt=[0.15, 0.2]
RankMe
300
tgt=[0.15, 0.2]
150
bgt=[0.15, 0.2]
tgt=(0.125,
200
tgt-10. 0.2]
100
tgt-(0:125,, 0.2]
20
Lgl=|0.2,0.255
tgt=[0.2, 0.25]
tgt=[0.2, 0.25]
100
50
tgt=[0.2. 0.3]
tgt=[0.2, 0.3]
tgt=[0.2, 0.3]
200
400
600
0
200
400
600
200
400
600
Epoch
Epoch
Epoch
Figure 1: ViT-Base architecture trained with I-JEPA (Assran al. 2023) by varying the target mask scale hyperparametr. We observe that (1) RankMe correlates poorly with downstream performance for most models, with the exception of the worst performing model, and (2) LiDAR correlates highly with downstream performance for all models.
3 METHOD
Our method is based on linear discriminant analysis (Bishop) 2007). classical label-aware dimen- sionality reduction and classification algorithm which we adopt the multi-view setting common in the SSL. literature. Absent downstream task and labels, we use clean samples as surrogate classes. For an input distribution DD consider a generic JE architecture with an embedding function e. For any clean input x, let D2 denote conditional distribution over all transformed inputs given x. Define μlz EznD, [e())] and μ = EE [ua]. we define the generalized covariance Eiidar by:
Mn (e) = (e) (e)
(1)
where (e) EE-D (w6- 1 (M) 44
Ei (w)()) EL-DEEDD [e(()) - p2) (e()) m )
7|+
5Tp where is small positive constant, and T, is the identity matrix of dimension Let = ... Ap be the eigenvalues of Enidars then LiDAR is defined by applying the smooth rank measure introduced in (Roy & Vetterli] 2007) on Eqidar
LiDAR(e) = exp( -
1-Ene -- log P: A |||p
+
(2)
where c is a small positive constant. practice, we use unbiased estimates of Ew, Eg using hyper- parameters where is the numbers of surrogate classes (clean samples), and q is the number of transformed samples per class. Note that, as in classical LDA, the eigenvalues A A, measure variance along discriminative directions. Hence, LiDAR(e) takes into account the SSL objective that being optimized, and ignores directions of variability e that are useless in solving the SSL task. We postulate the existence of such directions, JE techniques implicitly measures to preserve representations from collapsing. simpler terms, some variations in e may necessarily arise from high-level semantic features driven by the SSL objective, but rather from arbitrary attributes influenced by regularization. This phenomenon can clearly be seen in I-JEPA, where the rank of a representation inflates early on in training, only rapidly diminish, with peak downstream performance occurring far from the point of maximal rank (see Figure I This makes any standard data covariance rank based measures extremely limited in their ability to spot the point of optimal performance in training run. In contrast, the LiDAR method initiates at much lower rank and steadily ascends, more closely aligning with downstream linear probing performance, as depicted in Figure [
explicitly incorporate
We delve deeper into the theoretical rationale behind LiDAR in Appendix Section & where we prove a proposition which illustrates the effect of uninformative information on the LiDAR metric.
4 EXPERIMENTAL DETAILS
LiDAR is intended to serve as a proxy metric to compare the quality of representation as they relate downstream performance. This task is challenging due to the inherent uncertainty of the nature
Data augmentations, otherwise data points which treated positive samples
4

---

Published as a conference paper at ICLR 2024
of the downstream task. In this paper, focus downstream classification tasks, employing the widely adopted linear probing protocol. A robust metric in this sense is one that consistently replicates the downstream performance ranking observed with ground truth oracle across set of pre-trained models. As in existing solutions, we only compare models from the same class, each one pre-trained using different of predefined hyperparameters. It is important emphasize that linear correlation between the metric and the oracle is not implied, hence we resort to established statistical tests which assess the strength and direction of the monotonic relationship between two variables. In other words, we seek to measure how well the relationship between two variables, the metric and the oracle, can be described using a monotonic function. We note that such evaluations go beyond simply picking the best model according to the metric of interest out of a set of models, due to possible outliers in the set.
Spearman's 's rank correlation coefficient The Spearman correlation coefficient (Spearman 1987 1961] is essentially the Pearson correlation coefficient computed on the ranked values of two vari- ables. While Pearson's correlation assesses linear relationships, Spearman's 's correlation evaluates monotonic relationships, which can be linear or non-lineat nature. In cases where there are no duplicate data values, perfect Spearman correlation of either +1 or indicates that each of the vari- ables exhibits a flawless monotonic rel relationship with the other, forming perfect monotone function. Given two sequences of real numbers X I1, ..., In and Y y/...., Yn and their corresponding ranking R(X), R(Y), the Spearman correlation coefficient is given by:
Ts = E, R(y:))
(3)
nnn² - 1)
Kendall's Tau rank correlation coefficient The Kendall's Tau correlation coefficient (Kendall, 1938b a) another popular alternative the to Spearman rank correlation, and is known to be superior when the sample size is small and has many tied ranks. The Kendall's Tau coefficient uses concor- dant and discordant pairs its measure. For indices j > i, the pairs ( {x,, } E X,, (1A, Vi} E are said to be concordant if the sort order of both pairs agree, otherwise they are said to be discordant. Let C and D denote the number of concordant and discordant pairs in X, Y then the Kendall's 7 is given by CC DD
Top ranking model In similar spirit (Garrido et al. 2022), as additional evaluation we report the top ranking model given a metric of interest, and its downstream performance, and compare it to the optimal model according to the oracle. This protocol can be seen noisy estimate of Kendall's Tau rank correlation coefficient.
4.1 MODELS, HYPERPARAMETER AND DATA
We use 5 different multiview JE SSL methods, spanning contrastive and regularized methods, as well as ResNet and transformer based. We use 1-JEPA (Assran et al.] 2023) and data2vec (Baevski ctt al. (022) as representative of more recent masking based approaches that are not reliant on do- main specific data augmentation. We use SimCLR (Chen et al.] 2020) as representative of con- trastive methods, while we use DINO (Caron et al. 2021) as an example of self-distillation method and VICReg (Bardes et al. 2021) representatives of regularized methods. Note that I-JEPA and data2vec use transformer based encoders by design. We use a vision transformer (ViT) (Dosovit- skiy et al. 2021) based encoder for DINO as well, and ResNet-50 (He al. (016) based encoders for SimCLR and VICReg. We vary different hyperparameters per method. The varied hyperpa- rameters range from optimization related ones such as learning rate, and weight decay, architecture specific hyperparameters such as softmax temperature, and data augmentation and masking based hyperparameters. We stress that some SSL. objectives such as I-JEPA and data2vee, which rely on specific forms of input masking, are extremely sensitive to the the masking hyperparameters, hence providing an important testbed for LiDAR. We highlight the drawback of conducting grid search over single hyperparameter, due to the fact that all the remaining frozen ones are typically highly optimized to the task, offering a somewhat contrived testbed. Hence, in addition to a standard grid search, random search over all hyperparameters. This is done by uniformly sampling each hyperparameter from a fixed range, providing a better cover for the space. We use the Imagenet-Iy 1k dataset (Russakovsky al. 2015) for all experiments. We use the train split as the source dataset
5

---

Published as a conference paper at ICLR 2024
Hyperparameter
RankMe
RankMe (aug.)
LiDAR (Student)
RankMe (aug.) (Teacher)
LiDAR
(Student)
(Teacher)
Learning rate
0.6835
0.7829
0.8443
0.6734
0.7506
Weight decay
0.5040
0.7034
0.7331
0.4792
0.5278
Target mask scale
0.2867
0.6786
0.7937
0.2956
0.2927
Context mask scale
0.4246
0.7867
0.8482
0.3929
0,5556
Overall
0.5830
0.7513
0.8159
0.5713
0.6828
Table 1: I-JEPA: Kendall's 7 coefficient between effective ranks of RankMe, RankMe (aug.) and LiDAR and linear probe accuracy. Hyperparameters are varied via grid search.
RankMe
RankMe (aug.)
LiDAR (Student)
RankMe (aug.)
LiDAR (Teacher)
(Student)
(Teacher)
Random search
0.8314
0.8989
0.8434
0.8487
0,8616
Table 2: I-JEPA: Kendall's coefficient between effective ranks of RankMe, RankMe (aug.) and LiDAR and linear probe accuracy. Hyperparameters are varied via random sampling.
for pretraining and linear probing, and use the test split the target dataset. For each pretrained checkpoint, we train linear probe on the train split, which we denote as the oracle, and record its test performance the test split.
4.2 IMPLEMENTATION CONSIDERATIONS
The implementation of LiDAR entails computing empirical approximations Ew, Ex, which differs from one SSL method to another due to the inherent differences in the way imput pairs are sampled. As general rule, for each SSL method we use its own input transformations without alterations. In asymmetrical architectures, we compare both branches, denoted as the "student" and the "teacher" for evaluation. In transformer based architectures (as employed by design in I-JEPA, data2vec) we pool the final embedding//rppreeenatio to produce vectors. For methods such as I-JEPA and data2vec, computing the RankMe score on the embeddings is not a straightforward task. This is due to the fact that the projector function's 's task in both does not serve as representation expander, rather serves as a conditional predictor, predicting masked representations. conditioned on the spacial locations of the masked patches. For these methods, evaluate RankMe on the student teacher encoder instead. For SimCLR and VICReg, we copy the implementation details from (Garrido) ee al., 2022). The hyperparameters used train and evaluate the models listed in Appendix 9] Finally, in our analysis within the context of RankMe, we have noticed that utilizing the same data augmentations for both training and feature covariance matrix computation consistently leads to improved performance. In our experiments, we provide empirical results for both "vanilla" and augmented RankMe as baselines.
Metric
LR
WD
Target mask scale
Context mask scale
Overall
ImageNet Oracle
70,5800
70.5800
70.5800
70,5800
70,5800
RankMe
56.7580
60.2040
60.2040
59,8960
56.7580
RankMe (aug.) (Student)
67.8680
67.8680
67.8680
67.8680
67.8680
RankMe (aug.) (Teacher)
56.7580
52.6720
50.4200
59,8960
56.7580
LiDAR (Student)
67.8680
67.8680
67.8680
67.8680
67.8680
LiDAR (Teacher)
65.1080
64.1920
65.5820
67.5420
60.0080
Table 3: I-JEPA: Linear probe accuracy recovered by RankMe and LiDAR on ImageNet-I dataset. Hyperparameters set via grid search.

---

Published as a conference paper at ICLR 2024
ImageNet Oracle
RankMe
RankMe (aug.)
RankMe (aug.)
LiDAR (Student)
LiDAR (Teacher)
(Student)
(Teacher)
68.9840
53.3700
53.3700
53.3700
61.1760
66.8880
Table 4: I-JEPA: Linear probe accuracy recovered by RankMe and LiDAR on ImageNet-IK dataset. Hyperparameters set via random sampling.
4.3 COMPUTE CONSIDERATION
While calculating the LiDAR score is straightforward and computationally efficient procedure, it does introduce an additional computational cost when compared to standard covariance estimation. This additional cost arises from the need perform matrix inversion in E. It can be, in principle, time-consuming when dealing with high-dimensional embeddings. In our experiments, we have observed that this computational overhead is generally inconsequential for all tested models. The cost tends to be dominated by the computational demand of the feature extraction's 's forward process. Nonetheless, we note the following trivial bound the rank of Endarr
Rank (Eidar) S min (Rank(Ew)). Rank((
Since Rank(E;) is bounded by the number of surrogate classes used n (number of "clean" samples used to generate augmented samples), simple dimensionality reduction can be used when p >> n to reduce the rank of E., before inversion, without any loss in performance. We find in our experiments that n 1 g is sufficient to saturate performance of I-JEPA, data2vec and DINO methods. Our evaluations with SimCLR requires = 5k; 10 while we use 10k, q = 10 for VICReg as these methods provide much longer features compared to I-JEPA, data2vec and DINO.
RESULTS
As strikingly evident from our experimental results, the LiDAR metric correlates surprisingly well with downstream performance, as measured by linear probing. In the vast majority of experiments, see significant improvement over RankMe in terms of the Kendall's 7 and Spearman's 's rank correlation to the oracle, and an improved performance in hyperparameter selection. Due to space constraints, we defer the reader to the Appendix for comprehensive view of experimental de- tails, supplementary empirical results, and additional figures. In the main text, we have included representative selection to provide an overview.
Tables and 3 present a comprehensive analysis of the results obtained for the I-JEPA. The evalua- tion involves comparison between LiDAR, assessed on both the teacher and student branches, and RankMe, with and without data augmentation, alongside the oracle reference. We observe general trend where LiDAR applied on the student branch correlates much higher with the oracle than the teacher branch. Overall we observe significant outperformance of LiDAR over RankMe applied on clean and augmented inputs for all hyperparameters tested. A noteworthy observation from Table 3 is that RankMe, when coupled with data augmentation compute the feature covariance matrix, can match LiDAR's performance for the best-selected model, albeit trailing significantly when data augmentation is not employed. Table 2 and Table 4 present results from randomized trials where hyperparameters are randomly sampled within predefined ranges. In these trials, a consistent trend of LiDAR outperforming both versions of RankMe is evident, as can also be seen in Figure Tables 5 and 6 extend our analysis to the data2vec model, and the findings parallel those observed for I- JEPA. Notably, LiDAR consistently selects more performant hyperparameters than both variants of RankMe, narrowly missing oracle-level performance.
Table summarizes our results for VICReg (Bardes al. 2021). This table additionally reports metrics calculated at different checkpoints throughout the training process. Across all checkpoints (epochs 20, 40, 60, 80, 100), we consistently observe significantly higher correlations with the oracle performance. It's worth highlighting that, with VICReg, LiDAR achieves optimal results when applied to the representation rather than the embedding, as umbedding-basee evaluations result in dramatic performance degradation, phenomenon aligning with the mon-monoto relationship between rank and performance reported by (Garrido et al. (002). This illustrates that high rank is a necessary but not sufficient condition for high performance.
7

---

Published as a conference paper at ICLR 2024
Hyperparameter
RankMe
RankMe (aug.)
LiDAR (Student)
Rank (aug-) (Teacher)
LiDAR
(Student)
(Teacher)
Learning rate
0.2410
0.3419
0.5077
0.3172 0.2195
0,4716
Mask ratio
0.2683
0.2381
0.4657
0.4170
Overall
0.2238
0.2799
0.5531
0.2626
0.5227
Table 5: data2vec: Kendall's data2vec between effective ranks of RankMe, RankMe (aug.) and LiDAR and linear probe accuracy.
Metric
LR
Mask ratio
Overall
ImageNet Oracle
60.3920
55.9780
60.3920
RankMe
48.6040
48.6980
48.6980
RankMe (aug.) (Student)
48.6040
51.4500
51.4500
LiDAR (Student)
59.3720
52.7460
59.3720
RankMe (aug.) (Teacher)
48.6040
51.4500
51,4500
LiDAR (Teacher) 59.3720
52.7460
59.3720
Table 6: data2vee: Linear probe accuracy recovered by RankMe and LiDAR with data2vec ImageNet-11 K dataset.
Further validation of LiDAR's efficacy in hyperparameter selection is provided in Appendix Table TO where the method achieves results that are within a fraction percentage point from oracle-level performance across all considered hyperparameters. Table g shifts the focus to SimCLR. widely used contrastive sulf-supe learning method. The evaluation centers on the final checkpoint ob- tained after 100 training epochs. Table 8b reveals that LiDAA consistently demonstrates the highest correlation among the three metrics under scrutiny. Moreover, Table 8a confirms that LiDAR con- sistently selects the most optimal hyperparameters among the three methods considered. Lastly also study the behavior of LiDAR. Table [6 lists the results for a ViT-Small trained with DINO on ImageNet- 1K dataset. We observe that LiDAR evaluated with both the teacher and student branches show stronger correlation than RankMe.
LIMITATIONS
While we observe LiDAR significantly improves upon RankMe in most experiments, it is important to emphasize its drawbacks. Notably, we have observed instances where the LiDAR metric exhibits negative correlation with probe accuracy, particularly pronounced in scenarios like VICReg when dealing with higher dimensional embeddings. This phenomenon underscores the intrinsic complex- ity of the relationship between rank, however it is measured, and downstream task performance.
Epoch
RankMe
RankMe (aug. dataset)
LiDAR
Epoch
RankMe
RankMe (aug. dataset)
LiDAR
20
0.6081
0.6466
0.6957
20
0.4476
0.4718
0.5323
40
0,4315
0.5949
0.8699
40
0.2984
0,4597
0.7097
60
0.4065
0.5381
0.8809
60
0.2702
0.3750
0.7218
80
0.3904
0.5905
0.9032
80
0,2823
0,4435
0.7823
100
0.3174
0.5209
0.9161
100
0.2056
0.3790
0.8105
(a) VICReg: Spearman Rank coefficient
(b) VICReg: Kendall's 7 coefficient
Table 7: VICReg: Correlation between effective rank estimated by RankMe and LiDAR and probe accuracy evolution during training. Each row corresponds to a checkpoint collected at epoch speci- fied in the table.
8

---

Published as a conference paper at ICLR 2024
70
60
Probe accuracy
50
Probe accuracy
50
LiDAR (student)
LÍDAR (teacher)
RankMe
30
30
RankMe (aug.) (student)
RankMe (aug.) (teacher)
20
20
LIDAR (teacher)
RankMe
RankMe (aug.) (teacher)
0
100
200
300
400
100
200
300
400
Effective rank
Effective rank
(a) Linear probe accuracy RankMe, teacher- LiDAR and hasse nugmenteddRankMe.. RankMe.
basher--bsed based
(b) Linear probe accuracy for LiDAR, augmented- RankMe variants and vanilla RankMe.
Figure 2: I-JEPA: ViT-Base architecture trained Imagenet-1K 1K with 20 sets of hyperparameters
drawn uniformly at random. Each point represents a checkpoint while the o marker represents the model selected by each metric. We observe that (1) LiDAR outperforms RankMe (see Table (2. (2)
sugmented-RankMe our variant of RankMe, shows the highest correlation (see Table 2 and (3)
remarkably LiDAR shows the best hyperparameter selection performance by recovering the highest downstream performance.
Metric
LR
WD
Tomp.
Overall
Correlation
RankMe
RankMe
LiDAR
Imagenot Oracle
58.2370
56.6740
57,4710
59.1420
(aug. dataset)
RankMe
55,8950
1490
56,0390
RankMe laug. dataset)
57,2010
55.8170
56.3170
57.8260
Kendall's 0,4982
0.5761
0.8167
LiDAR
(a) Linear prohe accuracy.
(b) Kendall's 7 correlation coefficient.
Table SimCLR: (a) Linear probe accuracy recovered by RankMe, augmenteddRaantee RankMe and LiDAR on ImageNet- K. dataset at the end of training and (b)Kendall's т correlation coefficient between effective rank estimated by RankMe and LiDAR and probe accuracy per hyperparameter.
These factors not necessarily causally linked; high rank does guarantee superior per- formance. is essential to acknowledge the computational overhead associated with calculating the LiDAR metric, which often requires high-dimensional matrix inversion. This increases the method's overall computational cost as shown in Appendix [2 using I-JEPA as an example. Consequently, the feasibility of incorporating LiDAR as loss signal for pretraining should be carefully considered, as it could be prohibitively expensive to naively evaluate it at each iteration during the training process. Due to the sheer volume of experiments and necessary compute, we focus solely on linear probing in this work. We expect LiDAR's impressive performance to carry over to nonlinear probing proto- cols as well. Lastly, it's worth noting that our approach is contingent on the sampling strategy for positive pairs and ã which varies among methods, making cross-method comparisons challenge.
7 CONCLUSION
We introduced LiDAR, novel approach to evaluate ceff-supervised learning models. It builds upon foundation laid by the RankMe metric. Through a series of experiments, we have demonstrated LiDAR's superiority over RankMe, representative covariance rank-based approach. It enables curate and -ndeeenndennn self-supervised learning assessment of learned representations, Our method is powerful tool for practitioners seeking efficiently optimize their models in data-scarce environments. As for future work, investigating LiDAR as a possible loss term is a particularly com- pelling one. This would require different formulation make the method more SGD-compatible. We anticipate LiDAR's integration into standard SSL toolkits that has the potential to reshape model evaluation and have large impact on the future advancements of the field.

---

Published as a conference paper at ICLR 2024
REFERENCES
Kumar Krishma Agrawal, Amab Kumar Mondal, Ama Ghosh, and Blake Aaron Richards. aaSrrr Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. httttttss///pmeeee
net forum?
id 1d-119X4vt2G12
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski. Pascal Vincent, Michael G. Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint- embedding predictive architecture. 2023 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 15629, 2023. URL
SGc//ooppuID:22559911
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, 2022. URL https:://appppp htt.pp...mmmttttmmmmmmeemmmmam.e semant icscholas org/
[CorpusDD:246552))
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianc-iivvarane--cvvraaa regular-
ization for self-supervised learning. ArXiv, abs/2105.04906, 2021. URL https://api.
semant iescanntcscclarrrr org/ Corpus 22435755222
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statisties). Springer, edition, 2007. ISBN 0387310738. URL http:://ww.
amazon
dp/ B22/2//803FFFaccccaooooottttrron
3Dws 3hgp4//eiiiacccocmmmmm//////mo2e 6595533 2bcreaLiveAAS1193003377
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- mining discriminative com- ponents with random forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, ing. ISBN 978-3-319-10599-
pp. 146-461, Cham, 2014. Springer International Publish-
Deng Cai, Xiaofei He, and Jiawei Han. Training linear discriminant analysis in linear time. In 2008 IEEE 24th International Conference on Data Engineering, 2008
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference on Computer Vision, 2018. URL
httpps://app www.etiinnccci.oe.... 1 ar. www.oor//ooooooooo Corpus ID: 49865868)
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. ArXiv, abs/2006.098 09882, 2020. URL www.httpss.. / semant waw.mmanntsccooll.c. wtw.:o/aaw..or//oorrrra::202.3.0 :t:
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 JEEE/CVF
International Conference on Computer Vision (ICCV), pp. 9630-9640 2021. URL https:
//api. www.eeaatttccccoollr.orrooorrroo htt.hattooi..iiggggooaeoeooorrrr 233444273
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020. URL https:
7/api. www.mmtnniischo..ooom..... wtt.oloa.....oooo.......... :121006663
Xinlei Chen and Kaiming He.
Exploring simple siamese representation learning. 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 15745-15753, 2020. URI.
https://api www.nticsccchcooooooeo ar. wwt.or////ooooo corpss/ TD: ID:
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL hetps: //openreviee ew. net. wete/forrmme d=Y cwF.TTNTTTyyyycc
10

---

Published as a conference paper at ICLR 2024
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann LeCun. Rankme As- sessing the downstream performance of pretrained supervieed representations by their rank. ArXiv, abs/2210.02885, 2022.
https: o.t.r.aooto/oooooo.a..a.......o.
CorpusTD:2527.355
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour, 2018.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan 2021.
Misra. Vissl. https::/gpitiiii httpa//cccccccccccooeeeeereraeea
Bean-Bastien Grill, Florian Strub, Florent Altch'e Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supe learning. ArXiv, abs/2006.07733 2020. URL https: //api semant cacholaccaccooocccccc wwwpecchoaar.rg//oorooeee wttp/oorpssooooooooooooooooe
Jeff 7. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In Neural Information Processing Sys- tems, 2021. URL htttsttps:// /ap.. semant i cacholar wwwph/hholrrrrr//ooooooooooo www.or///orruusic : 235367882
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 15979-15988, 2021. URL https://api.
semanticscholass www.aarrrrrrooooeeoeeoeeeo wttp/oooooii::00333332
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, 2019.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang. Sucheng Ren, and Hang Zhao. On feature decorrelation in self-supervised learning. 202/ IEEF/CVF International Conference Com- puter Vision (ICCV), pp. 9578-9588, 2021. htt.rtttss//pppppp..taatttttcooo
CorpusID: 23330111:234316999
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. ArXiv, abs/2110.09348, 2021. URL httpss://app.
semanticschon wttp://ccc........oooooooosaaaa.
org :3:::11::23999666
Neha Kalibhat, Kanika Narang, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Measuring
self-supervised representation quality for downstream classification using discriminative features. arXiv preprint arXiv:2203.0/88 2022.
M. G. Kendall. new measure rank correlation. Biometrika, 30(1/2):81-95 1938a. ISSN 00063444. htLhhttt://www
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81 1938b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA. USA, 2015.
Alex Krizhevsky and Geoff Hinton. Learning multiple layers features from tiny images. 2009.
Alexander C. Li, Alexei A. Efros, and Deepak Pathak. Understanding collapse in non-contrastive
siamese representation learning. In European Conference on Computer Vision, 2022. URL
https://api semanticscho ar.
http:/cooooo...ooo/ooousoo/o ottg//oorpuu11::2255555

---

Published as a conference paper at ICLR 2024
Yuchen Lu, Zhen Liu, Aristide Baratin, Romain Laroche, Aaron Courville, and Alessandro Sor- doni. Using representation expressiveness and learnability to evaluate self-supervised learn- ing methods. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL
forum? httpss://cpenreview... net 2 www.eeeeteriddeeeeeeeeeee
(oorum?id=BBdrrpnRnH
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. 2007
15th European Signal Processing Conference. pp. 606 610, 2007. URL https://api,
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 1 115(3))111-255 2015. doi: 10.1007/s 1200007011122--1118111161
C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 100(3/4)444-47// 1987. ISSN 00029556. URL http: /wwww//www/////////////////// jsttrsstoooooo org/
stable/1422689
Charles Spearman. The proof and measurement of association between two things. American Journal of Psychology, 1961.
Carsen Stringer, Marius Pachitariu, Nicholas A. Steinmetz, Matteo Carandini, and Kenneth D. Har- ris. High-dimensional geometry of population responses in visual cortex. Nature, 571:361 365, 2019. URL/tttps://app CtL www.seemannncchooaaaac.cccccc
htt.://aaa...rrr/ooousaai::::::0 69116
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 14485-3492 2010. doi: 10.1 10.1109/CVPR.2010. 5539970.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks, 2017.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. ArXiv, abs/2103.03230. 2021. URL https:
7/api. semant cempntsemant... ccacholar..oco wtthhollrrrr//oopuueeeeeeeee : 232110471,
Liu Ziyin, Ekdeep Singh Lubana, Masakuni Ueda, and Hidenori Tanaka. What shapes the loss landscape of sule-supe learning? ArXiv, abs/2210.00633, 2022. URL httpps:/app.
semant: i cscho ar. Trg:/oorpssD::22255555
12

---

Published as a conference paper at ICLR 2024
8 THEORETICAL MOTIVATION
In this section we illustrate a heuristic as to why we might expect LiDAR to outperform previous methods relying on the covariance spectrum. As an exemplary JE SSL method, we consider the VICreg objective, which is comprised of three terms:
L
Ahiny
+ ph + V-rcov
(5)
Invariance
Regularization
where A, H, V/ are hyperparameters, and Linv, Lvar, Loov are the invariance, variance and covariance terms computed over a minibatch. In this objective, the regularization term which is comprised of the variance and covariance terms explicitly encourages the embedding function's covariance to be of high rank, while the invariance term insures that compatible pairs are mapped to similar embeddings. We highlight that a low regularization loss is achievable by random embeddings, which might be high rank, but devoid of any utility as for downstream tasks. A measure of representation quality which is based on covariance rank alone would therefore, theoretically, fail to capture this failure case, as it only pertains to the regularization term. Indeed, balancing the hyperparameters X, μ, 1 is necessary to prevent superficially inflating the covariance rank. On the other hand, LiDAR is invariant to information that is not used to discriminate between surrogate classes in the SSL objective. To make this point concrete, consider a (centered) embedding function e(x) Rd - R", a random independent noise vector E R' such that E[u] 0, consider the (random) embedding functions ē(x) = [e(x)] , _ TRd - RR++ Naturally, when measuring the downstream performance of e, é, we expect that e should not be worse than é. (in practice performance is measure on the representation f. however for the sake of a clean argument we will ignore this technicality). In turn, this should, ideally, translate to LiDAR(ē) S. LiDAR(e). LiDAR (unlike covariance spectrum based approaches) indeed captures this property, as illustrated in the following proposition:
Eluu
Σn E Rrxr, and
Proposition 1. Let D denote a distribution over inputs x E Rd, let Dr denote a conditional distri- bution of transformed inputs given Let X = > Ap be the eigenvalues of Exidar(e) Assume that
Dl < 1 - ex][-1] and set constants e, 6 such that e < 1
DAlla and 8 < (exp[- e)llAlla. I|All
Tilli
Then, it holds that:
|
rō
||ll1 |
LiDAR(ē) < LiDAR(e) exp - 2p
log(
(6)
||ll
|All1
Proof. From the independence of the noise vector μ4, it is easy to see that:
Σx(E) (c)
0
(x) = (e)
0 )p)
Eiittr (c)
0
0
Eiidar =
5I,
(7)
Then, we have that:
A
A ||al1 + r8 +c) + R
LiDAR(E) exp
||il|2 rg + +c)log
log
(8)
where:
|A||2 + ro +6) +c)logg log
8 [A rr + +o
R = exx
(9)
rō
5
S exp
||al/ log(
(10)
||Al|2
where we used the fact that 1 function| |z log( in the interval x E (0, exp[ 11). Note that:
< expl (-1] 1] by assumption to deduce the maximum of the
||ll2 + rr + +c) log ||||/1 +r8 r8
A
A + +c]
exp
(11)
exp
|illa 5r +c) log
A
d ||lli +6)
(12)
A
IAll +
||l|1 + 5r +&) log
(13)
| + E
13

---

Published as a conference paper at ICLR 2024
Since Vi, A: |all + c < 1 by assumption, we have:
exp
|||| + ör +5) log
||al1i +0)
A
(14)
==1
A
+() +0)100 log
||ll1 +c)] LiDAR(())
A
I exp
(15)
1
i=1
hence we can write:
X ||lla +r +2) log
||Al|2 + rō + +c)
A
exp
(16)
LiDAR(e) exp
- (1(1+e)
+e) min log
TAll+tro + 6)
1All22 + )
(17)
d
||al + rō )
||al|
LiDAR(e) exp 12plog
(18)
Combining equations 10 and
18 we get the result:
|| ||ll + r
r8
LiDAR(ē) < LiDAR(e) exp - 2p
log(
(19)
|
Note that an immediate consequence of Proposition is that for å << ||a| we have that Li)AR(ē) S LiDAR.
Eigenspectrum Decay As additional motivation, invoke the arguments made in (Stringer al. 2019) and later expanded in (Agrawal et al. (022) on the optimal eigenspectrum decay rate in an in- finite dimensional embedding space, given by A O(i-1), In (Stringer al. 2019). that a slower decay rate would necessarily imply a non-smooth kernel function, which, in turn, im- plies a non-monotonic relationship between rank and downstream performance. LiDAR circumvents this issue by implementing whitening operation through the inverse of Σ.,. This, in theory, permits the attainment of both a smooth embedding and the flexibility for eigenvalue decay in ELDA to oceur at very gradual rate. It is essential to acknowledge. however, that while the smoothness can be maintained, an excessively high LDA rank may, in theory, have adverse consequences on down- stream performance, owing to other underlying factors. We, therefore, defer more comprehensive theoretical exploration of the implications LiDAR to future research endeavors.
shown
14

---

Published as a conference paper at ICLR 2024
IMPLEMENTATION DETAILS
9.1 VICREG
VICReg (Bardes et al. 2021 proposes an explicit regularization prevent dimensional collapse in self-supervised learning methods. VICReg (Bardes et al. (2021) consists of the standard invariance term, variance term encourage networks to not suffer from dimensional collapse and covriance term that encourages the covariance matrix of embeddings be approximately diagonal. Let
Rd denote features from a neural network. The variance term is given by:
d-1 var (2) = max (0, 1 - S (zj, ()
(20)
==0
where S (3r, denotes the standard deviation and is small value to prevent numerical instability. The covariance term is given by:
c(2) Σ COZZ
(21)
while the invariance term is the L2 distance between the usual positive data pairs used to optimize the network. The complete loss function used optimize a network in VICReg is given by:
A Σ
L (2,
||
zz + [var (Z) + var (Z') + u - c (Z')
(22)
where A, 1 and y are hyperparameters that control the contribution from the invariance, standard deviation and covariance terms respectively.
We use the reference implementation provided by Bardes et al. (2021) 1 to train a ResNet-50 (He al.) 2016) backbone on Imagenet- IK dataset (Russakovsky et al.J 2013). The projector used standard multi-layer perceptron (MLP) with dimensions 2192-81992-8 92-8192. We train a ResNet-50 with VICReg for 100 epochs using hyperparameters and training protocol described in (Bardes) al. 2021). The trained backbone is probed by trianing classifier the frozen features of the backbone. We use setup that described in RankMe including data preprocessing and optimizer- related hyperparameters to train probe for 30 epochs. We 32 hyperparametr sets that include described in Garrido et al. (2022) in our experiments.
source dataset for self- supervised learning (SSL). We use 10000 images from the source dataset, i.e., ImageNet- training split and 10 augmentations per image construct labeled dataset needed calculate LiDAR and augmented-Ronimn We use 25600 images from the source dataset (ImageNet-IK) to calculate RankMe done by Garrido al. (2022). All pipelines that calculate various metries employ the exact data augmentation pipeline used for SSL pretraining. The results for VICReg are as follows:
We ImageNet- -K (Russakovsky
al. 2015) training data
We calculate the LiDAR, RankMe, sugmented-Ranklle for the final checkpoint after SSL training. Figure 3 shows scatter plot of the 32 checkpoints where plot the various effective rank metrics versus probe accuracy.
We calculate the effective rank metrics on checkpoints collected every 20 epochs during training. Figure 4 shows the evolution of the metrics during training.
Table 9 shows the correlations estimated via Spearman rank correlation and Kendall's correlation tests for all checkpoints collected during training.
15

---

Published as a conference paper at ICLR 2024
18
Probe
UDAR
Par Lite
Par late (aurg data)
Figure 3: VICReg: Performance of VICReg representations measured by RankMe and LiDAR. Each point refers checkpoint evaluated after 100 epochs of self--suprrvied pretraining. LiDAR shows strong correlation
Correlation
RankMe
RankMe (aug. dataset)
LiDAR
Spearman rank Kendall's 7
0.3174
0.5209 0.3790
0.9161
0.2056
0.8105
Table 9: VICReg: Compare RankMe and LiDAR using Spearman Rank correlation and Kendall's correlation measures after 100 epochs of training. VICReg representations used estimate RankMe and LiDAR for the 32 hyperparameter sets considered in our experiments. The hyper- paramter values are identical the values considered by (Garrido et al. 2022
Metric
COv.
inv.
LR
WD
Imagenet Oracle
64.7380
62,7800
63.9500
61.6500
RankMe
64.5400
59.5400
63.9500
59.5200
RankMe (aug. dataset)
64.5400
59,5400
63.9500
59.5200
LiDAR
64.7080
62.5720
63.9500
59.5200
Linear probe accuracy recovered by RankMe, augmented-Red -ankke and LiDAR ImageNet-1K dataset at the end of training. The metrics presented above calculated with representations
Table 10: VICReg:
Epoch
RankMe
RankMe (aug. dataset)
LiDAR
Metric
Cov.
inc
LR
WD
20
0.4476
0.4718
0.5323
Imagenet Oracle
64,7380
62.7800
63.9500
61.6500
40
0.2984
0,4597
0.7097
RankMe
64,5400
59,5400
63.9500
59,5200
60
0,2702
0.3750
0.7218
RankMe Coug, dataset)
64,5400
59.5400
63.9500
59,5200
LiDAR
59.5200
80
0.2823
0.4435
0.7823
100
0.2056
0.3790
0.8105
(b) Linear probe accuracy recovered by RankMe.
(a) Kendall's т correlation coefficient
Table 11: VICReg: (a) Kendall's correlation coefficient between effective rank estimated by RankMe and LiDAR and probe accuracy per hyperparameter and (b) Linear probe accuracy recovered by RankMe, augmented-RankMe and LiDAR ImageNet-1K 1 dataset at the end of training. The metrics presented above are calculated with representations.
16

---

Published as a conference paper at ICLR 2024
LDM
Ranictle
Rambble
18
Baniclie (aug) data)
Ranbbl laug. dina
50
Pribe 3UCUPTO
42
86
14
42-
bc
(a) training epoch
(b) training epochs
62
80
54
52
LIDR
JDAR
Ranickk
TantiMee
Raniche cuuq, datal
SanthMee (suc) dars)
Effeciive
(c) 40 training epochs
(d) 60 training epochs
00
00
JDAR
52
RonitWle dota)
SantMee
1100
1200
3700
1400
1500
3100
1200
1300
3000
1000
(c) training epochs
(f) 100 training epochs
Performance of VICReg representations measured by RankMe and LiDAR. Each point represents row of hyperparameters among the 32 sets of hyperparameters consdiered in our experiments. The hyperparamter values are identical to the values considered by (Garrido al. 2022)
Figure 4: VICReg:
17

---

Published as a conference paper at ICLR 2024
9.2 I-JEPA
Image-based Joint-Embedding Predictive Architecture (IJJEPA) (Assran et al., 2023) is a recently proposed non-generative approach to learn semantically strong representations manner. I-JEPA splits an image into a context block and several target blocks and uses the con- text block to predict the target blocks. Note that each block is composed of image patches. The core innovation in I-JEPA is the design of a masking strategy that is shown to lead to semanti- cally strong representations when used in conjunction with Vision Transformers (ViTs) (Dosovit- skiy et al.) 202T). I-JEPA uses a VIT to encode the non-masked context patches and another ViT to predict the encodings for the masked out target patches. The target representations are provided by target encoder which is an exponential moving average (EMA) version of the context encoder. In this work we refer to the context encoder as the student encoder and the target encoder as the teacher and use these terms interchangeably in our presentation. The loss function is applied to the embeddings that are output by the student and the teacher and is given by:
self-supervised
M EEΣN lv &.11
(23)
70 iEB:
where M denotes the number of target blocks, B3 denotes a set of block indices in target and y and ý denote the embeddings provided by the student and the teacher respectively. The asymmetry in- troduced due to student and teacher encoders allows I-JEPA avoid representation collapse (Assran
et al. 2023
In order to apply LiDAR for I-JEPA Assran et al. 2023) we create a labeled dataset by first selecting a fixed number of images at random and treat each image as a class. We then apply the masking approach proposed in I-JEPA (Assran et al. 2023) to create multiple instances of class to create a labeled dataset needed to calcualte the linear discriminant analysis matrix for LiDAR. The embed- dings y and ÿ described in 23 are used as inputs to estimate Student and Teacher LiDAR measures. We use the output of the student encoder to calculate RankMe measure. As described in Section 4.2 the embeddings y and lhaty are used to calculate the augmented RankMe metric which is denoted as RankMe (aug.) in the results. We use 1000 images and 50 augmentations to construct the la- beled dataset for LiDAR and nugmented-Randee
while we use 10000 image samples for RankMe. Self-supervised training is run for 600 epochs with an effective batch size of 2048 using the training protocol described in I-JEPA (Assran et al. (023). The downstream task consists of linear probing frozen representations on the ImageNet- K dataset (Russakovsky et al. 2015). The probe is opti- mized with Adam (Kingma & Ba, 2013) optimizer for 20 epochs with starting learning rate of 0.01 and a step learning rate schedule where the base learning rate is dropped by a factor 10 after 15 epochs. The following hyperparamter sets are used in our experiments to empirically estimate the performance of RankMe augmented-Rankee and LiDAR:
Learning rate from 0.001, 0.002, 0.004, 0.006 and 0.008. Figure 5 Figure 6 show the re- sults of experiments where we vary the learning rate parameter alone. Table 12 and Ta- [ the rank correlation coefficients while Table 3 show the accuracy recovered by RankMe and LiDAR.
Weight decay from 0.05, 0.2and 0.4. 7 Siow the results of experiments where we vary the weight decay parameter alone. Note that the weight decay is kept fixed throughought training in this set of experiments. Table T2] and Table [ show the rank correlation coefficients while Table show the accuracy recovered by RankMe and LiDAR.
Target mask scale factor from [0.15, 0.2], [0.125, 0.2), [0.2, 0.25] and [0.2, 0.3]. Figure 9 Figure [0] show the results of experiments where we target mask scale ratio alone. Ta- ble 12 and Table the rank correlation coefficients while Table 3 the accuracy recovered by RanKMe and LiDAR.
Target mask scale factor from [0.85, 1.0), [0.75, 1.0). [0.65, 1.0 and [0.4, Figure II Figure show the results of experiments where we target mask scale ratio alone. Ta- ble I2 and Tshl the rank correlation coefficients while Table the accuracy recovered by RankMe and LiDAR.
Additionally, we conduct random hyperparameter search experiment where sample 20 sets of hyperparamters uniformly and train a ViT-B (Dosovitskiy et al.) (02T) with I-JEPA (Assran et al. 2023). The hyperparametrs were sampled from the following uniform distributions:
18

---

Published as a conference paper at ICLR 2024
500
Ir=0.001
Probe accuracy
60
400
Ir=0.001
LDM (student)
200
Ir=0.002
40
t=00001
0.002
RankMe
300
Ir=0.004
1r=0.002
150
Ir-0.004
0r00006
0.004
200
1r00.006
100
Ir 0.000
20
00006
1r=0.008
100
50
200
400
600
200
400
600
200
400
600
Epoch
Epoch
(a) Linear probe accuracy
(b) RankMe
(c) LiDAR (student)
Figure 5: I-JEPA: ViT-Base architecture trained on Imagenet-1 K by varying the learning rate. Plots show the evolution of metrics over training time.
70
70
probe accuracy
50
accuracy
50
30
LiDAR (student)
LIDAR (leacher)
10
RankMe
10
RankMe
RankMe (aug.) (student)
RankMe (aug.) (teacher)
100
200
300
400
500
100
200
300
400
500
effective rank
effective rank
Figure 6: I-JEPA: Scatter plot of checkpoints collected during training of ViT-Base architecture Imagenet-1K IK by varying the learning rate.
learning rate from (0.0001250.0002]
weight decay from [0.050.4]
target scale (minimum) from [0.10.2]
target scale (maximum) from [0.20.4]
context scale (minimum) from [0.30.95]
The results of these experiments are shown in Figure and the correlations available in Table and Table 13 The probe accuracy recovered for hyperparameters generated via random search are shown in Table
Probe accuracy
60
400
LIDAR (student)
200
wd+0.05
40
RankMe
300
150
wd=0.05
0.05
wd=0.1
200
100
wd=0.1
wd 0.2
wd=0.2
50
wd=0.4
wd=0.4
400
600
0
200
400
600
200
400
600
Epoch
Epoch
Epoch
(a) Linear probe accuracy
(b) RankMe
(c) LiDAR (student)
Figure 7: I-JEPA: ViT-Base architecture trained on Imagenet- IK by varying the weight decay. Plots
show the evolution of metrics over training time.
19

---

Published as a conference paper at ICLR 2024
70
60
accuracy
40
30
LIDAR (student)
LIDAR (teacher)
RankMe
10
RankMe
RankMe (aug.) (student)
RankMe (aug.) (teacher)
100
200
300
400
100
200
300
400
effective rank
effective rank
Figure 8: I-JEPA: Scatter plot of checkpoints collected during training of ViT architecture on Imagenet-1K by varying weight decay.
Probe accuracy
60
LIDAR (student)
tgt-(0.15, 0.2]
tgt=(0.15, 0.21
tgt=(0.15, 0.2]
tgt=(0.125, 0.2]
200
bgt-10.125, 0.2]
100
bgt-(0.125, 0.2]
20
tgl=[0.2, 0.25]
tgt=[0.2, 0.25]
bgt=(0.2, 0.25]
100
50
tgt=(0.2, 0.3]
tgt-(0.2, 0.33
tgt-[0.2, 0.3]
400
600
600
Epoch
Epoch
Epoch
(a) Linear probe accuracy
(b) RankMe
(c) LiDAR (student)
Figure 9: I-JEPA: ViT-Base architecture trained Imagenet-IK IK by varying the target mask scale hyperparameter. Plots show the evolution of metrics over training time.
70
70
50
40
30
LiDAR (student)
LIDAR (teacher)
RankMe
10
RankMe
RankMe (aug.) (student)
RankMe (aug.) (teacher)
100
200
300
400
100
200
300
400
500
effective rank
effective rank
Figure 10: 1-JEPA: Scatter plot of checkpoints collected during training of ViT-Base architecture Imagenet-Ile IK by varying the target mask scale factor.
20

---

Published as a conference paper at ICLR 2024
Probe accuracy
60
400
LIDAR (student)
200
ctx=[0.85, 1.0]
ctx=|0.85, 1.0]
ctx=|0.85,
clx=[0.65, 1.0]
RankMe
300
[0.65, 1.0]
150
ctx=|0.65,
ctx (0.-5 1.0]
[0.75. 1.0]
(0.550.75,
20
ctx (0.4, 1.0]
ctx-
[0.4, 1.0]
c0.4
100
0
600
200
400
Epoch
Epoch
Epoch
(a) Linear probe accuracy
(b) RankMe
(c) LiDAR (student)
Figure 11: I-JEPA: ViT-Base architecture trained on Imagenet- IK by varying the context mask scale hyperparameter. Plots show the evolution of metrics over training time.
60
60
accuracy
40
40
20
LiDAR (student)
LiDAR (teacher)
RankMe
RankMe
RankMe (aug.) (student)
RankMc (aug.) (teacher)
4
8%
100
200
300
400
100
200
300
400
effective
effective
Figure 12: 1-JEPA: Scatter plot of checkpoints collected during training of ViT-Base architecture on Imagenet-1K K by varying the context mask scale factor.
70
70
50
accuracy
50
30
30
LiDAR (student)
LIDAR (leather)
10
RankMe
10
RankMe
RankMMe (aug.) (student)
RankMe (aug.) (teacher)
100
200
300
400
500
100
200
300
400
500
effective rank
effective rank
Figure 13: I-JEPA: Aggregated scatter plot of checkpoints collected during training of ViT-Base architecture on Imagenet- IK. Hyperparameters varied are one learning rate, weight decay, target mask scale context mask scale. Plots show that effective rank estimated by LiDAR has high correlation with linear probe accuracy.
21

---

Published as a conference paper at ICLR 2024
Hyperparameter
RankMe
RankMe (aug.)
LiDAR (Student)
RankMe (aug.) (Teacher)
LiDAR
(Student)
(Teacher)
Learning rate
0.8775
0.9258
0.9605
0.8747
0.9030
Weight decay
0.6381
0.8669
0.8884
0.6196
0.6978
Target mask scale
0,4008
0.8429
0.9407
0.4069
0.3851
Context mask scale
0,5800
0,9180
0.9590
0.5533
0.6940
Overall
0.7700
0.9104
0.9494
0.7627
0.8475
Table 12: I-JEPA: Spearman rank correlation coefficient for I-JEPA between effective ranks of RankMe, RankMe (aug.) and LiDAR and linear probe accuracy.
RankMe
RankMe (aug.)
LiDAR (Student)
RankMe (aug.)
LiDAR (Teacher)
(Student)
(Teacher)
Random search
0.9470
0.9770
0.9511
0.9486
0.9722
Table 13: I-JEPA: Spearman rank correlation coefficient between effective ranks of RankMe, augmented- RankMe dom sampling.
MankeMe and LiDAR and linear probe accuracy. Hyperparmaeters are generated via ran-
9.3 DATA2VEC
data2vec (Baevski
2022 is self-supervised learning approach that aims to predict la- tent representations of input data based on a masked view of the input data. The idea behind data2vec (Baevski al., 2022) similar I-JEPA (Assran al., 2023) but the details of mask- ing and the predictor used to predict embeddings are different.
We follow the same protocol described for I-JEPA above in terms of constructing labeled dataset for LiDAR and augmented-Ranine RannMMe with 10000 samples from ImageNet- IK and apply 50 augmenta- tions. The augmentations applied in this case are identical to the augmentation described in (Baevski) al. 2022) and available via a reference implementation provided by the authors We use 10000 source images to calculate RankMe. We train a ViT-B (Dosovitskiy et al.) 2021 model for 800 epochs with effective batch size of 2048 and with other hyperparameters described in (Baevski et al. 2022). During training, we save a checkpoint every 40 epochs for analysis and downstream linear probing. Linear probing done frozen representations of the encoder using LARS opti- mizer (You et al. (017)) with a base learning rate of 0.01 scaled via linear scaling rule with a base batch size of 256 GGoya al. 2018) and effective batch size of 16384 samples. We consider the following SSL training hyperparameters in our experiments to test the performance of LiDAR and RankMe and its augmented variant:
Learning rate from 0.0007, 0.001, 0.002 and 0.004. Figure 14 and Table 14 and Table show the results from these experiments
Set number of mask patches to either 120 (default) or while we fix the learning rate to 0.0007. Figure T6 Table 14 and T5 the results from these experiments
Table 15 shows the probe accuracy recovered by the various effective rank metrics.
22

---

Published as a conference paper at ICLR 2024
250
400
40
300
30
200
1r00.001 mask pwwppptcc_sstp 120
F_0.00, mast_ mas_mmmas_ssheeeee pat_checc__st_ 120
V,0,001, mask_aacches, 120
1r 0,0007, mask patches 125
0,0000,, musk patches 120
100
0.0000,, patches 120
maw. potches
50
y_0,0007, maxt patchhe 120
1.,0.004 masss_patches_120
200
400
600
800
200
400
600
800
20c
400
600
800
spoc/s
opochs
(a) Linear probe accuracy
(b) RankMe
(c) LiDAR (student)
Figure 14: data2vec: ViT-Base architecture trained on Imagenet- I by varying the learning rate.
Plots show the evolution of metries over training time.
60
50
50
probe accuracy
accuracy
40
40
30
LIDAR (student)
LIDAR (teacher)
10
RankMe
10
RankMe
RankMe (aug.) (student)
RankMe (aug.) (teacher)
0
100
200
300
400
0
100
200
300
400
500
effective rank
effective rank
Figure 15: data2vec: Scatter plot of checkpoints collected during training of ViT-Base architecture on Imagenet-IK. IK. The learning rate was varied this experiment.
50
400
40
300
20
200
50
mask_oarches__12 0.0007, musk penchee
F.0.0007, merk patcher
V.O. DOBT, mast putchee 120
1-0 0007, mask_patches__e
1.0. 0007, mask_patchessee
200
200
400 opocts
20c
60c
abc
epochs
spochs
(a) Linear probe accuracy
(b) RankMe
(c) LiDAR (student)
Figure 16: data2vec: ViT-Base architecture trained Imagenet-IK K by varying the masking ratio. Plots show the evolution of metrics over training time.
Hyperparameter
RankMe
RankMe (aug.) (Student)
LiDAR (Student)
RankMe (aug.)
LiDAR (Teacher)
(Teacher)
Learning rate
0.3639
0.4440
0.6680
0.4191
0.6240
Mask ratio
0.3720
0.3026
0.6089
0.2785
0.5466
Overall
0.3408
0.3795
0.7275
0.3627
0.6962
Table 14: data2vee: Spearman rank correlation coefficient for data2vec between effective ranks of RankMe, RankMe (aug.) and LiDAR and linear probe accuracy.
23

---

Published as a conference paper at ICLR 2024
50
probe accuracy
50
20
LiDAR (student)
LIDAR (teacher)
RankMe
RankMe
RankMe (aug.) (student)
RankMe (aug.) (teacher)
100
200
300
400
100
200
300
400
500
effective rank
effective rank
Figure 17: data2vec: Scatter plot of checkpoints collected during training of ViT-Base architecture on Imagenet-Ike IK. The masking ratio was varied in this experiment.
60
50
200
300
150
1,000011,o
120
200
Paii_patccee_11
20
0.0007,
patches
mask patches c20
10
0.0007, meak pwatteee_eete
200
000.000, mast petthee 120
0.0001, musk patches 50
r.0. 002, mark patches 120
0.9007, mask putcheu 80
0,004, mask potches 120
K.0.004, mart, pwithes 150
000, mask perches 170
600
opechs
4po276
spochs
(a) Linear probe accuracy
(b) RankMe
(c) LiDAR (student)
Figure 18: data2vee: ViT-Base architecture trained on Imagenet- IK by varying one of learning rate or masking ratio. Plots show the evolution of metrics over training time.
60
accuracy
accuracy
40
30
20
20
LiDAR (student)
LiDAR (leacher)
RankMe
10
RankMe
RankMe (aug.) (student)
(aug.) (teacher)
200
400
effective rank
effective rank
Figure 19: data2vec: Aggregated seatter plot of checkpoints collected during training of ViT-Base architecture on Imagenet- IK. Hyperparameters varied are one of learning rate and mask ratio (num- ber of masked patches).
24

---

Published as a conference paper at ICLR 2024
Metric
LR
Mask ratio
Overall
ImageNet Oracle
60.3920
55.9780
60.3920
RankMe
48.6040
48.6980
48.6980
RankMe (aug.) (Student)
48.6040
51,4500
51,4500
LiDAR (Student)
59.3720
52.7460
59,3720
RankMe (aug.) (Teacher)
48.6040
51.4500
51.4500
LiDAR (Teacher)
59.3720
52.7460
59.3720
Table 15: data2vec: Linear probe accuracy recovered by RankMe and LiDAR on ImageNet- K dataset.
9.4 DINO
DINO (Caron et al. 202T) is an example of a self-distillation approach to learning representations in a self- sulf-suuer manner. DINO uses student and teacher encoder (weights updated via exponen- tial moving average) .multiple-croe cropps and small patches to train VIT (Dosovitskiy et al. (021]) in
self-supervised manner. Evaluations on multiple tasks presented in DINO (Caron et al. 202] 1)
show
that the resulting encoder learns strong representations. The loss function for DINO (Caron et al.) 202T) minimizes the cross-entropy between the probability distributions provided by the teacher and
student network. We reproduce description below from DINO (Caron al. 202 for complete- ness:
min H (Pr P2),
(24)
where H denotes the cross-entropy function, P. and P. denote the probability distributions output by the student and teacher networks respectively. The probabilities are computed via:
(2)
(2) Σk=1 exp(fem.. (2) /Tnet)
(25)
where the subscript "net" denotes either the student the teacher network, is a neural network parameterized by 0 and z(1) is feature vector for sample i. A key hyperparameter in DINO (Caron et al. 202T) is the temperature value т, and Tę used to control the sharpness of the output distribution produced by the softmax function. We test the performance of RankMe, augmented-Ranktl and LiDAR to predict linear probing performance by varying these parameters. We use 1000 images from the source dataset and apply 50 augmentations used in DINO training construct dataset to calcuate LiDAR. We use 25600 samples to estimate RankMe.
al.
We train a VIT-S with patch size of 16 16 using the protocol described by DINO CCron al., 2021] and implemented in a reference implementation provided by the authors of DINO on the Imagenet- K (Russakovsky et al. (2015) dataset. The projection head consists of 3-layer MLP with hidden dimension of 2048 and an output dimension referred to bottleneck dimension of 256 that provides embeddings. These embeddings are that projected to a higher dimensional space of 65536 in our experiments. We use the embeddings to estimate RankMe, our experiments consistent with the methodology adopted in RankMe (Garrido et al.) 2022). We use an effective batch size of 512 images and train the model for 300 epochs. Our experiments consists of:
Rankke and LiDAR in
varying the teacher temperature (7t) while keeping the student temperature (7₂) fixed to 0.1. The values considered for (Tr) are {0.02, 0.04, 0.06, 0.07}
varying the student temperature (ra) while keeping the teacher temperature (Tt) fixed. The values considered for (T,) are {0.2, 0.4]
We keep the rest of the training and linear probing hyperparameters identical those provided in the official implementation 0 The results of these experiments are available in Figure 20 and the correlation values are quantified Table 16. Table lists the linear probing accuracy recovered by ImageNet Oracle, RankMe and LiDAR metrics for the source dataset (ImageNet-IK)) IK).
httpp::/ www.. http:////w.cceoooeeeeeeereeeaaaa
25

---

Published as a conference paper at ICLR 2024
/B
74-
72-
70
188
.6-
160
120
200
100
140
180
200
220
240
Eftectivn Benii
Effectiv Bank
Figure 20: DINO: Aggregated scatter plot of checkpoints collected during training of ViT-Small architecture on Imagenet- IK. Hyperparameters varied are one of teacher or student softmax temper- ature described in the implementation section.
Correlation
RankMe
LiDAR (Student)
LiDAR (Teacher)
Spearman rank
0.8191 0.6288
0.8807 0.7157
0.8732
Kendall's 7
0.7299
Table 16: DINO: Compare RankMe and LiDAR with Spearman rank and Kendall's correlation coefficient metrics. Observe that LiDAR performs better than RankMe for both measures.
Metric
Teacher temperature
Student temperature
Overall
ImageNet Oracle
75.45
75.37
75.45
RankMe
75.38
75.28
75.38
LiDAR (Student)
75.38
75.28
75.38
LiDAR (Teacher)
75.38
75.28
75.38
Table 17: DINO: Average linear probe accuracy recovered by RankMe and LiDAR ImageNet-IK: Hyperparameter values are described in Appendix
9.5 SIMCLR
SimCLR (Chen et al. 2020) is an example of contrastive joine-eeeii-- self-supservised learning
method. The loss function for SimCLR is given by (Chen et al. 2020: Garrido et al. 2022) and
included below for completeness:
(Smmllrrtty(x..s)
L= E IJEEP En StmlorrSmmtr))t
(26)
where Similarity denotes the cosine similarity between vectors z, and zy. denotes the indi- cator function, P is the set of all positive pairs and the number of examples in given by N.
We train ResNet-50 backbone (He al. 2016) with a layer MLP projector with hidden di- mensions 8192 and output dimension equal to 2048. In other words, the embeddings produced by SimCLR have dimension equal to 2048. The network described above trained with the LARS optimizer (You et al. (017) and other settings described in (Chen et al. (020) on the ImageNet- Ik Ruussakovsyy et al. 2015) training split. The representations from the backbone are evaluated via standard linear probing by training a linear layer on ImageNet- Ik training split and calculating test accuracy on the validation split. Probing is performed using SGD optimizer with Nesterov mo- mentum with hyperparameters described in RankMe (Garrido al.) 2022). We use 5000 images from the source dataset (ImageNet-Iee IK) and apply 10 augmentations per image to calculate LiDAR. We use 25600 images calcuate RankMe to be consistent with the setup used by Garrido et al.
26

---

Published as a conference paper at ICLR 2024
58
Probe accrroc)
52
50
LIDRR
RankMe
RankMe (aug. data)
250
300
350
400
450
Effective rank
Figure 21: SimCLR: Performance of SimCLR representations measured by RankMe and LiDAR. Each point refers to checkpoint evaluated with a randomly searched hyperparameter after 100 epochs of self-supervised pretraining. LiDAR shows strong correlation
58
P0
54
Probo octuissy
52
RankMe
(aug
rank
Figure 22: SimCLR: Performance of SimCLR representations measured by RankMe and LiDAR. Each point refers to a checkpoint evaluated with the grid search hyperparameter from Rankme paper after 100 epochs of self- self-supervised pretraining. LiDAR shows strong correlation
2022
We consider two sets of experiments to test the performance of LiDAR and RankMe with (Chen al. 2020):
SimC
A grid search with the hyperparametrs sets that include learning rate, weight decay, em- bedding dimension and softmax temperature used in RankMe (Garrido et al. 2022). Fig- shows scatter plot of the probe accuracy vs. effective rank estiamted by RankMe, quantifies the correlations for the above methods. Table 21 shows the linear probe accuracy recovered by the various metries and compares the results to ImageNet Oracle
ure 22
mugmented-RankMe and LiDAR.
A random hyperparameter search by randomly sampling learning rate, weight decay and softmax temperature to generate hyperparameter sets. create table hyper- pameters with learning rate chosen from [0.3, 0.4, 0.5, 0.6). weight decay chosen from [10-?, 10 -6,66,0 10-5] and softmax temperature chosen from [0.05, 0.1, 0.15, 0.2, 0.25). We select 30 sets of hyperparameters from this table and use these to train models with Sim- CLR and evaluate performance. Figure[1 shows the results for this experiment as scatter plot and Table quantifies the correlations.
27

---

Published as a conference paper at ICLR 2024
Hyperparameter
RankMe
RankMe
LÍDAR
Hyperparameter
RankMe
RankMe
LIDAR
(aug. dataset)
(aug. dataset)
Learning rate
0.5248
0.5301
0.9138
Learning rate
0.5706
0.4694
0.7512
Weight decay
0.4019
0.5573
0.8964
Weight decay
0.3580
0.4221
0.7148
Temperature
0.5704
0.6764
0.9182
Temperature
0.4209
0.7061
0.8435
Overall 0.5125
0.6304
0.9155
Overall
0.4906
0.5581
0.7967
(a) Spearman Rank correlation coefficient
Kendall's comeelatinn coefficient
Table 18: SimCLR: Correlation between effective rank estiamted by RankMe and LiDAR and probe accuracy per hyperparameter.
Correlation
RankMe
RankMe (aug. dataset)
LiDAR
Spearman rank
0.5125
0.6304 0.5581
0.9155
Kendall's r
0.4906
0.7967
Table 19: SimCLR: Compare RankMe and LiDAR using Spearman Rank correlation and Kendall's correlation measures evaluated during hyperparameter grid search.
Correlation
RankMe
RankMe (aug. dataset)
LiDAR
Spearman rank
0.5301
0.6389 0.5761
0.9188
Kendall's r
0.4982
0.8167
Table 20: SimCLR: Compare RankMe and LiDAR using Spearman Rank correlation and Kendall's correlation measures evaluated during hyperparameter random search.
Metric
LR
WD
Temp.
Overall
Imagenet Oracle
58.2370
56.6740
57.4710
59.1420
RankMe
55.8950
55.1490
56.0390
56.4630
RankMe (aug. dataset)
57.2010
55.8170
56.3170
57.8260
LiDAR
57.8940
56.3020
57.0920
58.9270
Table 21: SimCLR: Linear probe accuracy recovered by RankMe, augmented RankeMe and LiDAR on ImageNet- IK dataset at the end of training. Results are from trials run with hyperparameter random search.
28

---

Published as a conference paper at ICLR 2024
Metric
LR
WD
Target mask
Context mask scale
Overall
scale
ImageNet Oracle
78.49
78.49
78.49
78.49
78.49
RankMe
72.09
75.45
75.45
75.15
72.09
RankMe (aug.) (Student)
77.78
77.78
77.78
77.78
77.78
RankMe (aug.) (Teacher)
72.09
72.14
70.20
75.15
72.09
LiDAR (Student)
77.78
77.78
77.78
77.78
77.78
LiDAR (Teacher)
74.33
76.84
73.91
78.30
74.33
Table 22: I-JEPA: Average linear probe accuracy recovered by RankMe and LiDAR on OOD datasets. Hyperparameters set via grid search.
10 OUT-OF-DISTRIBUTION (OOD) DATASETS EVALUATION
10.1 EVALUATION PROTOCOL
In order to evaluate RankMe and LiDAR methods, we use CIFAR1O, CIFAR100 (Krizhevsky & Hinton 2009), EuroSAT (Helber et al. 2019), Food101 (Bossard et al. 2014) and SUN397 (Xiao ct at 2010 as unseen or OOD datasets. These commonly used datasets are different from our source dataset, ImageNet-Ik Ik, and are used to evaluate the effectiveness of representations on downstream tasks. The downstream task protocol considered here is linear probing with a frozen backbone. We follow the protocol used in VISSL (Goyal et al., 2021 train and evaluate linear probe with the above OOD datasets.
For all datasets, to optimize the linear classification layer. We use learning rate of 0.01, momentum value of 0.9, weight decay 0.0005 and cross-entropy loss to train the linear layer for 28 epochs. We use a multi- step learning rate schedule where the learning rate is dropped by factor of 0.1 every 8 steps. We follow the data augmentation procedure used VISSL (Goyal et al. 2021 that consists of random resized crops and random horizontal flips for training and center crop for evaluation. The results of the OOD datasets evaluation are listed in the following sections.
stochastic gradient descent with Nesterov (SGD w/Nesterov) momentum
10.2 I-JEPA
We study the OOD performance of I-JEPA model pretrained on ImageNet-1K 1K as described in Ap- pendix 9.2 The checkpoints chosen by ImageNet Oracle, RankMe, its augmented variant and LiDAR are trained on OOD datasets using the protocol described Appendix [TO Table and Table lists the average probe accuracy calculated over the five OOD datasets for hyperparameters selected by grid search and random search, respectively. We observe from Table 22 that LiDAR calculated with student branch outperforms RankMe in terms of linear probe accuracy OOD datasets except for the context scale parameter where-in LiDAR calculated with the teacher branch performs slightly better than its student counterpart. This observation is an interesting contrast the correspond- ing in-domain linear probe accuracy shown in Table 3 where-in the student branch-based LiDAR works well across all hyperparameters. Another remarkable observation here that our proposed augmented- -aankeee- variant performs better than vanilla RankMe on OOD dataset which is consis- tent with our observations made with in-domain data in Table 3 Finally, we see from Table 22 that while ImageNet Oracle shows the best performance on OOD datasets, LiDAR is able to recover most of the linear probe accuracy on these datasets as well. Table 23 lists per-dataset breakdown of the linear probe accuracy for checkpoints selected by various metrics.
shows the performance I-JEPA checkpoints on OOD datasets trained with hyperparam- Table 24
eters selected via random search. We observe from Table 24 that both variants of LiDAR outperform RankMe and its augmented variants and remarkably outperforms the OOD linear probe accuracy recovered by the ImageNet Oracle. This observation shows that OOD performance can be improved under certain conditions which is also an observation made by Garrido et al. (2022). Table lists a per-dataset breakdown of the linear probe accuracy for checkpoints selected by various metrics.
29

---

Published as a conference paper at ICLR 2024
Hyperparameters
Dataset
Metric
LR
WD
Target mask
Context mask scale
Overall
scale
ImageNet Oracle
89.66
89.66
89.66
89.66
89.66
RankMe
86.92
88.8
88.8
87.75
86.92
RankMe (aug.) (Student)
89.64
89.64
89.64
89.64
89.64
CIFARIO
RankMe (aug.) (Teacher)
86.92
86.04
85.04
87.75
86.92
LiDAR (Student)
89.64
89.64
89.64
89.64
89.64
LiDAR (Teacher)
87.86
89.29
82.73
89.99
87.66
ImageNet Oracle
69.03
69.03
69.03
69.03
69.03
RankMe
63.57
67.19
67.19
66.52
63.57
RankMe (aug.) (Student)
68.81
68.81
68.81
68.81
68.81
CIFARI00
RankMe (aug.) (Teacher)
63.57
63,97
63.07
66.52
63.57
LiDAR (Student)
68.81
68.81
68.81
68.81
68.81
LiDAR (Teacher)
65.04
68.66
56.46
69.28
65.04
ImageNet Oracle
95.02
95.02
95.02
95.02
95.02
RankMe
94.56
95.68
95.68
95.62
94.56
RankMe (aug.) (Student)
95.18
95.18
95.18
95.18
95.18
EuroSAT
RankMe (aug.) (Teacher)
94.56
95.42
94.92
95.62
94.56
LiDAR (Student)
95.18
95.18
95.18
95.18
95.18
LiDAR (Teacher)
93.16
95.54
94.6
95.6
93.16
ImageNet Oracle
72.65
72.65
72.65
72.65
72.65
RankMe
58.93
65.22
65.22
65.49
58.93
RankMe (aug.) (Student)
70.49
70.49
70.49
70.49
70.49
Food101
RankMe (aug.) (Teacher)
58.93
59.49
55.78
65.49
58.93
LiDAR (Student)
70.79
70.79
70.79
70.79
70.79
LiDAR (Teacher)
65.13
67.96
71.59
71.86
65.13
ImageNet Oracle
66.08
66.08
66.08
66.08
66.08
RankMe
56.46
60.36
60.36
60.37
56.46
RankMe (aug.) (Student)
64.47
64.47
64.47
64.47
64.47
SUN397
RankMe (aug.) (Teacher)
56.46
55,8
52.17
60.37
56.46
LiDAR (Student)
64.47
64.47
64.47
64.47
64.47
LiDAR (Teacher) 60.47
62.75
64.16
64.78
60.47
Table 23: I-JEPA: Top-1 accuracies computed on representations when tuning hyperparameters with ImageNet validation performance, RankMe, augmented RankMe and LiDAR. Hyperparameters set via grid search.
ImageNet Oracle
RankMe
RankMe (aug.) (Student)
RankMe (aug.)
LiDAR (Student)
LiDAR (Teacher)
(Teacher)
74.85
73.24
73.24
73.24
76.23
77.34
Table 24: I-JEPA: Average linear probe accuracy recovered by RankMe and LiDAR on OOD
datasets. Hyperparameters set via random search.
30

---

Published as a conference paper at ICLR 2024
Dataset
ImageNet Oracle
RankMc
RankMe (aug.) (Student)
RankMc (aug.) (Teacher)
LiDAR (Suudent)
LiDAR
(Teacher)
CIFARIO
82.69
87.32
87.32
87.32
86.21
87.29
CIFAR100
58.66
65.98
65.98
65.98
65.92
64.91
EuroSAT
95.60
95.68
95.68
95.68
96.04
96.06
Food101
70.94
61.16
61.16
61.16
69.60
71.86
SUN397
66.36
56.04
56.04
56.04
63.38
66.59
Table 25: I-JEPA: Top-1 accuracies computed on representations when tuning hyperparameters with
ImageNet validation performance, RankMe, augmented RankMe and LiDAR. Hyperparameters set via random search
Metric
LR
Mask ratio
Overall
ImageNet Oracle
68.69
69.07
68.69
RankMe
58.86
58.86
58.86
MankMe (aug.) (Student)
58.86
60.76
60.76
RankMe (aug.) (Teacher)
58.86
60.76
60.76
LiDAR (Student)
63.76
73.07
63.76
LiDAR (Teacher)
63.76
73.07
63.76
Table 26: data2vec: Average linear probe accuracy recovered by RankMe and LiDAR on OOD datasets. Hyperparameters set via grid search.
10.3 DATA2VEC
In this section, we study the OOD performance of data2vec models trained on ImageNet-IK, the details of which are described in Appendix 9.3 We select checkpoints based on ImageNet Oracle, RankMe and LiDAR as described in Appendix(9.3 and train them on the 5 OOD datasets described in Appendix [0. Table 26) shows the average linear probe accuracy of data2vec checkpoints chosen by ImageNet Oracle, RankMe,
Caanented-Rankkeee and LiDAR. We observe from Table 26 that LiDAR outperforms RankMe and its augmented variant and in the case of mask ratio hyperparameter outperforms the ImageNet Oracle as well.The latter observation supports those made by Garrido al. (2022) the need have new methods that do not rely ImageNet performance alone for checkpoint selection. Table 27 lists the per-dataset breakdown of linear probe accuracy for data2vec checkpoints chosen by various metrics listed above.
10.4 DINO
In this section, we study the OOD performance of DINO models trained on ImageNet- 1 the details of which are described Appendix 9.4 We select checkpoints based on ImageNet Oracle, RankMe and LiDAR as described in Appendix 9.4 and train them on the 5 OOD datasets described in Ap- pendix IO.I Table 28 lists the average linear probe accuracy for the OOD datasets from which we observe that LiDAR is able to show the same slightly better performance than both RankMe and ImageNet Oracle for DINO hyperparameter selection. Table 29 shows a per-dataset breakdown of probe accuracy OOD datasets.
10.5 VICREG
The VICReg model is pretrained on ImageNet using the same hyperparameter grid as the one pub- lished in (Garrido et 2022) (each grid point for 100 epochs). From the set of the last checkpoints grid point, a pool of the best-performing models, according to ImageNet validation performance, RankMe, and the proposed LiDAR, was selected. We conducted probe analysis on the resulting pool of checkpoints using five different OOD datasets. The probe analysis employed learning parameters reported in (Garrido et al. 2022) (standard protocol in VISSL (Goyal et al. 202T)). Table 30 below summarizes the probe accuracies averaged over the OOD datasets considered in our evaluation for the checkpoint selected by each metric (one of ImageNet Oracle, RankMe and LiDAR) for a given hyperparameter. We observe that LiDAR consistently outperforms RankMe across all hyperparam-
31

---

Published as a conference paper at ICLR 2024
Hyperparameters
Dataset
Metric
LR
Mask ratio
Overall
ImageNet Oracle
71.63
74.62
71.63
RankMe
60.21
60.21
60.21
RankMe (aug.) (Student)
60.21
61.11
61.11
CIFARIO
RankMe (aug.) (Teacher)
60.21
61.11
61.11
LiDAR (Student)
63.87
84.46
63,87
LiDAR (Teacher)
63.87
84.46
63.87
ImageNet Oracle
42.48
46.85
42,28
RankMe
31.15
31.15
31,15
RankMe (aug-) (Student)
31.15
31.88
31,88
CIFAR100
RankMe (aug.) (Teacher)
31.1
31.88
31.88
LiDAR (Student)
33.98
60.20
33,98
LiDAR (Teacher)
33.98
60.20
33.98
ImageNet Oracle
93.06
92.98
93,06
RankMe
84.30
84.30
84.30
RankMe (aug.) (Student)
84.30
85.22
85.22
EuroSAT
RankMe (aug.) (Teacher)
84.30
85.22
85.22
LiDAR (Student)
87.20
95.12
87.20
LiDAR (Teacher)
87.20
95.12
87.20
ImageNet Oracle
70.41
68.16
70.41
RankMe
61.18
61.18
61.18
RankMe (aug.) (Student)
61.18
65.64
65.64
Food101
RankMe (aug.) (Teacher)
61.18
65.64
65.64
LiDAR (Student)
69.04
64.80
69.04
LiDAR (Teacher)
69.04
64.80
69.04
ImageNet Oracle
66.09
62.73
66.09
RankMe
57.47
57.47
57.47
RankMe (aug.) (Student)
57.47
59.98
59.98
SUN397
RankMe (aug.) (Teacher)
57.47
59.98
59.98
LiDAR (Student)
64.72
60.79
64.72
LiDAR (Teacher)
64.72
60.79
64.72
Table 27: data2vec: Top-1 accuracies computed representations when tuning hyperparameters with ImageNet validation performance, RankMe, augmented RankMe and LiDAR. Hyperparameters via grid search.
Metric
Teacher temperature
Student temperature
Overall
ImageNet Oracle
84.66
84.59
84.66
RankMe
84.66
84.46
84.66
LiDAR (Student)
84.66
84.72
84.66
LiDAR (Teacher)
84.66
84.72
84.66
Table 28: DINO: Average linear probe accuracy recovered by RankMe and LiDAR on OOD datasets. Hyperparameter values are described in Appendix
32

---

Published as a conference paper at ICLR 2024
Hyperparameters
Dataset
Metric
Teacher temperature
Student temperature
Overall
ImageNet Oracle
95.41
95.35
95.41
RankMe
95.40
95.32
95.40
CIFARIO
LiDAR (Student)
95.40
95.51
95,40
LiDAR (Teacher)
95.40
95.51
95.40
ImageNet Oracle
81.27
81.31
81.27
RankMe
81.67
81.31
81.67
CIFAR100
LiDAR (Student)
81.67
81.94
81.67
LiDAR (Teacher)
81.67
81.94
81.67
ImageNet Oracle
96.38
96.40
96.38
RankMe
96.68
96.30
96.68
EuroSAT
LiDAR (Student)
96.68
96.30
96.68
LiDAR (Teacher)
96.68
96.30
96.68
ImageNet Oracle
79.23
79.19
79.23
RankMe
79.25
78.99
79.25
Food101
LiDAR (Student)
79.25
79.47
79.25
LiDAR (Teacher)
79.25
79.47
79.25
ImageNet Oracle
71.00
70.70
71.00
RankMe
70.29
70.40
70.29
SUN397
LiDAR (Student)
70.29
70.39
70.29
LiDAR (Teacher) 70.29
70.39
70.29
Table 29: DINO: Top-1 accuracies computed on representations when tuning hyperparameters with ImageNet validation performance, RankMe, augmented RankMe and LiDAR. Hyperparameter values are described in Appendix [
33

---

Published as a conference paper at ICLR 2024
Method
Cov
inv
LR
WD
ImageNet Oracle
77.83
77.61
76.94
76.97
RankMe
77.53
75.53
76.94
74.93
LiDAR
78.04
77.74
76.94
74.93
Table 30: VICReg: Average Top-1 probe accuracies (across OOD datasets) computed represen- tation (following VISSL protocol) when tuning hyperparameters with ImageNet validation perfor- mance.
Dataset
Method
Cov
inv
LR
WD
CIFARIO
ImageNet Oracle
88.49
88.06
87.25
87.48
RankMe
88.22
87.05
87.25
86.54
LiDAR
88.68
88.47
87.25
86.54
CIFAR100
ImageNet Oracle
69.62
69.46
68.76
69.5
RankMe
69.93
67.83
68.76
67.02
LiDAR
70.65
70.2
68.76
67.02
EuroSAT
ImageNet Oracle
95.9
95.88
95.0
95.6
RankMe
95.46
94.64
95.0
94.88
LiDAR
95.94
95.66
95.0
94.88
Food101
ImageNet Oracle
70.07
69.66
69.37
68.62
RankMe
69.43
66.34
69.37
65.35
LiDAR
70.01
69.84
69.37
65.35
SUN397
ImageNet Oracle
65.09
64.99
64.32
63.66
RankMe
64.63
61.79
64.32
60.86
LiDAR
64.94
64.51
64.32
60.86
Table 31: VICReg: Top-1 probe accuracies computed on representation (following VISSL protocol) when tuning hyperparameters with ImageNet validation performance.
eters except learning rate where LiDAR matches RankMe. Additionally, we observe that LiDAR is able to select hyperparameters that improve the performance over ImageNet Oracle in certain cases which adds to the observations made by Garrido et al.) (2022) on the need to have methods to select hyperparameters. Table 31 provides per-dataset breakdown the linear probe accuracy on the 5 downstream datasets considered in our expeirments.
10.6 SIMCLR
The SimCLR model is pretrained on ImageNet using the random hyperparameter search policy as specified in Appendix 99.] For better evaluation performance OOD datasets, use stronger augmentations during SimCLR pretraining. Specifically, besides the default augmentations, we follow the original SimCLR paper (Chen al.) (020) use extra augmentations of Sobel filtering, equalization and motion blur. We conducted probe analysis on five OOD datasets using the learning parameters listed in (Garrido et al.) 2022). p. Table 32 below summarizes the average probe accuracies for the checkpoint selected by given metric (ImageNet validation performance, RankMe, and our LiDAR) per hyperparameter. We observe that LiDAR outperforms RankMe in being able to select better hyperparameters across all settings considered in experiments. LiDAR also recovers most of the performance seen with ImageNet Oracle. Table 33 provides a per-dataset breakdown of the three methods used select hyperparameters.
34

---

Published as a conference paper at ICLR 2024
Method
Temp
LR
WD
ImageNet Oracle
80.68
80.84
80.76
RankMe
78.10
79.48
79.36
LiDAR
79.88
80.30
80.24
Table 32: SimCLR: Average Top-1 probe accuracies (across OOD datasets) computed on represen- lation when tuning hyperparameters (random search) based on ImageNet validation performance.
Dataset
Method
Temp
LR
WD
CIFARIO
ImageNet Oracle
90.51
90.83
90.07
RankMe
88.24
89.57
90.14
LiDAR
89.78
90.13
90.54
CIFAR100
ImageNet Oracle
73.52
73.47
73.38
RankMe
70.36
72.71
73.32
LiDAR
72.63
73.18
73.43
EuroSAT
ImageNet Oracle
96.27
96.51
96.54
RankMe
95.24
95.68
95.73
LiDAR
95.92
96.31
95.39
Food101
ImageNet Oracle
74.63
64.79
74.84
RankMe
71.53
72.58
71.22
LÍDAR
73.78
74.13
73.84
SUN397
ImageNet Oracle
68.47
68.59
69.03
RankMe
65.23
66,84
66.46
LiDAR
67.34
67.83
68.06
Table 33: SimCLR: Top-1 probe accuracies computed representation when tuning hyperparame- ters (random search) based on ImageNet validation performance.
35

---

Published as a conference paper at ICLR 2024
1000
1.000
0.998
Max ratio
0.996
0.994
0.992
0.990
0.988
LIDAR (Student)
LIDAR (Teacher)
0.986
60
100
Figure 23: LiDAR sensitivity to the number of samples per class hyperparameter. g is varied from 10 to 100 while n is fixed 1000 to evaluate an I-JEPA checkpoint.
11 SENSITIVITY OF LIDAR HYPERPARAMETERS
We note from Section 4.3 that the rank of LiDAR is bounded by the number of surrogate classes which in our case corresponds to the number of "clean" image samples we draw at random from the source dataset. In practice we recommend choosing value for that is greater than the length of feature vectors (p) (representations or emebddings) to avoid imposing an artificial constraint on the ranks estimated by LiDAR. The number of samples per class is another important hyperparame- ter. We conduct an ablation to study the impact of this hyperparameter with I-JEPA embeddings. Figure 23 shows the results from this ablation. As teacher and student is already close to 99% of its final value at 10 and reaches more than 99% of its final value q 50, which the value used in our experiments.
from Figure 23 LiDAR for both the
11.1 HYPERPARAMETER SELECTION
The results from Appendix IT suggest that a value of n 1000 and = 50 provides a satisfactory estimate LiDAR for I-JEPA. We copy these hyperparameters to data2vec and DINO settings as these methods produce features that are smaller than or equal to the features provided by I-JEPA. Our empirical results with these methods shown in the main text and the appendix indicate that these hyperparameters sufficient to saturate performance.
SimCLR and VICReg produce features of length 2048 that requires us to use a higher value of 5000 to ensure good estimates of the effective rank. We lower the value of g to 10 to ensure that the size of LiDAR dataset the same across all of our evaluations.
36

---

Published as a conference paper at ICLR 2024
Method
Sample count
Median time (std. dev.) (seconds)
Feature extraction (scconds)
RankMe
10K 10K (n=lK, q=
0.38 (2.38)
11.29
LiDAR (Student)
0.53 (0.07)
23.22
LiDAR (Student)
50K (n=lK, 5=50)
0.71 (0.06)
96.95
Wall-clock time to calculate effective rank for RankMe and LiDAR. The emebddings are gathered from an I-JEPA (Assran et al. 2023) checkpoint for all methods.
Table 34:
12 RUNTIME COMPARISON OF LiDAR AND RankMe
Table 34 shows the wall-clock time of RankMe and LiDAR methods. We use the checkpoint that corresponds to the ImageNet Oracle for I-JEPA and calculate RankMe and LiDAR using the above checkpoint. The feature extraction uses a batch size of 200 and uses 10K samples for RankMe and 10K for LiDAR to ensure comparison is conducted with the same number of samples and 50K as this is the operating point for LiDAR with I-JEPA. The feature extraction is done on one GPU while the metries are implemented on the CPU. We run 10 trials ensure we can obtain reasonable estimate of run-time variability. We present the median wall-clock times in Table 34 to guard against outliers while we also show the standard deviation parentheses. We observe from Table 34 that RankMe is faster than LiDAR expected. However, both methods are significantly faster than the feature extraction step, pinpointing the main place where optimizing the pipeline can be beneficial. We note here that feature extraction for LiDAR involves running multiple forward passes through predictor network for I-JEPA (Assran et al. 2023) which adds to the compute time. In this work, we use standard off-the-shelf implementations for RankMe and LiDAR. A detailed study of efficient implementation for LiDAR and its complexity analysis is left as a topic for future work. The work in (Cai al. 2008) could be relevant starting point. Notably, the running time difference between these methods is of the order of seconds/minute which is negligible compared to model training or downstream task probing that may require additional data and/or hyperparameter optimization.
37

---

Published as a conference paper at ICLR 2024
1.00
0.98
Max ratio
0.94
0.90
RankMe 99 percent
5000
10000
15000
20000
25000
Number of images
Figure 24: RankMe estimates versus the number of samples for I-JEPA.
13 RankMe ESTIMATE VERSUS NUMBER OF INPUT SAMPLES
We run an experiment with an I-JEPA AAsran et al. 2023) checkpoint to determine the beahvior of RankMe estimates function of the number of samples. We vary the number of samples from 1000 to 25600 where the former number corresponds to the number samples n used in LiDAR while 25600 samples represents the number of samples recommended by Garrido iet al. (2022). We note here that the value of 25600 is determined for 1048-dimensional vectors while I-JEPA outputs 768-dimensional vectors. We observe from Figrue [24 that we obtain a rank estimate that is greater than 99% of its maximum value by just using 10000 samples. This ablation suggests that the number of samples used to estimate RankMe is reasonable for I-JEPA and data2vec that produce feature vectors that have identical sizes.
I-JEPA:
38

---

Published as a conference paper at ICLR 2024
14 NUMBER OF SAMPLES FOR RankMe WITH 1-JEPA AND DATA2VEC REPRESENTATIONS
Hyperparameter
10000 samples (RankMe)
25600 samples (RankMe)
Learning rate
0,6835
0.6835
Weight decay
0,5040
0,5050
Target mask scale
0.2867
0.2847
Context mask scale
0.4246
0.4256
Overall
0.5830
0.5829
Table 35: I-JEPA: Kendall's 5 coefficient between effective ranks of RankMe and linear probe
curacy. RankMe estimated with 10000 samples and 25600 samples. Hyperparameters are varied via grid sampling.
Hyperparameter
10000 samples (RankMe)
25600 samples
(RankMe)
Overall
0.8314
0.8308
Table 36: I-JEPA: Kendall's 5 coefficient between effective ranks of RankMe and linear probe curacy. RankMe estimated with 10000 samples and 25600 samples. Hyperparameters are varied via random sampling.
Hyperparameter
10000 samples (RankMe)
25600 samples (RankMe)
Learning rate
0.2410
0.2392
Mask ratio
0.2683
0.2706
Overall
0.2238
0.2216
Table 37: data2vec: Kendall's coefficient between effective ranks of RankMe and linear probe accuracy. RankMe estimated with 10000 samples and 25600 samples. Hyperparameters are varied via grid sampling.
Our analysis in Appendix 13 suggests that using 10000 samples may be sufficient to obtain highly quality of estimate of the effective rank of the embeddings, i.e., RankMe. Garrido et al. (2022) suggest using 25600 samples based on their analysis of with 2048-dimensional vectors. We study the impact of the number of samples on RankMe as it pertains to the correlation between RankMe and linear probing performance. We test the behavior RankMe with 10000 and 25600 samples and calculate the correlation between RankMe linear probe accuracy. Table 35 Table 36 and Ta- ble 37 show the results of this analysis for I-JEPA with hyperparameters set via grid search, I-JEPA wtih hyperparameters set via random search and data2vec with hyperparameters set via grid search respectively. The Kendall's 7 correlation coefficient shows minor differences and the linear probe accuracy recovered for these models for in-domain data is identical (Table 3 Table 4 Table 6 and the performance on OOD data is also identical (aceuracies available in Appendix T0p as both esti- mates of RankMe recover identical checkpoints. Figure 25] Figure 26 and Figure 27| show scatter plot of linear probe accruacy versus RankMe. We observe that while there are minor differences in the effective ranks calculated with the number of input samples, the overall shape of the distribution that is also captured by Kendall's-- s-T are very similar which is also reflected in the data presented in the tables above. This ablation suggests that the predictive performance of RankMe on I-JEPA and data2vec representations is stable once with a sufficiently large number of samples.
39

---

Published as a conference paper at ICLR 2024
60
probe
30
10
10000 samples
25600 samples
100
200
300
400
500
RankMe
Figure 25: I-JEPA: Effect of the number of samples on RankMe, Parameters set via grid search.
10000 samples
60
25600 samples
40
20
300
RankMe
Figure 26: I-JEPA: Effect of the number of samples RankMe. Parameters set via random search.
40
30
probe
20
10000 samples
25600 samples
100
200
300
400
500
RankMe
Figure 27: data2vec: Effect of the number of samples on RankMe. Parameters set via grid search.
40


Reviews:
[{"review_id": "3sJviFnc5B", "reviewer": "Reviewer_bVwZ", "summary": "This paper proposes a new metric for the measurement of the representation quality within joint embedding architectures, which is called Linear Discriminat Analysis Rank.\nThis metric discriminate between informative and uninformative features by quantifying the rank of the Linear Discriminant Analysis matrix.\nThe experiments on several downstream tasks show the proposed metric improves the performance.", "questions": "Actually, I have no background of the topics of the paper. I cannot assess and ask any questions on this paper. I do not know why this paper is assigned to me. I think the reason maybe the abbreviation of the proposed method \"LiDAR\" is related with my previous paper, however, this paper is not related with lidar at all.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper generally explain their motivation and inspiration clearly.\n2. The experiments are sufficient to show the effectiveness of the proposed method on the downstream tasks.\n3. The supplementary material shows the details of the proposed method.", "weaknesses": "1. The abbreviation of the proposed method \"LiDAR\" is irrelavent of the problem it tries to solve. The irrelavent abbreviation misleads the readers what the paper want to express. Some readers may think the paper is on the lidar.\n\n2. Some typos:\n\na. In section 2.1: In practice, a set of downstream tasks {T_j} are used to asses ->  In practice, a set of downstream tasks {T_j} are used to assess\n\nb. in Section 4.0.1: There is a blank line, where a sentance seem missed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bd8dCj8PYd", "reviewer": "Reviewer_y3Lm", "summary": "The authors present LiDAR, a new metric designed to assess the quality of self-supervised representations in Joint Embedding (JE) architectures using the linear probing protocol, without requiring labeled downstream tasks. LiDAR employs Linear Discriminant Analysis (LDA) to evaluate dimensional collapse and implicitly integrates the SSL objective. Through thorough empirical validation, the authors demonstrate LiDAR's superiority over existing metrics in predicting optimal hyperparameters.", "questions": "1) Scatter matrices, $\\Sigma_w$ and $\\Sigma_b$, having dimensions ( p $\\times$ p ), are influenced by both n and the data dimensionality (p). In Section 4.1, the authors note they \"maintain a total of 50 features\" for VICreg. How does this align with rank constraints dictated by the data dimensionality?\n\n2) What guidance can be provided for practitioners to select appropriate values for the number of surrogate classes (n) and the number of samples per class (q)?\n\n3) The authors re-implemented RankMe (without data augmentation) for new architectures, namely I-JEPA and data2vec, using 10k images for rank estimation. This contrasts with the originally suggested 25.6k images. Meanwhile, LiDAR employs at least 50k total samples. What motivated this choice?\n\nThe rebuttal addressed the raised issues well. Therefore, I updated the review.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The addressed problem holds significance in self-supervised learning. By evaluating SSL representations without labeled tasks, improvements in hyperparameter selection and algorithm development are facilitated.\n\n2) Utilizing Linear Discriminant Analysis (LDA) to assess the dimensionality collapse of SSL representations is both theoretically grounded and considerably novel. While RankMe assessed SSL methods via the effective rank of the covariance matrix (related to PCA), LiDAR focuses on the rank of the scatter-ratio matrix through LDA.\n\n3) Detailed experiments highlight a significant and consistent positive correlation (measured by Spearman and Kendall coefficients) between LiDAR and linear probing performance on the same source-target datasets, surpassing RankMe across most settings.", "weaknesses": "1) The manuscript lacks evaluation on out-of-distribution (OOD) target datasets. Although the ImageNet dataset results are promising, the evidence for LiDAR's effectiveness in more realistic downstream tasks, often involving OOD datasets, is missing. Comparing with datasets like iNaturalist-2018 or Stanford Cars, following the protocol in RankMe, would strengthen LiDAR's position as a useful proxy metric.\n\n2) LiDAR introduces two hyperparameters: the number of surrogate classes (n) and the number of samples per class (q). Yet, the manuscript does not provide guidance on determining these values. Given that these values differ across methods, a hyperparameter sensitivity analysis seems essential.\n\n3) The comparative analysis appears limited, with LiDAR mainly being evaluated against RankMe. Other potential baselines, like $\\alpha$-ReQ mentioned in related work, are overlooked.\n\n4) In Section 4.1, the authors advocate for dimensionality reduction (DR) of embedding features, possibly adding another layer of dependency for LiDAR evaluation. The necessity of DR for LiDAR's effective application, and any specific DR algorithms used in experiments, remain ambiguous.\n\n5) Although the authors claim LiDAR integrates the SSL objective, the manuscript does not delve into the relationship between LiDAR and different SSL losses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iI31Lf9KzE", "reviewer": "Reviewer_d3VV", "summary": "This paper introduces LiDAR (Linear Discriminant Analysis Rank) as a novel  metric for assessing the quality of representations within joint embedding (JE) architectures. The authors conduct comprehensive experiments and demonstrate that their proposed LiDAR metric correlates significantly and consistently higher with downstream linear probing performance than RankMe. They further show that LiDAR demonstrates consistently strong performance in hyperparameter selection, outperforming RankMe.", "questions": "None.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed LiDAR metric for accessing the quality of representations within joint embedding (JE) architectures sounds novel.\n- The theoretical motivation for proposing the LiDAR metric is clear and logical.\n- Comprehensive experiments have been carried out to demonstrate the superiority of LiDAR over RankMe.\n- The paper is clear and easy to follow.", "weaknesses": "- No disucssions on computational overhead and runtime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)