RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank
Authors: Quentin Garrido
Randall Balestriero
Laurent Najman
Yann LeCun
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
evaluation
dimensional collapse
hyperparameter selection

Abstract: Joint-Embedding Self Supervised Learning (JE-SSL) has seen a rapid development, with the emergence of many method variations and few principled guidelines that would help practitioners to successfully deploy those methods. The main reason for that pitfall actually comes from JE-SSL's core principle of not employing any input reconstruction. Without any visual clue, it becomes extremely cryptic to judge the quality of a learned representation without having access to a labelled dataset. We hope to correct those limitations by providing a single --theoretically motivated-- criterion that reflects the quality of learned JE-SSL representations: their effective rank. Albeit simple and computationally friendly, this method ---coined {\em RankMe}--- allows one to assess the performance of JE-SSL representations, even on different downstream datasets, without requiring any labels, training or parameters to tune. Through thorough empirical experiments involving hundreds of repeated training episodes, we demonstrate how RankMe can be used for hyperparameter selection with nearly no loss in final performance compared to the current selection method that involve dataset labels. We hope that RankMe will facilitate the use of JE-SSL in domains with little or no labeled data.

Under review as a conference paper at ICLR 2023
ASSESSING THE DOWNSTREAM PERFOR- MANCE OF PRETRAINED SELF-SUPERVISED REPRESEN- TATIONS BY THEIR RANK
RANKME:
Anonymous authors Paper under double-blind review
ABSTRACT
Joint-Embedding Self Supervised Learning (JE-SSL) has seen rapid development, with the emergence of many method variations and few principled guidelines that would help practitioners to successfully deploy those methods. The main reason for that pitfall actually comes from JE-SSL's core principle of not employing any input reconstruction. Without any visual clue, it becomes extremely cryptic to judge the quality of a learned representation without having access to a labelled dataset. We hope correct those limitations by providing single -theoretically motivated- criterion that reflects the quality of learned JE-SSL representations:: their effective rank. Albeit simple and computationally friendly, this method -coined RankMe- allows one to assess the performance of JE-SSL representations, even on different downstream datasets, without requiring any labels, training or parameters to tune. Through thorough empirical experiments involving hundreds of repeated training episodes, we demonstrate how Ma can be used for hyperparameter selection with nearly no loss in final performance compared to the current selection method that involves dataset labels. We hope that RankMe will facilitate the use of JE-SSL in domains with little no labeled data.
INTRODUCTION
Self-supervised learning (SSL) has shown great progress to learn informative data representations
in recent years (Chen et al. 2020a; He et al. [2020] Chen et al.
2020b
Grill et /2.20 Lee et al.
2021
Caron et al.] 2020 Zbontar et al.
2021 Bardes et al.
2021
lomasev al.] [022] Caron
et al.
2021 Chen et al., 2021: Li et al.
2022a Zhou et al.
2022a by HaoChen et al.] 2021 He
al.
2022). catching up to supervised baselines and even surpassing them in few-shot learning. i.e.
when evaluating the SSL model from only a few labeled examples. Although various SSL families of losses have emerged, most are variants of the joint-embedding (JE) framework with a siamese JE-SSL for short. The only technicality
network architecture (Bromley et al. 1994,, denoted ought to introduce to make our study precise is the fact that JE-SSL has introduced some different notations to denote input's representation. In short, JE-SSL often composes backbone encoder network e.g., a ResNet50 and a projector network e.g., a multilayer perceptron. This projector is only employed during training. and we refer its outputs embeddings, while the actual inputs' representation employed for downstream tasks are obtained at the encoder's output.
Although downstream tasks performance of JE-SSL representations might seem impressive, one pon- dering fact should be noted: all existing methods, hyperparameters, models - and thus performance of JE-SSL are obtained by ad-hoc manual search involving the labels of the training samples. In words, JE-SSL is tuned by monitoring the supervised performance of the model at hand. Hence, although labels are not directly employed to compute the weight updates, they are used as a proxy to signal the JE-SSL designer deployment of JE-SSL in challenging domains where the number of available labelled examples is limited and such search can not be performed. Adding to the challenge, one milestone of JE-SSL is to move away from reconstruction based learning; hence without labels and without visual cues, tuning JE-SSL methods feature inversion methods e.g. Deep Image Prior (Ulyanov et al. 2018)) or conditional diffusion models (Bordes et al. 2021) to be deployed onto learned JE-SSL representation to try visualize the
how to refine their method. This single limitation prevents the
unlabeled datasets remains challenging. This led to the application of

---

Under review as a conference paper at ICLR 2023
learned features. This first step towards removing the need for labels has seen some success but is doomed by the computational complexity of the methods, and their biases towards natural images i.e. it is not clear how such methods would perform on different data modalities.
In this study we propose RankMe, which is able to assess a model's performance without having access to any labels and without requiring any training or tuning. RankMe accurately predicts model's performance both on In-Distribution (ID), i.e., same data distribution as used during the JE-SSL training, and on Out-Or-Distribttion (OOD), i.e., different data distribution scenarios. We highlight this property at the top of fig. T The strength of RankMe lies in the fact that it is solely based on the singular values distribution of the learned embeddings, and thus does not rely on any parameters that need training, nor requires any ID/OOD labels. In fact, RankMe's motivation hinges on Cover's 's theorem (Cover 1965) that states how increasing the rank of a linear classifier's input increases its training performance, and three simple hypotheses that we summarize below and thoroughly validate empirically. As such, RankMe provides a step towards (unlabeled) JE-SSI. by allowing practitioners to cross-validate hyperparameters and select models without resorting to labels or feature inversion methods. We hope that RankMe will enable JE-SSL to be deployed even in challenging domains that possess
little labelled data; we summarize our contributions below:
1. We introduce (eq. ( and motivate RankMe which combines Cover's theorem with the following three key hypotheses:
(H1) increasing training performance increases testing performance on both representations and embeddings i.e. no over-fitting is observed from the (non)linear probe, validated empirically in the bottom left of fig. 2
(H2) embeddings' rank scale linearly between datasets. Assuming a pretraining on the same dataset, if set of embeddings has greater rank than another on a dataset, it also holds on another one, validated empirically in the top row of fig. 2
(H3) increasing embeddings performance increases representations performance, validated
empirically in the bottom right of fig. 2
concluding that embeddings with greater rank will have greater train performance (Cover's theorem) and test performance (HI) on ID and OOD cases (H2), even before the projector (H3).
2. We demonstrate that RankMe's ' ability to assess JE-SSL downstream performance is robust across methods, e.g. VICReg, SimCLR, and their variants, and is also robust to architecture changes, e.g. using a projector network and/or a nonlinear evaluation method (see fig. 3 and section (.3))
3. We demonstrate that RankMe enables hyperparameter cross-vas validation for any given JE-SSL method; RankMe is able retrieve and sometimes surpass most of the performance previously found by manual search using labels, on both in domain and out of domain datasets; see bottom of fig. and table
We provide a hyperparameter free numerically stable implementation of RankMe in section 3.1 and pseudo-code for cross-validation in fig. []] Through extensive experiments involving 11 different datasets and more than 85 trained models over 4 methods, we demonstrate that in the linear and nonlinear probing regime, RankMe is able tell apart high and low performing models, even on different downstream tasks without having access to labels or downstream task data samples.
RELATED WORKS
Joint embedding self-supervised learning (JE-SSL). In JE-SSL, two main families of method can be distinguished: contrastive and non-contrastive. Contrastive methods (Chen et al.) [2020a] He et al.
2020
Chen et al., 20206 2021 Yeh et al. 2021) mostly rely on the InfoNCE criterion (Oord et al. except for HaoChen et al. (2021 which uses squared similarities between the embedding.
2018
clustering variant of contrastive learning has also emerged (Caron et al. 2018 2020: 202T) and can be thought of as contrastive methods, but between cluster centroids instead of samples. Non-contrastive Chen & He 2020, Caron et al., 2021 Bardes et al. 2021 Zbontar et al.) 2021 Li et al. 2022b) aim at bringing together embeddings of positive samples,
methods (Grill et al. 2020
2021 Ermolov et al.
similar to contrastive learning. However, a key difference with contrastive learning lies in how those methods prevent a representational collapse. In the former, the criterion explicitly pushes away negative samples, i.... all samples that not positive, from each other. In the latter, the criterion does not prevent collapse by distinguishing positive and negative samples, but instead considers the embeddings as a whole and encourages information content maximization e.g., by regularizing
2

---

Under review as a conference paper at ICLR 2023
VICReg
ViCReg-exp
VICReg-dtr
SumCLR
70.0
ImageNet-11 1k
NNturrllstt- 18
SUN397
67.5
38
rop 1 regresontatonn
65.0
36
62.5
34
66
60.0
32
57.5
30
55.0g
28
62
128
512
2048
128
512
2048
128
512
2048
Embedding rank ImageNet
Embedding rank ImageNet
Embodding rank ImageNet
VICReg
SimCLR
Dataset
Method
Labels
COv.
inv.
LR
WD
temp.
LR.
WD.
ImageNet Oracle
V
68.2
68.2
68.6
68.0
68.5
68.5
68.3
ImageNet
a-ReQ
X
67.9
67.5
59.5
67.8
63.5
68.1
32.3
RankMe
X
67.8
67.9
68.2
67.8
67.1
68.0
68.3
ImageNet Oracle
V 67.0
67.0
67.2
67.1
67.0
67.0
67.1
OOD
a-ReQ
X 66.4
66.0
62.0
66.7
63.3
66.4
66.8
RankMe
X
65.9
66.6
67.0
66.7
65.8
66.7
67.1
Figure (Top) Performance of JE-SSL representations (encoder output) in y-axis against the embeddings (projector output) RankMe values in x-axis on ImageNct-1k, lk. Except for some degenerate solutions at full- rank, RankMe values correlate well with in-distribution (left column) and out-of-distribution (right columns) classification performance. (Bottom) Hyperparameter selection using the common supervised linear probe strategy and proposed unsupervised RankMe strategy. OOD indicates the average performance over all the considered dataset other than ImageNet. Without any label, optimization or parameters, RankMe is able to recover most of the performance obtained by using ImageNet validation set, highlighting its strength as hyperparameter selection tool. RankMe also outperforms a-ReQ average and does not suffer from big performance drops worst cases.
the empirical covariance matrix of the embeddings. Such a categorization is not needed for our development, and we thus refer to any of the above method as JE-SSL.
Dimensional collapse in JE-SSL. The phenomenon of learning rank-deficient embeddings, or dimensional collapse, in JE-SSL has recently been studied from both a theoretical and empirical point of view. The empirical emergence of dimensional collapse was studied in Hua et al. (2021) where they proposed the use of whitening batch normalization layer to help alleviate et al. (2022). focus contrastive approaches in a linear setting enabled better understanding of dimensional collapse and the role of augmentations in its emergence. Performance a low label regime of partially collapsed encoder can also be improved by forcing the whitening of its output, as shown in He Ozay (022). Furthermore, was shown in Balestricro & LeCun (2022) phenomenon that should not necessarily happen in theory and how its emergence is mostly due to practical concerns. Interestingly, we will see through the lens of RankMe that while reducing dimensional collapse is often beneficial, doing "at all cost" can lead to degenerate solutions. The collapse induced from training using a softmax layer was also studied Ganea et al. (2019), where they show that high rank embeddings desirable.
Jing
how dimensional collapse
Evaluation of JE-SLL representations. Evaluating the representations learned by JE-SSL methods is fundamental to enable the optimal selection of those methods' hyperparameters. which are numerous. Yet, due to the imprecise nature of what makes good representation, multiple strategies have emerged which evaluate different properties of representations. The most common approach relies on the strong assumption of having labels on the dataset where the JE-SSL method is trained on. In this case, on trains a linear classifier on the JE-SSL representations (Misra & Maaten 2020) and directly use the test accuracy compare models. This method was extended to the use of nonlinear classifiers, e.g., a je-nn classifier (Wu et al., 2018 Zhuang et al. 2019). Performance evaluation without labels can also be done using pretext-task, such rotation prediction. This technique helped in selecting data augmentation policies in Reed et al. (00T). One limitation lies the need to select and train the classifier of the pretext-task, and the strong assumption that rotation were
3

---

Under review as a conference paper at ICLR 2023
not part of the transformations aimed to be invariant to. Since (supervised) linear evaluation is the most widely used evaluation method, we will focus on showing how RankMe compares with Most related to us is Ghosh al. (2022) where representations evaluated by their eigenspectrum decay, giving a baseline for unsupervised hyperparameter selection.
3 REPRESENTATIONS RANK CORRELATE WITH DOWNSTREAM PERFORMANCE ACROSS TASKS AND MODELS
The goal of this section is to formally introduce and motivate RankMe while providing a numerically stable implementation (section 3.1). The construction of RankMe hinges on three hypotheses that validate empirically throughout this section.
3.1 RANKME: FROM THEORY TO IMPLEMENTATION
We first want to build notations and intuition into the construction of RankMe. To that hand, we first quantify approximation and classification errors of learned embeddings function of their rank, and then motivate how embeddings rank can be sufficient to compare test performance of JE-SSL models's 's representations. This criterion should however only be used compare different runs of given method, since the embeddings' rank is not the only factor that affects performance. To ease notations, we refer the (train) dataset used to obtain the JE-SSL model the source dataset, and the test set on the same dataset or a different OOD dataset as the target dataset.
From Source Embeddings's Rank to Target Representations performance. We first build some intuition in the regression settings. In this casc, a common linear algebra result ties the best-case and worst-case approximation error of any target matrix Y E R. R.NCC from a rank- R matrix E R NxC to the singular values of Y that run from R to the rank of Y when ordered in decreasing order. Without loss of generality, we only consider the case > in this study, i.e., we have more samples than dimensions. Formally, this provides lower bound
Y PI
Σ (M, r=
which is tight for P of rank R, and with o the operator returning the 5thh singular value of its argument, ordered in decreasing order. This result, on which RankMe relies on, demonstrates that a necessary (but not sufficient) condition for approximation P to well approximate Y is to have least the same rank as Y. A similar result can be obtained in classification by considering multiple one-vs-all classifiers. In practice, however, we commonly employ linear probe network on top of given embeddings Z to best adapt them to the target Y, i.e., P = ZW + 16". However, linear transformation is not able increase the rank of the input matrix since
rank (P) S min((arr)) Z), rank W)) +
We directly obtain that minw b |Y Zw 3w-111 11 2 E E (Y). In short, the approximation lower bound is not improved by allowing linear transformation of the embeddings. Further supporting the above, we ought recall Cover's 's theorem (Cover 1965) stating that the probability of a randomly labeled set of points being linearly separable only increases if N is reduced or Rt is increased. We formalize those results below.
Proposition 1. The maximum training accuracy of given embeddings in linear regression or classifi- cation increases with their rank. For classification, it plateaus when the rank surpasses the number of classes.
We thus introduce RankMe formally the following smooth rank measure, originally introduced in Roy & Vetterli 2007).
K,)(N,K)
, Pk = Tk (Z) [Z)
RankMe( Z) = exp
Σ
Pk log
+
(I)
where Z is the source dataset's embeddings. By noticing that RankMe provides a smooth measure of the embeddings" rank (more details in the implementation section) lean proposition To
4

---

Under review as a conference paper at ICLR 2023
that given two models, the one with greater RankMe value will have greater training performance. This is only guaranteed for different models of the same method, since embedding rank is not necessarily the only factor that affects performance.
The above result is however not too practical yet since what we are truly interested in are (i) performance unseen samples, i.e., on the test set and out-of-distribution tasks, and (ii) performance on the representations and not the embeddings since it is common to ablate the projector network of JE-SSL models. Below, we validate three key hypotheses which, when verified, imply that we can extend the impact of RankMe such that (OOD) test performance of JE-SSL representations are increased when RankMe's value on their train set embeddings is increased.
Validating RankMe's Hypotheses The development of RankMe is theoretically grounded when it comes to guaranteeing improved source dataset embeddings performance. To empirically extend to target dataset representations performance we need to verify three hypotheses: (i) linear probes do not overfit, (ii) embeddings and representations performance are monotonically linked, and (iii) source and (OOD) target embeddings ranks are monotonically linked. Due the different nature of datasets used for downstream tasks, there is no inherent reason for the rank of embeddings to transfer in monotonic way to them. However, if the source dataset is diverse enough and target datasets have some semantic overlap with the source dataset, then we have
rank Zearget) X rank sourcee)
We observe in section 3.2 and fig. 2 that the rank of JE-SSL representations scales linearly between different input distributions e.g. going from a source task such as Imagenet (Deng et al. 2009) to a targer task such as iNaturalist. This is further confirmed by Pearson correlation coefficients greater than 0.99, except for StanfordCars where is 0.88. Interestingly, we observe that the StanfordCars dataset suffers from less distinctive linear scaling due to the dataset distribution having small overlap with ImageNet. This indicates that as long as the source dataset is relatively diverse, then using RankMe select model with greater embeddings rank will also correspond selecting model with greater embeddings' rank on the target dataset. Furthermore, as the train performance increases, so does the test performance. We validate this in 2 As a result, using MankMe to select a model with greater train performance is enough to also select model with greater test performance.
Finally, we report in fig. 2 that the performance of embeddings and representations scales almost monotonically. These results are supported by visualization of representations and embeddings from feature inversion models (Bordes et al. 2021). Hence, using RankMe to select the model maximizing the performance on the former also selects a model maximizing performance on the latter.
With these three hypotheses validated empirically, we can confidently say that RankMe computed on the embeddings of the source dataset is a predictor of representations' performance on target datasets.
Robust RankMe Implementation. One of the most erucial step of RankMe is the estimation of the embeddings' rank. A trivial solution could be to check at the number of nonzero singu- lar values. Denoting by a the hth singular value of the (N K) embedding matrix Z, this would lead to rank = >minn min( 1 (0,>0)- However, such a definition is too rigid for prac- tical scenarios. For example, round-off error alone could have a dramatic impact on the rank estimate. Instead, alternative and robust rank definitions have emerged (Press et al. 2007) such as rank Z) Aiia(N 1 [0 o, x (M.. Xee] where is small constant dependent on the data type, typically 10-7 for float32.
An alternative measure of rank comes from a probability viewpoint where the singular values are normalized to sum to 1 and the Shannon Entropy (Shannon 1948) is used, which corresponds to our definition of RankMe from eq. (I.. As opposed to the classical rank, the chosen eq. (T) does not rely on specifying the exact threshold at which the singular value is treated as nonzero. Throughout our study, we employ eq. (1) and provide the matching analysis with the classical rank in the appendix. Another benefit of RankMe's s eq. (I) is in its quantification of the whitening of the embeddings in addition to their rank, which is known to simplify optimization of (non)linear probes put on top of them (Santurkar& et al., 2018). Lastly, although eq. (I) is defined with the full embedding matrix Z. observe that not all the samples need to be used to have an accurate estimate of RankMe. In practice, we use 25600 samples as ablation studies provided in appendix G and fig. [SIT that this provides highly accurate estimate.
5

---

Under review as a conference paper at ICLR 2023
Embodding rank Tar larget
2048
0.53 Ringeli +
2048
Aroco
0.63 Amoupate +
2048
0.55 Ronapote
+
2048
Rim
0.43 Aerautte!
512
512
512
512
128
128
Vidling Vidlieg up di StniLL
32
22
8 32 128 512 2018
8 32 126 512 2046
8
32 128
512 2018
32 128 512 2048
Embedding rank ImageNet
Embodding rank ImageNet
Embodding rank ImageNet
Embedding rank ImageNat
100
ImageNet
Thei reprosnnttton top-1
iNad18
Top-1 repressotaaiions
60
80
SUN397
Care
50
pedaae
60
ImageNet
INat
40
Placee
30
30
StantordCars
Train representation top-1
Top-1 embeddings
Figure 2: Validation of the hypotheses motivating RankMe. (Top) Embeddings rank transfers from source target datasets. The estimates use 25600 images from the respective datasets. dBotsots,(Bottom Left) Train and test accuracy are highly correlated across datascts. (Bottom Right) An increase performance on embeddings leads to an increase in performance on representations.
601 Tmmaagn Nert
20
|Nat18
Top-1 regonaen at allkas
ImagaNat
iNat18
embedaiag
35
30
Milleg
10
80
AtCRop Mtokie FerriA
20
SevcLk
10
32
128
512
2068
128
512
2048
50
32
128
512
2008
22
128
512
2048
ding ImageNet
ImageNat
deing
ImaçeNet
ImageNot
enbeddinga
Places205
EuroSal
Top- Meneenttaaion
122
Places205
EuroSel
95
50
30
80
48
70
95.5
20
52
128
512
2048
125
512
2038
125
512
2048
22
128
512
2048
Embedddnng runk ImageNNet
Embeiding ronk ImageNet
Embedking ronk ImageNNtt
Embeddding runk ImageNNt
60F SUN397
Cars
Tojp- representatiin tenns
SUN397
Cara
andocddig
20
666
60-
10
64
45
30
32
126
51."
2010
512 Embedding rank ImageNet
32
128
2038
128
312
2000
11
32
120
512
2048
Embedding runks ImageNet
Embedding mank ImageNNt
Embedding runk ImageNet
Figure 3: (Left) Validation RankMe on embeddings, a higher ImageNet rank leads improved performance across methods and datasets. (Right) Validation of RankMe on representations, where the link is even clearer, reinforcing RankMe's ' practical use.
3.2 RANKME PREDICTS LINEAR PROBING PERFORMANCE EVEN ON UNSEEN DATASETS
In order to empirically validate RankMe, compare it lincar evaluation, which is the default evaluation method of JE-SSL methods. Finetuning has gained in popularity with Masked Image Modeling methods, but this have significant impact the properties of the embeddings and alters what was learned during the pretraining. As such, we do not focus on this evaluation.
Experimental Methods and Datasets Considered In order provide meaningful assessment of the embeddings rank's 's impact on performance, we focus on JE-SSL methods. We use SimCLR representative contrastive method, VICReg representative covariance based method, and

---

Under review as a conference paper at ICLR 2023
VICReg-exp exp and VICReg-ctt ctr which were introduced in Garrido et al. (2022). To make work self-contained, we present the methods in appendix [A] We chose to use ViCReg-exp and VICReg-ct -tr as they provide small modifications to VICReg and SimCLR while producing embeddings with different rank properties. For each method we vary parameters that directly influence the rank of the embeddings, whether it is the temperature use in softmax based methods, which directly impacts the hardness of the softmax, or the loss weights to give more or less importance to the regularizing aspect of loss functions. We also vary optimization parameters such as the learning rate and weight decay to provide more complete analysis. We provide the hyperparameters used for all experiments in a All approaches were trained in the same experimental setting with a ResNet-50 (He et al.) 2016) backbone with a MLP projector having intermediate layers of size 8192, 8192, 2048, which avoids any architectural rank constraints. The models were trained for 100 epochs on ImageNet with the LARS YYou et al., 2017: Goyal et al. (2017) optimizer.
In order to evaluate the methods, we used ImageNet (our source dataset), as well as iNatural- ist18 (Horn et al., 2018), Places205 (Zhou et al. 2014), EuroSat (Helber et al. 2019). SUN397 (Xiao) et al. 2010) and StanfordCars (Krause et al.) 2013) to evaluate the trained models on unseen datasets. These commonly used datasets provide wide range of scenarios that differ from ImageNet and provide meaningful ways to test the robustness of RankMe. For example, iNaturalist consists of 8412 classes focused fauna and flora which requires more granularity than similar classes on Ima- geNet, SUN397 focuses on scene understanding, deviating from the single object and cejerc-centtic images of ImageNet, and EuroSat consists of satellite images which again differ from ImageNet. Datasets such as iNaturalist can allow theoretical limitations to manifest themselves more clearly due to the number of classes being significantly higher than the rank of learned representations. While we focus on these datasets for our visualizations, we also include CIFARIO 10, CIFAR 10FAR100PP Krizhevsky et al. (2009), Food101 [Bossadd et al. (2014), VOC07 [veringham et al.] and CLVR-count Johnson Johhnnoonn et al. (2017) for our hyperparameter selection results, and provide visualizations in appendix D In order to evaluate on those datasets, we relied on the VISSL library (Goyal et al., 2021). We provide complete details on the pretrainings and evaluations in appendix [I]
RankMe as a prediction of linear classification accuracy. As can be seen in fig. 3 for a given method the performance the embedding is improved by with higher embedding rank, whether we look on ImageNet on which the models were pretrained or on downstream datasets. Nonetheless, there are some visible outliers, but they mostly SimCLR in settings with very high error rates compared to before the projector, such as in iNaturalist or StanfordCars. The conceptual closeness between VICReg-ctr and SimCLR pointed out in Garrido et al. (2022) would also suggest that these results need to be interpreted carefully, but they do reinforce the fact that a high rank is necessary and not sufficient condition for improved performance on downstream tasks. It is also very tempting to draw conclusions when comparing different approaches, especially when looking at the ImageNet performance, however since dimensional collapse not the only performance deciding factor one should refrain from doing so. The link between embedding rank and performance is even clearer when evaluating on the representations, as is usually done. In this scenario the link is more consistent across datasets, where we observe again that a higher rank is necessary for improved performance. This solidifies the use of RankMe as a performance metric that can be used in practice.
3.3 GOING FURTHER: RANKME ALSO HOLDS FOR NONLINEAR PROBING AND FOR DIFFERENT ARCHITECTURES
Non-linear evaluation. While we have been focusing on linear evaluation, one can wonder if the behaviour changes when using a more complex task-related head. We thus give some evidence that the previously observed behaviours are similar with non-linear classification head. We used a simple 3 layer MLP with intermediate dimensions 2048, where each layer is followed by a ReLU activation. This choice of dimensions ensures that there are no architectural rank constraints on the embeddings. We focused on SUN397 and StanfordCars for this study due to their conceptual differences to ImageNet. The low rank of embeddings produced by SimCLR on these datasets would suggest that a non-linear classifier might help improve performance, since it is not as theoretically limited by the embeddings" rank it is in the linear setting. However, as behaviors for all methods is the same as in the linear regime. This would suggest that RankMe is also suitable metric to evaluate downstream performance in non-linear setting.
see in fig. 4 the
Dimensional collapse on different architectures. Our results so far have only focused on ResNet- 50s, and concern could be that the architecture played significant role in the introduction of
7

---

Under review as a conference paper at ICLR 2023
SUN397
Cars
60
represotaii
50
55
20
Topa 10
(10be
30
SaaNab-0 Com-Nest-Tiyy
32
128
512
2045
125 5(2 2048
128 256 512 1024 2048 Embedding ranks Image
20
128 256 512 1024 2048 Embedding ImugeNet
Embedding ImagoNet
Embedding ImageNet
Figure Impact rank on performance on other architectures and evaluation protocols. (Left) Using 3 layer MLP as classification head does not alter the performance before or after the projector, showing that RankMe can go beyond linear evaluation. (Right) ConvNexts are also sensitive to dimensional collapse, showing that rank-deficiency is not artifact of ResNets.
Temperaturg
Algorithm 1 Hyperparameter selection with RankMe
0.1
0.2
0.4
07.1*
68.5
68.5
Require: Models fi..... fN compare, in increasing value of the hyperparameter
300
68.0
Require: Corresponding ranks "1,..",""
200
1: fbest f- f1. Thest r1
Rank
2: for i = 2 do
3: if > Tbest then
100
4: fbest + Si. Thest f- ri
5: else if Tbest and (r; > ri-1 73 > 1141) then
6:
Sbest & Sú, Thest f- Ti
0.2
0.4
0.6
0.8
7:
return Jbest
Base LR
Figure 5: (Left) Algorithm describing how to use RankMe for hyperparameter selection. We select either the highest rank model, if there are multiple ones, the one with the minimal/maxinal value achieving (Right) Visual example of the hyperparameter selection applied SimCLR's temperature and learning rate. The star indicates the value that selected using RankMe, and the triangle one with the ImageNet oracle. Notice high rank oracle selected models.
collapse. As such, we trained VICReg in the same setting before but using ConvNext-T (Liu et al. 2022) as backbone architecture. As we can see in fig. 4 collapse still appears in this case, with stronger impact performance ImageNet. This reinforces the findings of Jing al. (2022) He & Ozay| 2002)). which study collapse through the used loss function independently of the architecture of the backbone.
RankMe in more diverse settings. While our focus has been on contrastive methods, we further study in appendix how RankMe be applied clustering methods such DINO, where it shows great effectiveness. We also take a look at the effectiveness of RankMe when pretraining on other source datasets in appendix B validating RankMe |Naturalist 8 pretraining.
RANKME FOR LABEL-FREE HYPERPARAMETER SELECTION IN SSL
We previously focused on validating RankMe by comparing overall performance compared to linear evaluation. In this section we focus on the evolution rank and performance when varying one hyperparameter at time in order to demonstrate how RankMe can be used for hyperparameter selection. We focus on loss specific hyperparameters such the loss weights or temperature as well as hyperparameters related to optimization, such as the learning rate and weight decay.
4.1 USING RANKME TO CHOOSE CORRECT HYPERPARAMETER VALUE
As we have shown before, having a higher rank is necessary for better performance, and using RankMe to find the best value an hyperparameter as simple as choosing the value that leads to the highest rank, as illustrated in fig. 5 Certain hyperparameters will lead to plateaus of equal rank, and in those the value that first achieves the maximal value should be selected. This second part is however only applicable when hyperparameter values be ordered.
Even in cases where the values cannot be compared, and equal ranks are found a different setting, this still makes it possible to discard some runs and only focus the one that achieve the maximal
8

---

Under review as a conference paper at ICLR 2023
Table I: Top-1 accuracies obtained by doing hyperparameter selection using ImageNet validation performance, a-ReQ or RankMe. OOD indicates the average performance over all the considered datasets other than ImageNet. The performance is computed on the embeddings.
VICReg
SimCLR
Dataset
Method
COv.
inv.
LR
WD
temp.
LR.
WD.
ImageNet Oracle
59.7
59.7
59.7
59.7
56.9
56.9
57.1
ImageNet
a-ReQ
59.6
59.2
36.2
59.3
51.5
56.4
49.0
RankMe
59.6
59.7
59.7
59.5
56.5
56.0
57.1
ImageNet Oracle
52.1
52.5
52.1
52.3
51,4
51.4
51.5
OOD
a-ReQ
52.3
52.5
44.2
51.9
54.2
51.3
51.5
RankMe
52.3
52.5
52.1
51.8
53,4
51.1
51.5
rank. This further highlight how maximal rank is only a necessary condition for good performance. Nonetheless, when the hyperparameters are ordered we can go one step further and use the rank alone to find good hyperparameter value.
4.2 EXPERIMENTS
In order to demonstrate the effectiveness of RankMe for hyperparameter selection, we apply the algorithm presented in fig. to find the best values for given of hyperparameters for VICReg and SimCLR. Our focus is on the covariance and invariance weights VICReg, the temperature in SimCLR, and on learning rate and weight decay for both. We compare the performance IetgeNet as well as the average performance on the previously discussed OOD datasets to models selected by their ImageNet top-1 accuracy on its validation set. For per dataset performance, confer appendix []] As can be seen in table ! using RankMe we are able to retrieve most of the performance on ImageNet, with gaps being lower than half point. It is not possible to beat the selection using ImageNet's validation, since this is the metric we are evaluating on. However, on OOD datasets we are able to improve the performance certain settings, and match in the others. Thus, when comparing performance after the projector, RankMe is the better approach of the two to select the hyperparameters that will generalize best to unseen datasets. When comparing to a-Req, RankMe achieves better in domain performance, but OOD datasets a-ReQ performs slightly better, though with bigger worst case performance drops. We provide an in-depth analysis of a- a-ReQ in appendix E where we find that the power law prior of a- R-ReQ fails the embeddings and such those results must be interpreted with care. As pointed out in Girish et al. (2022), using ImageNet performance to select models lead to suboptimal performance in downstream tasks, which our results further confirm and reinforces the need for a new way of selecting hyperparameters. When looking at performance before the projector in fig. D we can see that here RankMe does not beat the models selected with ImageNet's 's validation set, even on OOD datasets. However, RankMe performs better than a-ReQ in most settings, while not suffering from severe drops in the worst cases. Nevertheless, the gaps between RankMe and the ImageNet oracle are on average of less than half point, which shows how competitive RankMe can be for hyperparameter selection, despite using no labeled data, having no parameters to tune, and being able to be computed in couple of minutes.
CONCLUSION
We have shown how the phenomenon of dimensional collapse in self-supervised learning can be used as a powerful metric evaluate models. By using theoretically motivated analogue of the rank of embeddings, we show that the performance on downstream datasets can easily be assessed by only looking at the training dataset, without any labels, training, or parameters. While our work focuses on linear classification, we show promising results in non-linear classification that raise the question of how general this simple metric can be. Furthermore, its competitiveness with traditional oracle based hyperparameter selection methods makes it a promising tool in settings where labels are scarce, such as in the case of large uncurated datasets. As such, this work makes a step towards completely label-less self-supervised learning, as most existing approaches' hyperparameters tuned with the help of ImageNet's validation set. Further work will explore the use of RankMe in more varied scenarios, further legitimize its use in designing better self- superviedd approaches.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
While reproducing the pretrainings is prohibitively expensive since each training takes around a day on & VI00 GPUs, we provide all of the hyperparameters used appendix K We also provide all of the pretraining details in appendix [ along with the hyperparameters used for the linear evaluations in the same section. We also provide all the performance and rank values to reproduce our main figures in appendix K While the implementation of RankMe is straightforward, we provide an example algorithm to use it in fig. All of these efforts should make our results reproducible and verifiable.
REFERENCES
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205.11508 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Viereg: Variance-ivarrane--vvariene regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.
Florian Bordes, Randall Balestriero, and Pascal Vincent. High fidelity visualization of what your self-supervised representation knows about. arXiv preprint arXiv:2112.09164 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014, 4.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Sackinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. In NeurIPS, 1994.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, and Julien Mairal Piotr Bojanowski Ar- mand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In CVPR, 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, (3):326-344, 334, 1965.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-
supervised representation learning, 2021.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal
Octavian Ganea, Sylvain Gelly, Gary Bécigneul, and Aliaksei Severyn. Breaking the softmax bottleneck via learnalle monotonic pointwise non-linearities. In International Conference on Machine Learning, pp. 2073-2082, PMLR, 2019.
10

---

Under review as a conference paper at ICLR 2023
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning, arXiv preprint arXiv:2206.02557, 2022.
Arna Ghosh, Amab Kumar Mondal, Kumar Krishma Agrawal, and Blake Richards. Investigating power laws in deep representation learning. arXiv preprint arXiv:2202.05800 2022.
Sharath Girish, Debadeepta Dey, Neel Joshi, Vibhav Vineet, Shital Shah, Caio Cesar Teodoro Mendes, Abhinav Shrivastava, and Yale Song. One network doesn't It rule them all: Moving beyond handcrafted architectures in self--uper learning. arXiv preprint
2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint
2017.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. 2021.
Vissl. https: //op::://iiee git.hut.hh.... http://c...acboooreecrccc///////
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. NeurlPS, 34, 2021.
Bobby He and Mete Ozay. Exploring the gap between collapsed & whitened features self- supervised learning. In International Conference on Machine Learning, pp. 8613-8634. PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard. Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decor- relation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9598 0608,960,6600 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. URL htttps::/opperreiiess htw.::/tee.....oooooeeeeeeeoea..
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
11

---

Under review as a conference paper at ICLR 2023
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 2544561,,013 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. In NeurIPS, 2021.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao. Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. In ICLR, 2022a.
Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich T Sommer. Neural manifold clustering and embedding. arXiv preprint 20t::22001100000 2022b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976 11986, 2022.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pettexxtinaariiaateeresettttttt In CVPR, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744 2018.
William H Press, Saul A Teukolsky, William T Vetterling, and Brian P Flannery. Numerical recipes 3rd edition: The art of scientific computing. Cambridge university press, 2007.
Colorado J Reed, Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. Selfaugment: Automatic augmentation policies for self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2674-2683, 2021.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEEE, 2007.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in neural information processing systems, 31, 2018.
Claude Elwood Shannon. mathematical theory of communication. The Bell system technical journal, 27(3):379 1948.
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels imagenet? arXiv preprint arXiv:2201.05/199 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 9446-9454, 2015 8.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 20/0 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Chun-Hsiao Yeh, Cheng-Yar Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning, arXiv preprint 10.00688, 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: /708 03888, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML,, pp. 23100-13300. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In NeurIPS, 2014.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022a.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: multi-granular self- supervisd learning framework. 2022b.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In ICCV, 2019.
13

---

Under review as a conference paper at ICLR 2023
A BACKGROUND
In order to make our work as self-contained as possible, we recall the loss functions of the methods study. For conciseness, refer to the outputs of the encoder as representations and the outputs of the projection head as embeddings, which we denote by 24 SimCLR loss given by
Rd. We first briefly recall that the
CoSimm(z..5))
C= E E
(k.j)P k= (Aezaee
with P the set of all positive pairs the current mini-batch or dataset that comprise N exemplars.
VICReg's loss is defined with three components. The variances loss the dimensions, and the covariance loss aims at decorrelating dimensions the embeddings. They respectively defined as
regularizer for
c
Z
Σ max 0, 0.
S1--VVarzz.,)) Var(Z..)) and c(Z) 1 Σ Cov(Z)²
itf
Both of these loss combined with invariance loss that matches positives pairs, giving final loss of
C= Σ 2 # c(2) + v(Z).
(K.)EP
VICReg-exp defined similarly, but with the exponential covariance loss defined as
1 7 Σ log Σ Covl Z)../7
Cemp (
VICReg-ctris is then VICReg-exp but applied Z making it contrastive approach and conceptually similar to SimCLR. These methods give us different scenarios of collapse and allow us to make more general study of the rank of representations as powerful metric.
B INATURALIST PRETRAINING
SimCLR
VICReg
Naturalist188.
55 Nat Not
Dataset
Method
Top-1 Ropresntat
temp.
cov.
iNat18 Oracle
28.6
37.0
20
45
iNat18
RankMe
27.1
36.9
Difference
1.5
-0.1
10-
iNat18 Oracle
53.8
55.6
2016 8192
35
201168122
ImageNet
RankMe
52.5
55.9
Embedding
Embeddiin
Difference -1.3
+0.33
Figure $1: RankMe applied to INaturalist18 pretrainings (Lefft. RankMe is able to select hyperpa- rameters when pretraining on iNaturalist (Right).
While our experiments have previously focused on ImageNet pretraining due to its wide use in the community, one can wonder if RankMe still applicable when trained on another source dataset. To verify this, we pretrained VICReg and SimCLR on iNaturalist18, 18, and respectively varied the covariance loss" weight and temperature to study the influence on the rank embeddings. We used the same protocol ImageNet pretraining but trained for 300 epochs instead of 100 to obtain a similar number of iterations. We then evaluated the performance on iNaturalist 8 and ImageNet. We use 8192 dimensional embeddings due to the high number of classes of iNaturalist, but since the representations are 2048 dimensional, the rank cannot intrinsically go higher so we consider that all highers ranks effectively 2048.
14

---

Under review as a conference paper at ICLR 2023
As see in fig. ST RankMe provides similar level of performance ImageNet pretrainings, validating on a different source dataset. RankMe is even able to improve performance on ImageNet compared to the INaturalist18 oracle, further showing the limitations of such oracles downstream tasks.
C APPLICABILITY TO CLUSTER BASED METHODS
While we have studied the applicability of RankMe on contrastive methods, cluster based methods such DINO have become extremely popular, and since the definition of embeddings is not as clear cut them, a thorough analysis required. We will proceed in two steps:
Show that dimensional collapse happens right before the clustering layer, and not on the prototypes
Show that RankMe good measure of performance on DINO
109
Head
Log singular values
Projector
Clustering
10-2
Representations
N-layer MLP
Embeddings
Linear layer
12-norm.
Output
10-4
10-6
Protolypes
Embeddiags
0
500
1000
1500
2000
Singular values
Figure S2: DINO's projection head can be split in two parts, classical projector and clustering layer (Left). Collapse happens before the clustering layer and not on the clustering prototypes (Right).
DINO's projector be interpreted both classical projector and clustering layer, whose weights are clustering prototypes. This interpretation comes from the softmax that is applied on the output of the projection head which can be interpreted as an InfoNCE between the embeddings and the clustering prototypes that make up the clustering layer. While prototypes themselves are not particularly collapsed, the embeddings that are obtained before the clustering present dimensional collapse.
As we can see in figure S2
embeddings
Top-1 regresentation
Temperature
Method
40
20
Student
Teacher
ImageNet Oracle
72.4
72.3
20
RankMe
72.4
72.2
DING deacher DitG-etttten
Difference
-0.1
128 512
2048
60
128 512
2048
Embedding rank ImageNet
Embedding rank ImageNet
Figure S3: RankMe able to measure DINO's performance on its source dataset (Left). DINO's hyperparameters can be selected by using RankMe (Right).
As in fig. the phenomenon of dimensional collapse is highly visible in DINO, which enables the use RankMe to find optimal hyperparameter values. Validating this on ImageNet, we see that RankMe is able match the performance the oracle, or leads to slightly lower performance. further validating RankMe on another popular method.
D RESULTS ON SUPPLEMENTARY DATASETS
15

---

Under review as a conference paper at ICLR 2023
While previously focused certain datasets for their interesting natures, will provide additional visualizations for the remaining datasets.
60
ImageNet
75.03
FOODI01
80
VOCO7
Top-1 embeddings
50
75.02
40
75.01
30
ViCReg
65
VICReg-exp
20
ViCReg-stz
75.00
SunCLR
10
55
32
128
512
2048
8
32
128
512
2048
32
128
512
2048
Embedding rank ImageNet
Embedding rank ImageNet
Embedding rank ImageNet
50 COVCLEVR-onnt
CIFAR-10
CIFAR-100
80
Top-1 embeddings
50
40
75
70
30
65
30
20
60
32
2048
128
512
2048
Embedding rank ImageNet
Embedding rank ImageNet
Embedding rank ImageNet
Figure S4: Link between embedding rank and downstream performance on the embeddings.
70
ImageNet
75.02
FOOD101
VOC07
Top-1 reprosontations
84
65
75.01
75.00
60
VICReg
75.00
55
VICReg-sep
VICReg-dte
SimCLR
50
32
512
2048
2048
Embedding rank ImageNet
Embedding rank ImageNet
Embedding rank ImageNet
Couutt
CIFAR-10
74
CIFAR-100
Top-1 reprosentations
55
90
72
50
89
70
45
68
40
66
32
128
512
2048
8
32
128
512
2048
8
32
128
512 2048
Embedding rank ImageNet
Embedding rank ImageNet.
Embedding rank ImageNet
Figure S5: Link between embedding rank and downstream performance the representations.
find similar behaviours before, apart from Food101 where performance are almost identical for all methods. This reinforces the previous validation of RankMe. The relative simplicity of the datasets targeted here make the theoretical limitations of rank-defien defeceefcin embeddings harder to see, even though we still see that a high rank helps generalization.
As we see in figs. S4 and S5
E DETAILED RESULTS FOR a- REQ
16

---

Under review as a conference paper at ICLR 2023
In order to further study the performance of a-ReQ, reproduce plots for RankMe using a-ReeQ instead the rank of embeddings. We compare both the intended use of a-ReQ in fig. [S6] as well as applying it on the embeddings measure performance the representations, which we foudn was necessary for RankMe in fig. S7
in fig. 56 there performance. Especially we are unable to see the tendency of performance to increase as a tends to one. Nonetheless a-ReQ was still able lead to good performance when used for hyperparameter selection.
As
clear link visible between the value of a-ReQ and downstream
When applying a-ReQ performance increase when a tends to one. On the contrary we even find that performance tends to get better with lower a, as is most visible in StanfordCars, iNaturalist18 or ImageNet for example. going towards one means that the singular values of the embeddings tends to uniform distribution, line with the goal of RankMe.
would RankMe,
in fig. S7 that there is again trend of
Top-1 representations
70
ImageNN Net
NNa118
Places205
52
65
35
50
60
VICReg
48
55-
VICReg-exp VICRog-ctr
SimCLR
25
46
$8.3
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Power law coefficient.
Power law coefficient
Power law confficient
Top-1 representation tions
EuroSat
SUN397
Cars
96.5
68
55-
96.0
66
50
95.5
64
45
62
40
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Power law coefficient
Power coefficient
Power coefficient
75.02
F0
VOCO7
CLVR-cnt
Top-1 1 representations
55
75.01
82
50-
75.00
80
45
75.00
74.99
78
40
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Power law coefficient
Power law coefficient
Power coefficient
CIFAR10
74
CIFAR100
Top-1 reprosentations
90
72
89
70
88
87
66
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Power law coefficient
Power coefficient
Figure S6: Link between cx-ReQ measured on the representations and performance on the representa- tions.
As see in figs. 58 and S9 the power-law prior of a- Re- holds well in the case of non-collapsed embeddings, but when we apply on collapsed ones, this assumptions fails. It even provides a poor approximation of the main rank "plateau" with the highest singular values as be seen the right
17

---

Under review as a conference paper at ICLR 2023
ImageNett
i
Places205
Top-1 represontations
52
65
35
50
ViCReg VICReg-ax
30
55-
VICReg-dtr
SimCLR
25
$8.2
0.4
0.6
0.8
0.2
0.8 Power law coefficient o
0.4
0.6
0.2
0.4
0.6
0.8
Power law coefficlent a
Power law coefficient a
EuroSat
SUN397
Cars
Top-1 representations
96.5
68
66
50
64
95.5
62
40
0.2
0.4
0.6
0.8
0.2
0.2
0.4
0.6
0.8
Power law coefficient
Power law coefficient
Power law coefficient
Top-1 reprosentations
75.02
11
VOCO7
CLVR-cnt
84
55
75.01
50
75.00
80
45
75.00
78
0.2
0.4
0.6
0.8
0.2
0.8 Power law coefficiett o
0.4
0.6
0.2
0.4
0.6
0.8
Power law coefficient a
Power law coefficlent a
Top-1 reprosentations
CIFAR10
CIFAR100
90
72
89
70
68
66
0.2
0.4
0.6
0.8
0.2
Power law coefficient
Power law coefficient
Figure S7: Link between a-ReQ measured on the embeddings and performance on the representations.
of fig.
$9 This further confirms the findings He & Ozay 2022 and shows that when applying
a-ReQ directly on the embeddings one must be careful since the core assumptions of the method violated.
18

---

Under review as a conference paper at ICLR 2023
19
Singular valium
Siogular value
Index
Index
Figure S8: Validation the pawer-law prior on un-collapsed representations. (Left) Overall visual- ization. (Right) Zoom the high singular values.
17
30
Stagrdar value
Siog dogr dla valun
18
Index
Index
Figure S9: The pawer-law prior does not hold on collapsed representations. (Left) Overall visualiza- tion. (Right) Zoom on the high singular values.
19

---

Under review as a conference paper at ICLR 2023
F COMPARISON OF THE RANK ESTIMATORS
2048
1024
512
Entropic Rank
256
128
64
32
16
8
16
32
128
256
512
1024
2048
Classical rank
Figure S10: Relationship between the two rank estimators, Pearson correlation coefficient of 0.99. Outliers indicate embeddings with singular values to the threshold, showing how the entropic rank takes into account this information.
Since do not rely on the classical Threshold-basee rank estimator, is important to verify how well our entropy based one correlates with it. As we can see in fig. STO, both estimates discussed previously correlate extremely well, showing that using one or the other should not lead to significant differences, as validated in appendix H Nonetheless, the entropic estimator takes into account the degree of whitening of the embeddings, which links better theoretical results.
G CONVERGENCE OF THE RANK ESTIMATORS
Entropic Rank
Classical Rank
1.0
1.0
Max rank ratio
0.8
0.8
0.6
0.6
vicreg
vicreg
0.4
0.4
simclr
simclr
20
40
0
20
40
Number of data points in thousands
Number of data points in thousands
Figure S11: Convergence of the rank estimators ImageNet function of the number of samples for 2048 dimensional outputs, as indicated by the vertical line.
As we can see in fig. SII the rank estimates converge extremely quickly, especially for VICReg. For both VICReg and SimCLR, 10000 samples enough to obtain more than 95% of the final rank. It worth noting that the entropic rank estimator converges more slowly than the classical rank estimator, it is more sensitive to the singular values. The fact that the rank be approximated with few samples is encouraging for its use during training and not only as a measure performance after pretraining.
20

---

Under review as a conference paper at ICLR 2023
Emboaddng rank Tar harget
2048
0.96 Areapaie +
2048
Aressa
0.97 Rasagolt
-22
2048
Rowwer
0.95 Reugelee +
2048
Rim 0.27 Rerugutte! +
512
512
512
512
128
Vidlig Vidlle up di SaniLL
8 32 128
512 2018
22
8 32 126 512 2046
22 6 32 128 512 2018 Emhodding rank ImageNet
32 128 512 2048 Embedding rank ImageNat
Embedding rank ImageNet
Embodding rank ImageNet
Figure S12: Reproduction of the top fig. with the classical rank estimator. Embeddings' rank
transfers from source to target datasets, The estimates used 25600 images from the respective datasets.
Image
20
Top-1 Ieprosstaatioc
ImageNet
Top-1 enbeddings
50-
8
15
to
60
30
30
Vicleg-r
SExCLA
512
Embalding
Embeediin rank ImageNet
Embeeding EuroSat
Places205
EuroSat
Top-1 Norussntaation
Places205
Tep-1 ambodiingn
40-
90
96.5
50
80
96.0
48
70
95.5
46
Embecding
SUN3397
Cars
SDN397
Cars
onboldings
Top-1 Ipprosentation
BR
50
15
50
10-
4.5
30
32
ImageNNtt
Embesiiing ImageNNt
Embedking
ImagenNet
Embelding ImageNNt
Figure S13: Reproduction of fig. 3 with the classical rank estimator. (Left) Validation of RankMe on embeddings, higher ImageNet rank leads to improved performance across methods and datasets. (Right) Validation of RankMe on representations, where the link even clearer, rein- forcing RankMe's practical use.
H REPRODUCTION OF FIGURES WITH THE CLASSICAL RANK ESTIMATOR
As can be seen in figs. S12 and S13 the results that we obtain using the classical threshold- rank estimator
extremely similar the ones with the entropic estimator. The exact values do differ, but the behaviors stay the same. One of the main differences is illustrated in fig. S13 where we can see that the target rank is almost identical to the source one when we previously saw drop of around 50%. This can be explained by the fact that some features may be less present in the target dataset, reducing the associated singular values, and thus the entropic rank.
All of this shows that using one or the other will lead to similar results in practical scenarios.
I DETAILED TRAINING AND EVALUATION PROCEDURES
1.1 PRETRAINING
All pretrainings were done with ResNet-50 backbones. The projector used is a MLP with intermediate dimensions 8192, 8192, 2048. They were trained with the LARS optimizer using momentum of 0.9, weight decay 10 and varying learning rates depending on the method. VICReg used 0.3 base learning rate, SimCLR 0.5 0.6 depending the experiment, VICReg-exp 0.6 and VICReg-ctr 0.6.
21

---

Under review as a conference paper at ICLR 2023
Table S1: Image augmentation parameters, taken from Grill al. (2000)
Parameter
View 1
View 2
Random crop probability
1.0
1.0
Horizomtal flip probability
0.5
0.5
Color jittering probability
0.8
0.8
Brightness adjustment max intensity
0.4
0.4
Contrast adjustment max intensity
0.4
0.4
Saturation adjustment max intensity
0.2
0.2
Hue adjustment max intensity
0.1
0.1
Grayscale probability
0.2
0.2
Gaussian blurring probability
1.0
0.1
Solarization probability.
0.0
0.2
The learning rate is then computed as Ir = base_lr * batch_size//25. We do a 10-epochs linear warmup and then use cosine annealing. We used batch sizes of 2048 for SimCLR and 1024 for other methods. SimCLR and VICReg-ctr also use default temperature of 0,15, and 1 for VICReg-exp. We used the image augmentation strategy from Grill et al. (2020) illustrated in table
1.2
EVALUATION
Table S2: Optimization parameters used to evaluate on downstream datasets
Dataset
Optimizer
Weight decay
Momentum
Learning rate
Epochs
1Naturalist18
SGD (w/ Nesterov)
0.0005
0.9
0.01
Places205
SGD (w/ Nesterov)
0.0005
0.9
0.01
14
EuroSat
SGD (w/ Nesterov)
0.0005
0.9
0.01
Sun397
SGD (w/ Nesterov)
0.0005
0.9
0.01
28
StanfordCars
SGD (w/ Nesterov)
0.0005
0.9
0.1
28
CIFARIO
SGD (w/ Nesterov)
0.0005
0.9
0.1
28
CIFAR 100
SGD (w/ Nesterov)
0.0005
0.9
0.1
28
CLEVR-count
SGD (w/ Nesterov)
0.0005
0.9
0.1
50
Food101
SGD (w/ Nesterov)
0.0005
0..9
0.1
28
VOCO7
N/A,
text
For datasets except StanfordCars, we use the standard protocol in VISSL. StanfordCars we mostly tuned the learning rate. The parameters that used are described in table S2] For data augmentation, we used random resized crops and random horizontal flips during training, and center crop for evaluation. For VOC07, follow the common protocol using SVMs, used in Bardes et al. (2021). We use the default VISSL. settings for this evaluation.
22

---

Under review as a conference paper at ICLR 2023
J DETAILED TABLES FOR HYPERPARAMETER SELECTION
Table S3: Top-1 accuracies computed on representations when tuning hyperparameters with ImageNet
validation performance with RankMe.
VICReg
SimCLR
Dataset
Method
cov.
inv.
LR
WD
temp.
LR.
WD.
ImageNet Oracle
68.2 2
68.2
68.6
68.0
68.5
68.5
68.3
ImageNet
RankMe
67.8
67.9
68.2
67.8
67.1
68.0
68.3
a-ReQ
67.9
67.5
59.5
67.8
63.5
68.1
32.3
ImageNet Oracle
38.4
38.4
38.8
38.3
39.2
39.2
38.9
iNat18
RankMe
36.7
37.2
38.4
38.3
37.8
38.1
38.9
a-ReQ
37.8
36.9
28.9
38.3
34.1
38.4
38.7
ImageNet Oracle
51.2
51.2
51.8
51.3
52.4
52.4
52.6
Places205
RankMe
51.2
51.4
51.2
51.6
52.3
52.3
52.6
a-ReQ
51.1
51.4
47.8
51.6
50.7
52.3
52.6
ImageNet Oracle
96.2
96.2
96.3
96.2
96.5
96.5
96.4
EuroSat
RankMe
96.1
96.1
96.2
96.0
96.6
96.4
96.4
a-ReQ
96.1
96.1
95.1
96.0
96.4
96.6
96.2
ImageNet Oracle
68.4
68.4
68.6
68.6
68.9
68.9
69.2
SUN397
RankMe
68.6
68.3
68.4
68.8
69.1
68.5
69.2
a-ReQ
68.7
67.9
64.1
68.8
66.4
68.4
68.5
ImageNet Oracle
55.7
55.7
55.8
55.6
54.4
54.4
54.9
Cars
RankMe
51.1
54.0
55.7
55.4
51.5
53.9
54.9
a-ReQ
54.2
51.7
43.2
55.4
45.2
54.3
54.7
ImageNet Oracle
75.0
75.0
75.0
75.0
75.0
75.0
75.0
FOOD101
RankMe
75.0
75.0
75.0
75.0
75.0
75.0
75.0
a-ReQ
75.0
75.0
75.0
75.0
75.0
75.0
75.0
ImageNet Oracle
84.3
84.3
84.3
84.0
84.5
84.5
83.9
VOCO7
RankMe
84.1
83.8
84.3
84.0
83.8
83.9
83.9
a-ReQ
84.0
83.9
80.4
84.0
81.2
84.3
84.4
ImageNet Oracle
55.7
55.7
56.0
56.8
51.9
51.9
53.2
CLEVR-Count
RankMe
53.0
55.4
55.7
53.1
48.0
52.3
53.2
a-ReQ
52.1
55.1
50.6
53.1
44.0
50.5
51.3
ImageNet Oracle
90.1
90.1
90.0
89.8
90.6
90.6
90.3
CIFAR10
RankMe
89.5
89.8
90.1
89.7
89.4
90.6
90.3
a-ReQQ
89.7
89.2
86.8
89.7
88.0
89.7
90.3
ImageNet Oracle
72.3
72.3
72.8
72.2
73.8
73.8
73.7
CIFAR100
RankMe
71.6
72.3
72.3
72.1
72.2
73.1
73.7
a-ReQ
72.4
71.0
66.2
72.1
69.5
72.5
74.0
ImageNet Oracle
68.7
68.7
68.9
68.7
68.7
68.7
68.8
Average
RankMe
67.7
68.3
68.7
68.3
67.5
68.4
68.8
a-ReQ
68.1
67.8
63.4
68.3
64.9
68.2
65.3
23

---

Under review as a conference paper at ICLR 2023
Table S4: Top-1 accuracies computed on embeddings when tuning hyperparameters with ImageNet validation performance or with RankMe.
VICReg
SimCLR
Dataset
Method
COv.
inv.
LR
WD
temp.
LR.
WD.
ImageNet Oracle
59.7
59.7
59.7
59.7
56.9
56.9
57.1
ImageNet
RankMe
59.6
59.7
59.7
59.5
56.5
56.0
57.1
ax-eeQ
59.6
59.2
36.2
59.3
51.5
56.4
49.0
ImageNet Oracle
13.5
14.2
13.5
13.6
10.3
10.3
10.1
iNat18
RankMe
14.2
14.2
13.5
13.4
16.7
9.9
10.1
a-ReQ
14.2
14.8
2.5
13.2
21.5
10.0
10.0
ImageNet Oracle
42.7
43.3
42.7
43.4
41.2
41.2
41.2
Places205
RankMe
43.2
43.3
42.7
42.7
43.4
40.8
41.2
a-ReQ
43.2
43.6
29.6
42.9
42.6
41.0
41.5
ImageNet Oracle
91.3
91.7
91.3
91.0
90.4
90.4
89.5
EuroSat
RankMe
91.0
91.7
91.3
91.3
92.3
89.0
89.5
a-ReQ
91.0
91.4
85.1
90.8
94.4
89.6
89.8
ImageNet Oracle
57.3
57.0
57.3
57.3
56.4
56.4
56.2
SUN397
RankMe
57.4
57.0
57.3
56.7
59.1
55.4
56.2
a-ReQ
57.4
57.4
42.5
57.2
59.9
56.2
56.2
ImageNet Oracle
12.0
12.0
12.0
11.9
14.0
14.0
13.2
Cars
RankMe
11.6
12.0
12.0
11.5
17.6
13.4
13.2
a-ReQ
11.6
12.0
7.5
11.3
21.3
13.9
13.5
ImageNet Oracle
75.0
75.0
75.0
75.0
75.0
75.0
75.0
FOOD101
RankMe
75.0
75.0
75.0
75.0
75.0
75.0
75.0
a-ReQ
75.0
75.0
75.0
75.0
75.0
75.0
75.0
ImageNet Oracle
79.5
79.2
79.5
79.7
79.7
79.7
79.7
VOCO7
RankMe
79.2
79.2
79.5
79.3
78.5
79.3
79.7
a-ReQ
79.2
79.2
73.1
79.6
76.8
79.5
79.9
ImageNet Oracle
43.9
44.4
43.9
46.1
43.5
43.5
46.0
CLEVR-Count
RankMe
43.9
44.4
43.9
43.0
43.0
44.8
46.0
a-ReQ
43.9
43.8
41.7
44.9
37.0
45.2
45.9
ImageNet Oracle
80.4
81.2
80.4
79.7
79.3
79.3
79.8
CIFARIO
RankMe
80.6
81.2
80.4
80.3
79.5
79.5
79.8
a-ReQ
80.6
81.0
72.5
79.6
79.2
78.5
79.4
ImageNet Oracle
52.8
53.3
52.8
52.9
52.6
52.6
52.2
CIFAR100
RankMe
53.8
53.3
52.8
52.5
54.0
52.2
52.2
a-ReQ
53.8
53.9
41.5
52.2
56.5
52.0
52.3
ImageNet Oracle
55.3
55.5
55.3
55.5
54.5
54.5
54.5
Average
RankMe
55.4
55.5
55.3
55.0
56.0
54.1
54.5
a-ReQ
55.4
55.6
46.1
55.1
56.0
54.3
53.9
24

---

Under review as a conference paper at ICLR 2023
K
COMPLETE PERFORMANCE TABLES
Table S5: Hyperparameters for all runs.
Method
Run
Batch size
Learning rate
Weight decay
Loss hyperparameters
1024
0.3
10-6
25 p 25, v 0.3
1024
0.3
10-6
:: :4:22,,0 25, 0.4
2
1024
0.3
10-6
p: 25. 0,5
1024
0.3
10-6
A: 2:: 0.6
4
1024
0.3
10-6
22, 0.7
5
1024
0.3
10-6
:: 25 0.8
1024
0.3
10-6
2:25, 2:.. 25,
1024
0.3
10-6
À ::25, :2, v::
8
1024
0.3
10-6
À :5, 25,
9
1024
0.3
10-6
24:
10
1024
0.3
10
À :: 88
11
1024
0.3
10-6
X p:: 16
12
1024
0.3
10-6
:::5 v:
1024
0.3
10-6
A: : :.,
14
1024
0.3
10
A: 1:15, 25,
VICReg
15
1024
0.3
10-6
X 2: 25,
16
1024
0.3
10-6
3:
17
1024
0.3
10
A: 3:
18
1024
0.3
10
A : 25.
1024
0.3
10-6
X 4:45 25.
20
1024
0.3
10
À :000 25, V:4
21
1024
0.1
10-6
A: :5,
p 25.
1024
0.2
10-6
X 1:: 25:
23
1024
0.3
10-6
2:25, 25,
24
1024
0.4
10
X 2:25, 25, v
1024
0.5
10-6
::25, 20:
26
1024
0.3
10-7
:: 25,0:4
27
1024
0.3
10-6
X :: 25.
1024
0.3
10-5
À 2525, :5.5::5.111
29
1024
0.3
10-4
:5 p: 25,
30
1024
0.3
10-3
À 25, 25, #:4
1024
0.3
10-2
2:25, p: 25,
O
1024
0.5
10-6
1,1.,=1,v:2.73
1024
0.5
10-6
1,1;p;1,v:2.771
1024
0.5
10-6
A:1.p:1v::2.:000
3
1024
0.5
10-6
A:1.u::.v:2::003
1024
0.5
10-6
A:1.p::1v::2::00:
VICReg-exp
1024
0.5
10
A:1.4:1.1:2::00.
6
1024
0.5
10-6
X: 1:L.::LP::11:::00
1024
0.5
10-6
A:1.p:1,p::.3:f:0000
1024
0.5
10-6
A:1.p:1,v:1::001
9
1024
0.5
10-6
A:1.p:12::4:011
10
1024
0.5
10-6
A:1.w:1,v:87:01
1024
0.5
10-6
A:1,::1,v:16.T:001
25

---

Under review as a conference paper at ICLR 2023
Table S6: Hyperparameters for all runs, continued.
Method
Run
Batch size
Learning rate
Weight decay
Loss hyperparameters
0
1024
0.5
10 10-6
1: 0.05
1
1024
0.5
10
X A; 0.07 0.07
2
1024
0.5
10
A: A:1.u:1::: 0.1
3
1024
0.5
10
::
4
1024
0.5
10
À: A:I.u;1vi1,;:00
VICReg-ctr
5
1024
0.5
10
A:1.4:1,2:1.7:00.
6
1024
0.5
10 10-6
A:1.p:L.:::1,::000
7
1024
0.5
10
A:1:::::::05:::..
8
1024
0.5
10 6
A:1:u:11::2::11
9
1024
0.5
10 10-6
A: A:1:::11::::111
10
1024
0.5
10
À A:I.u:1,v:8.r:0..
0 2048
0.6
10
d:512,
1
2048
0.6
10
d:512,
2
2048
0.6
10 6
d:512,7:0.
3
2048
0.6
10
d:512,7:0.2
4
2048
0.6
10 6
d:512,7:03
5 2048
0.6
10 6
d:512, 7::12,=:000
6
2048
0.6
10
:04048, 7 0.05
7
2048
0.6
10 6
2048, 7 0.07
8
2048
0.6
10 6
2048, T 0.1
9
2048
0.6
10
d 2048, 0.2
10
2048
0.6
10
d 2048, T 0.3
11
2048
0.6
10 6
d 2048, T : 0.4
12
2048
0.5
10
d 2048, 0.05
13
2048
0.5
10
d: 2048, 0.07
14
2048
0.5
10 6
d 2048, T 0.1
SimCLR
15
2048
0.5
10
d 2048, 7 0.15
16
2048
0.5
10
d 2048, 0.2
17
2048
0.5
10
d 2048, 7 0.3
18
2048
0.5
10 6
d 2048, 7 0.4
19
2048
0.5
10
d : 2048, 0.15
20
2048
0.5
10
d 2048, 7 0.15
2048
0.5
10 3
d 2048, 0.15
22
2048
0.5
10
d 2048, 0.15
23
2048
0.5
10
d: 2048, 0.15
24
2048
0.5
10-2
2048, T 0.15
25
2048
0.2
10
d: 2048, 0.15
26
2048
0.3
10 10
d: 2048, 0.15
27
2048
0.4
10 10-6
2048, T 0.15
28
2048
0.5
10 10-6
d: 2048, T 0.15
29
2048
0.6
10
d: 2048, 0.15
30
2048
0.8
10 6
d 2048, 0.155
26

---

Under review as a conference paper at ICLR 2023
Table S7: Top-1 on representations all settings.
Method
Run
ImageNet
iNat18
Places205
EuroSat
SUN397
Cars
0
63.90
34.12
48.77
95.94
65.96
52.56
65.08
35.65
49.68
96.10
66.87
54.47
2 65.67
36.97
49.73
96.02
67.33
55.76
3
66.17
37.20
50.13
96.10
67.55
56.37
66.40
37.42
50.15
96.34
67.99
56.86
5
66.83
38.05
50.53
96.06
68.40
57.63
67.30
38.13
50.96
96.20
68.08
57.83
7
67.34
38.26
50.96
96.36
68.
58.89
68.00
38.68
51.28
96.36
68.46
56.90
9
68.16
38.36
51.17
96.20
68.42
55.70
10
67.91
37.75
51.14
96.14
68.75
54.21
11
67.77
36.70
51.20
96.06
68.57
51.05
12
64.12
31.37
49.83
95.56
66.17
42.56
13
66.67
34.81
50.76
95.68
67.61
47.33
14
67.49
36.91
51.40
96.10
67.95
51.72
15
67.87
37.18
51.40
96.06
68.26
54.00
VICReg
16
67.99
38.71
51.11
96.16
68.68
56.05
17
67.78
38.52
50.79
96.38
68.39
57.13
18
67.25
38.08
50.85
96.34
68.69
56.29
19
66.95
37.93
50.88
96.06
67.98
57.67
20
66.51
37.79
50.11
96.10
67.74
57.23
21
59.54
28.85
47.80
95.10
64.14
43.15
22
66.36
35.47
50.32
96.04
67.45
51.64
23
68.16
38.36
51.17
96.20
68.42
55.70
24
68.56
38.80
51.75
96.30
68.60
55.75
25
62.77
31.85
48.02
95.72
64.82
43.23
26
67.79
38.25
51.57
96.04
68.84
55.38
27
67.97
38.26
51.29
96.16
68.62
55.57
28
67.87
38.43
51.31
96.08
68.52
54.53
29
63.36
38.31
51.17
96.06
68.43
55.06
30
54.52
37.92
51.32
96.10
67.99
54.82
31
40.73
37.03
50.97
96.30
68.40
52.28
0
67.74
37.53
51.44
96.36
68.41
52.12
67.64
38.00
51.42
96.46
68.60
54.16
2
67.84
38.25
51.07
96.44
68.12
55.94
3
65.09
36.64
49.65
96.54
67.12
56.37
60.67
31.22
48.04
95.80
64.28
46.96
5
57.46
26.54
46.25
96.02
62.33
41.90
VICReg-exp
55.12
24.73
45.68
95.44
61.82
39.71
7
64.87
36.51
49.69
96.16
66.82
55.30
66.84
38.25
50.85
96.24
68.34
57.12
9
68.08
38.03
51.34
96.40
69.28
53.72
10
67.80
37.20
51.57
96.46
68.61
52.15
11
66.68
35.02
50.94
96.00
67.81
47.49
27

---

Under review as a conference paper at ICLR 2023
Table S8: Top-1 representations in all settings, continued.
Method
Run
ImageNet
iNat18
Places205
EuroSat
SUN397
Cars
0
65.54
35.00
50.15
95.88
67.62
49.63
66.32
35.72
50.69
96.10
68.16
51.66
2
66.09
35.26
50.80
96.42
68.32
50.72
64.06
33.16
50.48
95.98
67.40
44.91
4
62.06
30.80
49.53
96.22
66.08
43.24
VICReg-ctr
60.17
28.76
48.78
95.90
64.92
41.13
6
61.66
31.05
49.40
96.10
66.47
44.12
65.47
34.63
50.71
96.20
67.55
48.05
8
65.99
34.77
50.63
95.98
67.60
51.14
9
63.87
33.63
49.64
96.20
66.17
50.35
10
58.81
29.24
47.60
95.78
63.77
46.23
57.68
30.50
48.51
96.32
63.36
42.42
1
62.79
33.50
50.56
96.18
66.34
43.76
66.13
35.94
52.10
96.22
68.29
49.17
3
66.35
35.60
51.96
96.64
68.17
49.68
65.17
34.38
51.32
96.10
67.78
48.17
5
63.54
33.29
50.71
96.22
67.39
48.31
6
57.84
30.82
48.64
96.34
64.07
41.97
7
62.73
33.30
50.57
96.56
66.03
44.99
8
66.30
36.25
51.79
96.40
67.99
48.95
66.71
36.56
51.82
96.52
68.52
50.47
10
65.29
34.90
51.32
96.30
67.40
49.16
63.52
33.35
50.92
96.42
66.89
48.59
12
59.49
31.13
48.80
95.94
64.11
42.46
63.51
34.14
50.75
96.42
66.44
45.18
14
67.14
37.80
52.29
96.62
69.06
51.47
SimCLR
68.48
39.20
52.37
96.46
68.92
54.43
16
68.27
38.48
52.29
96.46
69.19
55.22
67.48
37.07
51.72
96.58
68.30
51.92
18
66.44
35.87
51.58
96.44
68.15
49.76
19
68.33
38.93
52.56
96.40
69.21
54.86
20
68.13
39.09
52.42
96.42
69.15
54.83
21
66.47
38.80
52.81
96.58
69.03
55.19
22
59.62
38.86
52.69
96.62
69.07
55.47
23
47.58
39.03
52.70
96.16
68.77
54.96
24
32.27
38.70
52.62
96.18
68.53
54.67
25
66.37
36.06
51.62
96.84
68.22
52.17
67.96
38.12
52.33
96.44
68.54
53.86
27
68.32
38.44
52.42
96.80
69.08
54.63
68.48
39.20
52.37
96.46
68.92
54.43
29
68.41
38.77
52.42
96.24
68.65
55.81
68.12
38.45
52.33
96.64
68.41
54.30
28

---

Under review as a conference paper at ICLR 2023
Table S9: Top-1 on representations in all settings, continued.
Method
Run
ImageNet
CIFARIO
CIFAR100
FOODI01
VOCO7
CLEVR-count
63.90
88.94
69.92
75.00
81.49
48.94
65.08
88.74
69.96
75.00
82.14
49.88
2 65.67
88.25
70.23
75.00
82.35
54.96
3
66.17
89.17
71.51
75.00
82.97
52.35
66.40
89.41
71.70
75.01
82.81
55.27
5
66.83
89.91
72.12
75.00
83.10
55.95
67.30
90.11
71.90
75.01
83.15
54.37
7
67.34
90.34
72.42
75.00
83.21
53.92
68.00
89.79
72.73
75.00
83.77
49.75
68.16
90.14
72.26
75.01
84.27
55.69
10
67.91
89.67
72.39
75.00
83.99
52.10
11
67.77
89.45
71.63
75.00
84.10
53.05
12
64.12
86.68
67.02
75.00
82.44
51.46
13
66.67
88.32
69.86
75.00
83.50
55.48
14
67.49
89.22
71.01
75.01
83.85
55.05
15
67.87
89.82
72.30
75.00
83.76
55.36
VICReg
16
67.99
90.29
72.81
75.00
83.90
55.00
67.78
90.09
73.14
75.00
83.74
51.97
18
67.25
90.40
72.75
75.00
83.36
53.18
19
66.95
89.62
72.14
75.00
82.99
50.33
20
66.51
89.94
72.41
75.00
82.89
52.83
21
59.54
86.81
66.23
75.00
80.44
50.64
22
66.36
88.92
71.05
75.00
82.94
56.19
23
68.16
90.14
72.26
75.01
84.27
55.69
24
68.56
89.95
72.80
75.00
84.27
56.03
25
62.77
87.63
67.92
74.99
82.38
52.63
26
67.79
89.70
72.11
75.00
83.98
53.08
27
67.97
89.83
72.22
75.00
84.05
56.77
28
67.87
90.23
72.13
75.00
83.72
56.20
29
63.36
89.76
72.36
75.00
84.04
54.71
30
54.52
89.58
71.89
75.00
84.14
53.45
31
40.73
89.65
71.50
75.01
83.97
56.34
67.74
89.66
72.17
75.00
84.67
52.79
67.64
90.12
72.30
75.00
84.58
55.29
67.84
89.55
72.07
75.00
84.20
53.45
65.09
89.18
71.55
75.00
82.19
54.41
60.67
88.06
68.94
75.00
80.20
51.35
57.46
86.70
65.09
75.00
78.54
49.30
VICReg-exp
55.12
87.23
65.53
75.00
77.87
53.14
7
64.87
89.28
71.38
75.00
82.39
49.13
66.84
89.66
71.92
75.00
83.91
50.41
68.08
89.56
71.90
75.00
84.64
56.00
67.80
89.50
71.61
75.00
84.45
55.80
11
66.68
88.99
70.13
75.00
84.29
55.87
29

---

Under review as a conference paper at ICLR 2023
Table S10: Top-1 on representations in all settings, continued.
Method
Run
ImageNet
CIFARIO
CIFARI00
FOOD101
VOC07
CLEVR-count
65.54
88.87
70.77
75.01
83.28
53.97
I
66.32
89.57
70.93
75.00
84.17
53.19
2
66.09
89.49
71.17
75.00
83.90
53.29
3
64.06
89.62
71.39
75.00
83.18
48.57
4
62.06
88.60
69.41
75.00
82.35
46.48
VICReg-ctr
5
60.17
88.97
68.61
75.00
81.43
51.27
6
65.47
89.65
71.62
75.01
84.09
51.07
65.99
88.97
70.40
75.00
83.69
46.92
8
63.87
88.51
69.02
75.00
82.99
51.36
58.81
86.96
66.06
75.00
79.95
55.33
0
57.68
86.31
66.69
75.00
77.56
37.65
62.79
87.15
68.71
75.00
80.98
50.26
2 66.13
89.19
71.13
75.00
83.57
47.75
3
66.35
89.99
72.44
75.00
84.25
54.73
4 65.17
89.89
72.18
75.01
83.99
50.78
5 63.54
89.50
71.09
75.01
83.37
52.87
6
57.84
86.44
66.42
75.00
77.01
42.72
7
62.73
87.57
68.33
75.00
81.26
45.19
8 66.30
89.07
71.55
75.00
83.61
52.95
9 66.71
90.12
72.52
75.00
84.17
52.93
10
65.29
89.44
71.62
75.00
83.81
54.83
11 63.52
89.32
70.88
75.00
83.39
48.44
12
59.49
86.41
66.45
75.00
77.98
50.64
13
63.51
87.98
69.53
75.00
81.19
44.03
14
67.14
89.40
72.20
75.01
83.80
47.97
SimCLR
15
68.48
90.57
73.78
75.00
84.54
51.91
16
68.27
90.34
73.63
75.01
84.48
50.11
17
67.48
90.04
72.81
75.00
84.31
47.31
18
66.44
89.80
72.02
75.00
84.35
49.94
19
68.33
90.29
73.65
75.00
83.95
53.17
20
68.13
90.67
73.85
75.00
84.61
54.20
21
66.47
90.33
73.39
75.00
84.22
55.01
22
59.62
90.53
73.63
75.00
84.61
50.53
23
47.58
90.29
72.99
75.00
84.44
48.27
24
32.27
90.29
73.96
75.00
84.42
51.33
25
66.37
89.95
73.16
75.00
83.01
50.75
26
67.96
90.65
73.13
75.00
83.94
52.29
27
68.32
90.21
73.13
75.00
84.31
53.57
28
68.48
90.57
73.78
75.00
84.54
51.91
29
68.41
90.17
73.35
75.00
84.27
52.97
30
68.12
89.67
72.53
75.01
84.27
50.47
30

---

Under review as a conference paper at ICLR 2023
Table S11: Top-1 on embeddings in all settings.
Method
Run
ImageNet
iNat18
Places205
EuroSat
SUN397
Cars
0
26.35
0.95
21.48
65.10
31.23
5.10
30.54
1.39
21.07
63.80
31.60
5.24
2 36.92
1.85
24.77
69.92
35.24
5.12
3
47.60
4.34
35.48
87.84
47.62
7.56
51.26
6.14
36.43
88.58
49.29
8.82
5
54.39
7.87
38.04
88.44
51.88
9.76
55.66
8.76
38.97
89.42
53.14
10.31
7
56.33
9.56
39.65
89.76
53.39
10.53
58.65
12.08
41.73
90.88
56.35
11.75
9
59.71
13.47
42.72
91.28
57.26
11.95
10
59.58
14.18
43.22
90.96
57.43
11.58
11
59.22
14.63
43.48
91.34
57.75
11.99
12
53.78
12.80
42.35
92.22
55.41
10.46
13
57.94
14.36
43.61
91.60
57.89
11.30
14
59.20
14.77
43.63
91.40
57.36
12.00
15
59.73
14.20
43.31
91.66
57.04
11.95
VICReg
16
59.09
12.35
42.21
90.34
56.21
11.55
17
58.23
11.34
41.02
90.16
54.97
11.69
18
56.82
10.15
40.19
89.94
54.50
10.76
19
55.22
9.26
39.02
90.00
53.07
10.98
20
53.75
8.29
37.87
89.76
52.16
10.60
21
51.53
12.97
40.84
91.64
54.26
13.13
22
57.57
13.60
42.40
91.64
56.43
12.47
23
59.71
13.47
42.72
91.28
57.26
11.95
24
56.22
9.92
40.18
88.36
53.69
8.46
25
36.22
2.48
29.59
85.06
42.46
7.55
26
59.33
13.22
42.86
90.76
57.21
11.28
27
59.51
13.37
42.69
91.34
56.66
11.53
28
59.70
13.64
43.37
90.96
57.32
11.89
29
59.03
14.00
43.10
91.50
57.44
12.27
30
56.37
14.10
43.23
91.36
57.69
12.52
31
49.96
12.36
41.93
91.52
57.13
11.37
0
58.19
12.56
41.93
91.82
57.13
10.19
58.53
12.10
42.03
91.64
56.85
10.96
2
57.41
10.78
40.89
90.44
55.24
10.73
3
47.80
5.02
34.67
88.92
47.14
8.39
25.14
1.02
22.04
77.22
32.66
5.17
5
19.24
0.75
20.08
75.70
30.22
5.77
VICReg-exp
12.03
0.51
16.95
70.18
26.06
4.15
7
47.33
4.78
33.88
87.32
46.90
8.87
53.72
7.74
38.04
89.46
52.37
9.76
9
58.87
12.22
42.27
91.34
56.97
10.93
10
58.09
12.70
42.56
91.58
56.90
10.42
11
57.24
14.01
43.18
92.38
57.36
11.18
31

---

Under review as a conference paper at ICLR 2023
Table S12: Top-1 on embeddings in all settings, continued.
Method
Run
ImageNet
iNat18
Places205
EuroSat
SUN397
Cars
0 50.26
10.83
38.54
89.54
52.73
11.39
50.99
9.81
38.43
90.06
53.90
10.53
2
48.27
7.67
36.78
88.02
51.45
10.21
36.77
3.20
30.01
84.12
43.85
6.68
4
25.92
1.51
23.93
77.54
36.57
4.46
VICReg-ctr
17.69
0.70
18.19
69.00
28.30
3.73
6
26.90
1.65
24.69
75.46
36.72
4.86
44.31
5.81
34.82
87.84
49.23
9.25
8
49.31
8.71
37.64
89.16
51.98
10.78
9
46.43
8.38
36.11
89.44
50.27
10.25
10
38.33
6.21
32.10
86.44
44.07
8.97
43.05
20.48
37.83
94.12
55.20
22.57
1 49.69
21.28
41.82
93.48
58,23
21.38
54.45
17.49
42.92
91.88
57.86
16.55
3
50.24
8.36
39.22
88.42
51.94
11.58
45.77
6.36
36.55
87.16
48.59
10.20
5
41.14
4.81
34.04
84.76
45.36
9.10
6
43.31
20.51
38.16
94.44
55.40
23.24
7
49.60
21.51
41.93
93.88
59.05
22.43
8
54.48
17.92
43.00
92.86
59.53
18.22
50.72
8.65
39.64
89.44
54.47
12.96
10
43.32
5.84
36.51
87.62
50.85
11.33
41.15
5.02
34.26
85.52
48.16
10.87
12
44.61
21.33
39.20
94.08
56.15
22.82
51.54
21.50
42.64
94.40
59.87
21.30
14
56.51
16.68
43.39
92.26
59.10
17.56
SimCLR
56.89
10.35
41.21
90.38
56.37
14.04
16
54.18
7.16
39.42
87.94
54.24
11.47
49.19
4.98
37.12
87.14
50.85
10.33
18
44.72
3.89
35.11
86.00
48.38
9.49
19
57.06
10.12
41.18
89.52
56.20
13.19
20
56.72
10.17
41.38
90.08
56.40
13.23
21
56.14
10.26
41.47
89.12
56.74
13.18
22
48.98
10.03
41.48
89.84
56.15
13.47
23
35.92
10.03
41.33
89.74
56.38
13.87
24
28.26
9.68
41.22
86.88
56.02
13.13
25
52.65
9.98
40.47
90.20
55.38
13.72
55.98
9.88
40.84
89.00
55.43
13.41
27
56.43
10.03
41.04
89.60
56.23
13.89
56.89
10.35
41.21
90.38
56.37
14.04
29
56.65
10.30
41.40
89.22
56.42
13.79
56.56
10.60
41.61
90.14
56.82
13.90
32

---

Under review as a conference paper at ICLR 2023
Table S13: Top-1 on embeddings in all settings, continued.
Method
Run
ImageNet
CIFARIO
CIFARI00
FOODI01
VOCO7
CLEVR-count
26.35
59.23
25.84
75.00
66.15
21.75
30.54
60.57
25.67
75.01
67.33
19.64
36.92
63.77
29.67
75.00
70.94
23.72
3 47.60
75.59
44.58
75.01
75.90
41.44
51.26
76.88
45.26
75.01
76.85
34.20
5
54.39
78.34
49.63
75.01
77.85
40.24
55.66
78.71
49.89
75.01
78.17
38.37
7
56.33
78.89
50.52
75.01
78.41
39.88
58.65
79.57
50.95
75.01
79.46
43.13
59.71
80.43
52.75
75.01
79.50
43.93
10
59.58
80.59
53.80
75.02
79.19
43.87
11
59.22
80.94
53.66
75.03
78.90
44.90
12
53.78
78.96
51.83
75.03
76.17
43.71
13
57.94
81.43
53.92
75.02
78.04
45.23
14
59.20
81.04
53.88
75.03
79.18
43.75
15
59.73
81.16
53.35
75.02
79.18
44.39
VICReg
16
59.09
80.46
52.82
75.02
79.20
45.92
58.23
79.76
51.77
75.01
79.58
36.53
18
56.82
79.20
51.10
75.01
78.62
37.81
19
55.22
78.82
50.19
75.01
78.36
38.50
20
53.75
77.87
49.34
75.01
78.07
38.62
21
51.53
78.45
51.46
75.01
75.99
49.13
22
57.57
80.67
52.87
75.02
78.53
45.93
23
59.71
80.43
52.75
75.01
79.50
43.93
24
56.22
75.80
45.73
75.02
78.90
39.71
25
36.22
72.55
41.50
75.00
73.12
41.73
26
59.33
79.58
52.25
75.02
79.61
44.89
27
59.51
80.26
52.55
75.01
79.29
42.99
28
59.70
79.74
52.92
75.02
79.67
46.11
29
59.03
81.25
54.96
75.01
80.10
43.33
30
56.37
80.81
53.55
75.01
80.11
46.97
31
49.96
80.86
53.10
75.01
79.68
47.09
58.19
80.80
52.30
75.01
79.73
45.89
58.53
80.15
53.08
75.01
80.18
43.75
57.41
79.22
51.69
75.01
79.39
43.92
47.80
76.70
45.93
75.00
76.21
43.52
25.14
66.91
33.49
75.00
66.12
37.21
19.24
65.85
29.87
73.00
62.04
34.52
VICReg-exp
12.03
62.48
26.06
75.00
55.71
34.17
7
47.33
76.91
46.47
75.00
76.48
41.40
53.72
77.99
48.65
75.00
78.72
44.80
58.87
80.36
53.78
75.01
80.15
43.85
58.09
80.64
53.47
75.01
79.98
45.68
11
57.24
81.10
54.28
75.01
79.58
43.71
33

---

Under review as a conference paper at ICLR 2023
Table S14: Top-1 on embeddings in all settings, continued.
Method
Run
ImageNet
CIFARIO
CIFARI00
FOOD101
VOC07
CLEVR-count
50.26
78.76
49.34
75.00
77.89
37.61
I
50.99
78.63
49.80
75.00
78.77
43.76
2
48.27
77.86
48.75
75.00
78.48
40.49
3
36.77
73.83
40.44
75.00
72.19
38.16
4
25.92
66.81
32.82
75.00
65.32
31.63
VICReg-ctr
5
17.69
63.94
25.50
75.00
58.48
31.40
6
44.31
76.82
46.43
75.00
77.16
42.48
49.31
77.70
48.66
75.00
78.50
40.81
8
46.43
75.51
46.54
75.00
76.97
44.25
38.33
71.51
40.68
75.00
72.67
41.31
0 43.05
75.84
52.39
75.00
71.43
46.86
49.69
78.32
53.20
75.00
76.19
50.11
2 54.45
79.24
51.56
75.00
78.10
49.63
3
50.24
78.36
47.70
75.00
79.07
46.29
4
45.77
75.73
44.37
75.00
77.46
46.63
5
41.14
74.04
41.88
75.00
75.03
45.47
6
43.31
75.66
53.31
75.00
71.37
41.40
49.60
78.36
54.79
75.00
76.47
46.46
8 54.48
80.39
55.39
75.00
78.38
49.53
50.72
79.32
51.01
75.00
79.27
42.25
10
43.32
76.43
48.07
75.00
77.59
44.67
11 41.15
74.75
45.06
75.00
74.79
46.56
12
44.61
76.97
53.29
75.00
72.71
47.15
13
51.54
79.20
56.45
75.00
76.80
37.03
14
56.51
79.48
53.99
75.00
78.55
43.05
SimCLR
15
56.89
79.35
52.58
75.00
79.73
43.51
16
54.18
78.17
49.62
75.00
79.71
43.32
17
49.19
76.59
46.98
75.00
78.61
43.48
18
44.72
76.68
45.62
74.99
76.95
42.88
19
57.06
79.83
52.19
75.00
79.73
45.97
20
56.72
79.41
53.12
75.00
80.05
45.65
21
56.14
79.46
52.25
75.00
79.64
47.19
22
48.98
79.39
52.28
75.00
79.89
45.92
23
35.92
79.52
52.15
75.00
79.74
43.86
24
28.26
79.28
51.25
75.00
79.90
45.39
25
52.65
79.30
52.85
75.00
78.74
42.71
26
55.98
79.51
52.21
75.00
79.34
44.75
27
56.43
78.52
52.04
75.00
79.55
45.19
28
56.89
79.35
52.58
75.00
79.73
43.51
29
56.65
78.98
51.80
75.00
79.82
43.69
30
56.56
78.36
51.81
74.99
79.78
44.50
34

---

Under review as a conference paper at ICLR 2023
Table S15: Rank after projector in all settings.
Method
Run
ImageNet
iNat18
Places205
EuroSat
SUN397
Cars
102.07
38.10
44.39
14.61
32.40
7.03
229.81
92.53
129.47
88.78
98.44
12.58
374.25
135.79
206.29
120.31
163.31
19.77
612.12
261.34
336.16
228.60
265.64
38.90
831.49
382.55
467.68
366.78
374.50
59.15
952.55
449.44
539.24
428.87
435.94
77.36
1033.93
493.50
587.19
477.69
478.34
88.28
1088.15 13
531.16
630.80
514.70
517.47
99.97
1442.63
726.28
849.29
693.16
723.53
161.76
1809.06
947.81
1110.80
855.76
954.83
210.06
10
1920.81
1054.70
1247.93
870.56
1075.89
258.33
11
1938.44
1087.45
1275.60
924.66
1119.33
306.90
12
1937.78
1100.54
1337.88
963.14
1172.38
382.18
1944.95
1095.95
1307.62
968.96
1155.65
352.50
14
1940.04
1095.91
1280.85
910.16
1126.89
324.51
15
1942.12
1049.72
1240.87
893.25
1070.12
269.96
VICReg
16
1521.07
782.39
919.54
725.49
771.86
169.75
1278.67
637.18
757.19
606.98
627.48
128.96
18
1079.67
532.00
634.88
527.59
524.80
111.28
909.71
446.52
525.65
454.22
431.44
88.55
20
777.82
376.39
447.53
378.06
360.41
73.57
21
1409.29
890.97
996.12
814.00
889.66
352.57
22
1652.41
936.47
1070.40
837.76
932.17
275.04
23
1809.66
947.81
1110.80
855.76
954.83
210.06
24
1422.16
648.60
813.33
532.92
650.33
91.44
25
101.29
44.12
46.00
20.77
36.60
10.68
26
1821.80
959.98
1130.27
840.12
962.04
221.58
27
1814.64
948.47
1107.25
856.12
946.73
218.36
28
1728.89
913.31
1065.74
814.04
911.39
216.25
29
1587.36
859.56
1008.93
807.57
864.18
244.56
1384.68
757.81
881.53
716.36
767.14
229.93
31
974.91
508.81
613.44
508.01
526.43
143.61
1006.58
530.95
637.48
501.16
551.60
142.88
1002.17
521.34
626.39
515.00
534.56
132.72
922.59
473.26
564.18
475.88
472.06
119.75
399.09
192.27
233.31
202.71
189.78
36.95
63.82
30.25
36.98
21.39
30.63
7.90
19.47
12.49
9.57
6.33
7.96
3,58
VICReg-exp
9.42
7.19
5.41
3.80
4.73
2.55
375.38
180.63
216.71
191.99
176.94
31.86
636.60
314.20
380.13
341.21
312.04
66.31
1002.29
528.76
629.07
517.84
536.91
139.28
1048.58
556.24
673.46
547.15
581.30
158.24
11
1326.31
733.86
875.62
707.34
771.39
208.83
35

---

Under review as a conference paper at ICLR 2023
Table S16: Rank after projector in all settings, continued.
Method
Run
ImageNet
iNat18
Places205
EuroSat
SUN397
Cars
0
382.33
224.68
252.33
207.81
220.68
69.48
278.88
163.91
183.32
154.70
160.71
50.29
2
169.33
101.44
114.49
97.89
99.84
34.97
48.47
32.38
34.93
32.77
31.76
12.53
4
23.22
16.72
17.90
17.70
16.63
7.38
VICReg-ctr
12.88
10.03
10.31
10.66
9.71
5.01
6
22.96
16.87
17.77
17.30
16.55
7.61
7
96.33
62.08
68.05
60.68
60.39
22.59
8
251.52
146.09
166.32
138.75
143.81
45.73
9
309.22
177.32
204.38
170.65
175.81
53.83
10
316.89
184.83
213.74
175.10
185.91
59.07
0 109.07
105.65
104.65
76.13
105.64
92.59
1
164.07
148.71
149.89
100.17
148.00
113.61
244.34
184.32
203.04
129.53
188.30
105.89
3
150.90
94.61
116.94
83.98
102.17
40.64
4
87.69
57.78
67.23
54.62
59.79
25.36
5 63.68
42.23
48.22
40.83
43.22
18.41
6
110.59
106.83
105.83
76.97
106.82
93.98
7
165.49
149.55
150.27
103.19
148.65
113.60
8 246.56
184.69
204.24
128.96
189.86
107.43
9
164.66
102.61
128.12
95.47
112.20
43.29
10
9.88
30.27
2.74
55.46
65.08
25.57
11
63.61
42.00
48.40
40.86
43.20
18.62
12
122.60
118.57
116.93
85.13
118.16
103.25
197.36
173.50
176.32
116.61
173.24
128.89
14
313.67
220.05
239.53
160.52
222.80
111.73
SimCLR
15
299.47
172.75
209.43
140.66
183.51
61.44
16
220.63
122.46
150.73
106.96
130.16
40.02
17
128.33
71.75
90.40
65.77
78.64
26.24
18
71.75
48.95
64.25
48.65
54.84
18.93
19
301.92
173.11
211.03
147.45
185.04
60.83
20
299.75
173.05
208.52
141.84
182.21
61.56
299.96
173.61
209.18
144.25
181.99
61.11
22
300.90
173.89
209.47
147.78
184.45
61.40
23
300.58
174.18
207.19
142.58
184.29
60.94
24
300.83
174.63
207.18
146.11
182.27
60.50
25
11.56
15.95
31.99
144.87
13.55
3.92
26
293.13
172.80
211.66
139.57
184.94
65.02
27
295.23
173.07
208.46
139.91
181.05
62.32
28
299.47
172.75
209.43
140.66
183.51
61.44
29
298.69
172.12
206.63
142.92
181.39
60.88
30
294.42
170.26
201.98
141.29
177.39
58.94
36


Reviews:
[{"review_id": "wzlFOUeR97", "reviewer": "Reviewer_NJ5U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new methodology for tuning hyperparameters of SSL methods, without using labels. The method is motivated by a result in the linear regression setting wherein the train accuracy of a linear probe improves with embedding rank. Based on this, the authors propose that a robust rank measure (originally introduced in prior work) that can be used for model selection.", "summary_of_the_review": "Overall, I am leaning towards rejection because the utility of the proposed approach seems to be fairly limited (hyperparameter selection for a single objective+architecture). Even in that setting, the current experimental results are not sufficiently convincing. I would be willing to increase my score if the authors expanded their analysis to include: (i) models trained on a different source dataset other than ImageNet (eg, the CLIP vision encoders from https://github.com/mlfoundations/open_clip), and (ii) more downstream tasks (eg, all the tasks from Kornblith et al. [arXiv:1805.08974]).\n\n", "strength_and_weaknesses": "I find the premise of the paper—selecting hyperparameters for SSL methods without labels—compelling, particularly given the rise of large vision/language models trained on massive amounts of data. However, both conceptually, and experimentally, the effectiveness of the proposed approach is not clear. \n\n[Conceptually] There are several leaps from the setting in which the first equation in S3.1 holds to practice: (i) regression to classification, (ii) the linear probe does not overfit, (iii) embedding rank scales monotonically across datasets and (iv) embeddings and representations are monotonically linked. It is not clear that these hold true in practice. For instance, depending on the amount of data available for the target task, overfitting of the linear probe is quite likely in practice. Additionally, for many natural transfer tasks, it is possible that the embedding rank on the source dataset does not perfectly correlate with that on the target. \n\n[Experimentally] Even in the ablations conducted by the authors, there are several instances where practice doesn’t line up with the authors intuitions (eg, Figures 2, 3 and 4). As a result, it is hard to assess how reliably this method will work as the source dataset is varied or we consider a larger assortment of downstream tasks. For instance, the assumptions do not hold on Stanford Cars, where rankme also fails to perform good model selection (Table S3).\n\nThe results would be a lot more compelling if the authors considered: (i) models trained on a different source dataset other than ImageNet (eg, the CLIP vision encoders from [https://github.com/mlfoundations/open_clip](https://github.com/mlfoundations/open_clip)), and (ii) more downstream tasks (eg, all the tasks from Kornblith et al. [arXiv:1805.08974]).\n\nAnother drawback of this approach (compared to the oracle of training a linear probe on ImageNet) is that it cannot be used to contrast models trained with different objectives/architectures, which is arguably the bigger consideration in practice. For instance, in Figure 3, on  every dataset, there are points corresponding to SimCLR and VICReg with the same test performance but very different rankme score.\n\nOther comments/questions:\n\n- What was the motivation for the four selected downstream tasks considered in the paper? Given that the average results that the authors report are entirely tied to the choice of datasets, the authors should expand their selection to include more standard tasks. The numbers reported in the paper should also be per task rather than the average.\n- How is the linear probe fit and how are its hyperparameters selected?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kajVRmAlul", "reviewer": "Reviewer_U7yP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle an interesting and pertinent problem in self-supervised learning, specifically model selection without using task-specific labels. To that end, the authors propose using the rank of the embedding matrix learned in an SSL pipeline as a surrogate metric for model selection. They provide theoretical motivation based on Cover's theorem and Shannon Entropy of the eigenspectrum to use rank as the metric of performance. Given the recent literature on dimensionality collapse, the proposal seems reasonable and is supported by extensive empirical results spanning large-scale datasets and multiple SSL frameworks. \nOverall, I think it's an interesting proposal but has some caveats, as described below.", "summary_of_the_review": "I believe this paper is in the correct direction and has valuable insights that could help the community. I do feel in its current form, there are some gaps that need to be addressed, and thereby I have rated it 5. But if the authors can address some of the aforementioned concerns, I am happy to increase my rating above the acceptance threshold. ", "strength_and_weaknesses": "Strengths:\n1. The paper is very well-written and easy to follow and understand. \n2. The method is theoretically motivated and provides an insight into why such a metric is useful to observe.\n3. The authors carry out extensive experiments to support their proposal. Also, this work could allow other researchers to replicate the experiments (given they have sufficient compute budget, which is indeed a challenging ask) and thereby lead to advancing the state of SSL in vision.\n\nWeakenss:\n1. A key concern I have regarding the work is the claim about monotonic relationship between RankMe and performance. A detailed theoretical explanation in the Appendix of Stringer et al. 2019 (https://www.nature.com/articles/s41586-019-1346-5) where they describe how high rank indicates lack of smoothness of the representation manifold, thereby leading to inferior generalization. Indeed, follow up work from Ghosh et al. 2022 (https://arxiv.org/abs/2202.05808) demonstrate a non-monotonic relationship between eigenspectrum decay and linear probe performance. Furthermore, Fig. 3 right plots (corresponding to the performance for representations -- arguably the more common practice in SSL) seems to demonstrate a similar non-monotonic relationship (possibly under-represented due to the log scaling of the x-axis) between rank and performance (clear in VicReg points). Given this non-monotonic relationship, the authors would probably have to do minor tweaks to their model selection proposal and algorithm but it is worth mentioning in the paper for the reader to clearly understand this phenomenon.\n2. Another concern I have corresponds to the discrepancy in the language of the claims in the introduction/abstract and the main text. The authors present their method as a surrogate for model performance, although it is only a necessary but not sufficient condition. I would suggest indicating that RankMe has a necessary but not sufficent relationship with model performance in their claims to make the scope and utility clear to the reader. \n3. The necessary but not sufficient relationship is not surprising because RankMe is agnostic to the label structure. A similar phenomenon is noted in Ghosh et al. 2022 (https://arxiv.org/abs/2202.05808), where they use eigenspectrum decay as a measure of representation quality. \n4.  A minor point, mostly related to a particular claim in the submission, is that RankMe is the first attempt at unsupervised model selection. Although this is concurrent work, but a recent NeurIPS submission aims to achieve similar goals (https://nips.cc/Conferences/2022/Schedule?showEvent=53893). Given the abstract (and the author list), I believe this work is based on the eigenspectrum decay coefficient from Stringer et al. 2019 and I would be curious to know what the authors think about the relationship between RankMe and $\\alpha$. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "il72uZB2Sx", "reviewer": "Reviewer_Sb8P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes RankMe, a method that can assess the quality of JE-SSL representations and work as a predictor of the representations’ performance on target datasets. Without requiring any labels, training, or parameters to tune, this method is simple and computationally friendly. Furthermore, this paper demonstrates that RankMe can be used for hyperparameter selection.", "summary_of_the_review": "Given the poor writing, limited novelty, and some technical issues on hypotheses, I am learning to reject this paper at this time.", "strength_and_weaknesses": "# Strength \n- The introduced method is simple, computationally friendly, and effective.\n- Extensive experiments have proved the correctness of RankMe.\n- The idea of using rank as a metric to select hyperparameters is interesting. Its competitiveness with traditional oracle-based hyperparameter selection methods makes it a promising tool.\n\n# Weaknesses\n- The novelty of this work is limited. It is known that self-supervised learning suffers from dimensional collapse. The rank of the embedding is a measure of the degree of dimensional collapse, so it can reflect the downstream performance to some degree. The main result Eq 2 comes from the empirical observation, which is pretty not rigorous (but the authors have claimed things like principled guidances and theoretically motivated balabala).\n- The writing, presentation, and organization of the paper are poor. There are many grammar errors, such as, in page 1, INTRODUCTION, “tuning JE-SSL methods on unlabeled datasets remain challenging” would be “tuning JE-SSL methods on unlabeled datasets remains challenging”. And There are typos, e.g., in page 3, Fig.1, “selwction” would be “selection”. The derivation of RankMe in section 3.1 is a bit difficult to follow and sometimes obscure. Please check the manuscript carefully.\n- RankMe relies on multiple hypotheses. Though the authors have validated them with some studies, they should also clarify when/where/how these hypotheses apply and fail so that the followers know the applicability of RankMe.\n- RankMe can only be used to tune hyperparameters but not for model/method selection? We know cross-validation is indeed a canonical method for hyperparameter tuning and model/method selection. Why/How RankMe is better than cross-validation? In the aspect of performance of applicability?\n\n## Minor issues\n- In Figure 1, the lines are emphasized by deliberately ignoring the outliers. I cannot see a strong correlation when separately inspecting the red points (corresponding to SimCLR).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject