Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods
Authors: Yuchen Lu
Zhen Liu
Aristide Baratin
Romain Laroche
Aaron Courville
Alessandro Sordoni
Venue: Accepted by TMLR ()

Abstract: We address the problem of evaluating the quality of self-supervised learning (SSL) models
without access to supervised labels, while being agnostic to the architecture, learning
algorithm or data manipulation used during training. We argue that representations can
be evaluated through the lens of expressiveness and learnability. We propose to use the
Intrinsic Dimension (ID) to assess expressiveness and introduce Cluster Learnability (CL) to
assess learnability. CL is measured in terms of the performance of a KNN classifier trained
to predict labels obtained by clustering the representations with K-means. We thus combine
CL and ID into a single predictor – CLID. Through a large-scale empirical study with a
diverse family of SSL algorithms, we find that CLID better correlates with in-distribution
model performance than other competing recent evaluation schemes. We also benchmark
CLID on out-of-domain generalization, where CLID serves as a predictor of the transfer
performance of SSL models on several visual classification tasks, yielding improvements with
respect to the competing baselines.

Published in Transactions on Machine Learning Research (11/2023)
Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods
Yuchen Lu
Mila, University of Montreal
Zhen Liu
Mila, Universily of Montreal
Aristide Baratin
SAIT AI Lab, Montreal
Romain Laroche
Microsoft Research
Aaron Courville
Mila, University of Montreal, CIPAR
Alessandro Sordoni
Microsoft Research, MILA
Reviewed on OpenReview: https: /t optpr/veen.eee.e.e net/ forum? id- BadrpnRHM
Abstract
We address the problem of evaluating the quality of self-supervised learning (SST.) models without access to supervised labels, while being agnostic to the architecture, learning algorithm or data manipulation used during training. We argue that representations can be evaluated through the lens of expressiveness and learnability. We propose to use the Intrinsic Dimension (ID) to assess expressiveness and introduce Cluster Learnability (CL) to assess learnability. CL is measured in terms of the performance of a KNN classifier trained to predict labels obtained by clustering the represenlations with K-means. We thus combine CL and ID into a single predictor CLID, Through a large-scale empirical study with a diverse family of SSL algorithms, we find that CLID better correlates with in-distribution model performance than other competing recent evaluation schemes. We also benchmark CLID out-of- domann generalization, where CLID serves as predictor of the transfer performance of SSL models on several visual classification tasks, yielding improvements with respect to the competing baselines.
1 Introduction
Despite impressive recent progress self-supervised learning (SSL.) Chen et al. 2020a Caron et al. 2021 Grill et al., 2020; Caron al. 2020; 2018 He et al. 2020 Chen & He 2021 b, the problem of properly evaluating the quality of the learned representations without using labelled data has not been fully explored. This is an important problem: solving it could give us practical tool to choose which SSL model to use when downstream task labels are not accessible, or when the costs of fine-tuning every model to choose the best one are prohibitive. It can also shed light on how these methods work.
In this paper, we approach by drawing an analogy between unsupervised learning and the evolution of human language. It has been suggested in the language evolution literature (see e.g., Smith et al., 2013) that linguistic structure results from competing pressures for expressiveness (to discriminate objects and concepts

---

Published in Transactions on Machine Learning Research (11/2023)
BYOL
deepcluster22
35
Sup_RN18 RN18
Pearson Corr: 0.92
simclrvz awav
swavo deepclusterv2
sup RN34
655
Topl ImageNet Accuracy
swev
deepclustervi dine viTb16 @
Intrinsic Dimension (ID)
30
O RN50
mocovl
simcinv
dino VTTba
dina DetTs16
60
sola
no Delts
macov2
25
dino xcits
mocov2
6
InfoMin
45
15-
PCLv1
40
CLID
55
60
65
85
Cluster Learnability (CL)
Figure 1: We propose to use Cluster Learnability (CL) to mcasure learnability and Intrinsic Dimension (ID) for expressiveness. Left: Each circle is a SSL pre-trained checkpoint, and the color is used to show its KNN Top1 ImageNet accuracy. Red indicates high accuracy, while blue indicates low accuracy. Good self-supervised learning representations are learnable and expressive, distributed over the upper-right portion of the graph. Right: Our prodictor is highly correlated with ImageNet performance, even without access gold labels and by staying agnostic to the model's architecture training algorithun
of the world) and learnability (to be transmitted across generations). Here, we take the view that a similar guiding principle may be applied to representations of natural images. Just as language associates objects and scenes with utterances, SSL algorithms do it with continuous vectors to images. We hypothesize that good representation should not only cover the diverse visual concepts in the datasets, but also compress them in a manner that could be efficiently learned. Our goal is to test this hypothesis by (t) proposing tractable metrics to assess expressiveness and learnability, (ti) designing performance predictors basod on these, and (i1i) lesting these predictors through a large-scale empirical study of a diverse class SSL methods.
Concretely, we propose to quantify expressiveness via an estimator of the intrinsic dimension (ID) of the data representations (Facco et al. 2017 Ansuini et al. 2019 To assess learnability, take inspiration from Laina et al. (020)) and propose novel method called Cluster Learnability (CI.), based the performance of KNN learner trained to predict labels induced by clustering held-out representations with K K-Means. We show that a combination of CL and ID, dubbod CLID, correlates with the model performance across different architectures (see Figure D. better than existing evaluation scheme baselines (Wang & Isola 2020 Reed et al.) 2021: Yu al.] 2020). To further demonstrate the usefulness of our framework, we conduct out-of-domain generalization prediction experiments on seven downstream transfer tasks. We find that the proposed CLID predictor outperforms baseline concurrent methods at predicting transfer performance.
Our contributions are summarized below:
We propose an evaluation framework for self-supervised learning relying on expressiveness and lcarnability, yiclding new insights into existing techniques in the literature.
We propose tractable and generic metrics to quantify (i) expressiveness via the Intrinsic Dimension (ID) and (ii) learnability via novel Cluster Tearnability (CI.) metric.
2

---

Published in Transactions on Machine Learning Research (11/2023)
We show that CLID predictors well correlated with Top-1 KNN classification accuracies ImageNet, and are robust with respect to the choice of hyperparameters.
We show that CLID predictors predict the transfer performance of pretrained SSL models on several visual classification task, especially when used jointly with in-domain accuracies.
2 Learnability and Expressiveness
In this section, we briefty discuss existing evaluation schemes of self-supervised learning methods, which will be used baselines in our experiments. We then describe our evaluation scheme, based specific metrics to quantify learnability and expressiveness.
Notation We consider the following setting: we assume we have an unlabelled dataset { represented in X' C Rdd where 2; ~ are sampled i..dd from some distribution P over X. We also consider representation maps X' + RT typically pretrained neural networks, which represent any input as an m-dimensional vector. Any such map forms representation dataset Z { where 2; === F(x;).
2.1 Existing SSL Evaluation
A Alignment and Uniformity (Wang & Isola 2020) By decomposing contrastive learning loss, Wang & Isola (2020) proposes to understand SSL with alignment and uniformity. Formally, they introduce two metrics,
Calign == EEMP, Pang (+1)) [F(x) F(2') a> 0
Canif log Ex, Ex -(
t>
(1)
where aug is the conditional distribution defined by the data augmentation procedure, F(a)/F(())
are the normalized features and t, are tunable parameters. These two metrics respectively correspond the intuition of learnability and expressiveness. By making the feature of positive pairs to be similar, the minimization of Lalign induces learnable representations that can generalize from positive example to another. Minimizing the other component Lunif ensures that the (normalized) features are distributed over hyper-sphere, so that the representations can cover more concepts and achieve high expressiveness.
Maximal Coding Rate Reduction (MCR2) YYu et al. 2020) MCR2 measures the reduction of the average coding length per sample, a.k.a. the coding rate, of representation, induced by the knowledge of some category structure (e.g., the membership of the samples in different classes). The goal of this approach is to learn representations that discriminates between classes while being maximally diverse (MCR2 corresponds to the intuition of learning a diverse and yet compressible (w.r.t. the class partition) representations.
Mutual Information (MI) and Pretext Tasks Maximizing mutual information between inputs and representations (Oord et al. 2018 Bachman et al. 2019a) can be viewed as increasing the expressiveness of the representations. However, as pointed out by Tschannen et al. (2020), MT is not good predictor of the model performance. Our view is that this is because MI alone does not cover take into account the lcarnability of the representations. Similar arguments can be applied to pretext tasks evaluation scheme like rotation prediction and solving jigsaw puzzles (Reed et al. 2021
2.2 Our Proposal
Here we describe our evaluation scheme under the view of learnability and expressiveness. Our proposed metrics can be efficiently evaluated on the validation set of the datasel
We have also experimented with metrics computed the train set but we observed significant difference the results.
3

---

Published in Transactions on Machine Learning Research (11/2023)
Intrinsic Dimension (ID)
We propose to use the notion of intrinsic dimensionality (ID) of the data in the 1979a) to quantify expressiveness. Our intuition is that, as more and more
representation space (Pettis et al. fine-grained categories emerge in the representation space, we expect the manifold complexity to increase. Intrinsic dimension is a way to quantify this complexity, characterized as the number of parameters needed to describe the representation manifold without loss of information.
Inferring the intrinsic dimension of highly nonlinear manifold is a challenging problem (e.g., Levina & Bickel 2005]. In this work, leverage the nearest neighbor- based method of Facco et al. (2017) to estimate I This estimator (TwoNN) is shown to be reliable with respect to representation dimensions and scalable to real-world datasets with deep neural networks (Ansuini et al. 2019). The concept of intrinsic dimension is
also connected to entropy (we discuss this point in detail in Appendix B
Formally, given 2 E Z and an integer 2 1, we denote by Tik = (2,, (Z, (()),)) k)) the distance of 24 to its k-th ncarest neighbor N k). Assuming that the points are sampled on manifold with intrinsic dimension d, it can be shown that, under the assumption of local uniformits the ratio of distances p4 = r/2/rn, S N, follow a Pareto distribution with parameter d + on [1, 20), L.e. H4 ~ P(uld) == dp (d+1), While d can be computed by maximizing the likelihood, we follow a much simpler method proposed by Facco et al. (2017) based on the cumulative distribution F(u) = 1 μー associated with P(uld). The idea is to estimate d with a linear regression on the empirical cumulate of the distribution. Specifically, assuming we sort ascendingly, that is #1 S #2 S S uN, we estimate the cumulative distribution as FTMP = =i/N and fit a straight line on the datasets {(log Uu; - log(1 - )) in the two dimensional plane. The slope is the estimated ID.
Cluster Learnability (CL) Let (2 Ji be the labelled dataset obtained by clustering the represent a ion (e.g., with K K-means). We define the learnability of the representation as the performance of classifier on this labelled dataset. For practical purpose, split to assess the performance of KNN classifier on this labelled dalaset. Let 5 K NNN {ztrain, frrasn) }) be the prediction of after seeing the training data. Learnability is defined as:
choose KNN classifier due to its efficiency. We re-use the dataset
N ceal
CL =
V1
voa
(2)
CLID Predictor We apply our CL and ID in the context of predicting models performance ranking. As a result, for each checkpoint model, we combine the computed CL and ID values into a single numeric predictor, which can be used to rank the models. To deal with the different numeric scales of CL and ID, we firstly standardize them across models into a range from 0 and then adding these values together:
CLID:
Furthermore, we have access the in-domain performance of the models, we can learn weighted sum
W-CLID: w WTCL,,DD
where w E 2 weight that can be learned by linear regression on the in-domain performances. This predictor is mainly used when predicting out-of-domain performances.
We experiment with the ID estimator based on the maximum likelihood estimation (Levina & Bickel 2004 without noticing difference in performance.
Ma et al. 2018
While gencrally consider ncarest neighboors w.r.t Euclidean distance, in practice we will also the cosine distance function, 22) = cos((11, 22). Both produce similar results in experiments. While the original derivation (Facco al. 2017) assumes global uniformity, post-hoc analysis shows that only needs local uniformity up to the second neighhor.
4

---

Published in Transactions on Machine Learning Research (11/2023)
3 Empirical Study
3.1 Setup
We select in total 28 self-supervised learning checkpoints trained on ImageNet over different algorithms,, architecture, and training epochs. A complete list can be found in Table I in the appendix. We use the KNN evaluation on the validation data using the ground-truth labels to measure the performance of the model, which has been shown to be well correlated with the linear evaluation but computationally less expensive Caron et al. 2021
For the computation of cluster learnability, we choose the square root of the dataset size as the number of clusters in Kmeans. We report results with 1 neighbor for our KNN learner. We also show that results are robust to other choices of seper-parameters. We normalize the features and use cosine distance for the K-means clustering and KNN learner Other configurations of cluster numbers and neighbor numbers are also explored in section
All our experiments are computed on a single V100 GPU.
3.2 Baselines
Wang & Isola 2020) also proposes to predict the ImageNet performance of pre-trained SSL checkpoints as an evaluation scheme. Therefore, in our experiments, we follow the official implementatior y with a and 1 = 2 as default values for the tunable parameters in Equ We additionally define -Coontrast = Calign Lunif compute the predictor for Wang & Isola (2020 's method, which reduces the negative contrastive loss.
We add negative sign, since we require predictor to be in proportional to the model performance.
The original MCR2 (Yu et al. 2020) requires class partition to be pre-specified. In order to adapt it into an unsupervised evaluation scheme, we use a K-means clustering as the dataset partition. We follow the default setting: and we normalize the features. We also experiment with using the ground-truth label as the dataset partition, but we found no improvements. We use the coding rate reduction AR (see Yu al., 2020 Equation 6) to be maximized as a predictor.
For the Mutual Information baseline, use MINE (Belghazi et al. 2018 with fixed student ResNet18 network (He et al.] 2016) to estimate the mutual information between inputs and representation. We use a batch size 128, learning rate 0.0005 and weight decay 0.001. The network is trained for 50000 steps on the training images, and we report MINE on the validation data. We find that training longer is computational intensive, while the results are similar.
For the pretext task, we experiment with rotation prediction by constructing a 4-way classification. We randomly rolate the training images by 0, 90, 180, 270 degrees, train KNN classifier on the training images to predict 4- way classification, and then report the rotation prediction accuracy on the validation images. This is shown to be an effective evaluation in both self-supervised learning (Reed et al. 2021 and architecture search (Liu et al. 2020).
3.3 Is CLID scheme correlated with ImageNet performance?
We first investigate whether the proposed evaluation scheme is useful for in-domain (ImageNet) generalization. We perform both qualitative and a quantitative examination and show the results in Figure [ On the left, we find that the sclf-supervised learning checkpoints with higher ImageNet accuracies tend to be both more learnable and more expressive (c.g., in the upper-right corner of the graph). In the meantime, we find that methods favouring only of these qualities at the expanse of the other, like PCLv1 and PIRL, also have poor ImageNet accuracies. In Figure D middle and right, compute the correlation of our CLID predictors with respect to the ImageNet accuracy of the model considered and show that we achieve a Pearson p of 0.92 for CLID.
We find similar results when using L2 distance. 65ee http://gtttub..co///////// unni uniform
7 See httpp://gthabb.cc//yaaaaaaaaatap
5

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: Correlation results between ImageNet performances and different predictors. We compute both Pearson p and Kendall т. Our CLID predictors achieve the highest correlation.
Predictors
Pearson p Kendall т
Lalign
0.42
0.26
Lunis Lcontrast
-0.05
0.03
0.37
0.24
AR
-0.62
-0.33
MI
0.13
0.08
Pretext
0.61
0.27
Ours
CL
0.74
0.44
ID
0.12
0.09
CLID
0.92
0.75
In Table we compare our results with the baselines. The regression plots for the baselines can be found in Figure Our proposed predictors achieve the highest correlation both in terms of Pearson P and Kendall 7 coefficient. We find that Lcontrast achieves relatively low correlation 0.37, which apparently contradicts the observation made in Wang & Isola (2200). We hypothesize that this is due to the fact that their analysis is restricted to SSL checkpoints trained with contrastive learning, and therefore Leontrast might not. be general enough to characterize representations obtained by alternative approaches. Additionally, the results in Wang & Isola (2020) are computed on the representations usod compute the noiso-contrastive objective, which are usually transformed by a final projection layer that is not always present in other SSL methods. This limits its generality. Interestingly, we find that MCR2 (AR) gives negative correlation which warrants further investigation. While the MI baseline only achieves 0.13, the pretext task baseline is still surprisingly good with 0.61, suggesting that the rotation prediction task remains a simple but effective baseline.
3.4 Is CLID sensitive its hyper- ?oper-parameter?
In this section, check the sensitivity of CLID to its hyperparameters. Since IDD does not have any hyper-parameters, mainly examine the following hypet-parameters for CL: the number of clusters and the nunber of neighbors used in the KNN Learner.
The results can be found in Figure 2 The resulting predictors can still produce a higher correlation than the rotation prodiction for a wide range of the configurations we tried. We observe that docrcasing the number of neighbors leads to a better predictor. Given our result, we recommend selting this number be
We observe that there is an optimal cluster number so that the choice of neighbor numbers can be more flexible. If the cluster number too low, the results become worse. Recall that for different checkpoints, the clusters are produced with the same K-means algorithun, so the separability among clusters night be roughly controlled. As result, we hypothesize that the number of clusters might control the underlying difficulty of the classification problem faced by the KNN learner. In the extreme case, we either have 1 cluster or as many clusters as data points. In the former, KNN accuracy would always be 100% and in the latter always be close to 0%4 As a result, neither of them would be suitable enough to distinguish the collected checkpoints. As a rule of thumb, we recommend using a reasonably large number of clusters, e.g., the square root of the dataset. size.
3.5 Can CLID be used to predict transfer performance?
Here we investigate whether CLID can be a good predictor of the performance on downstream tasks. We collect 7 out-of-domain downstream visual classification tasks. For each domain, we compare the ranking of our SSL checkpoints induced by CLID, and the ranking induced by the actual test performance on that
8Too be specific it's 1/N, where the dataset size. It's close 0% when large.

---

Published in Transactions on Machine Learning Research (11/2023)
1.0
100
0.86
0.83
0.74
200
0.9
0.89
0.87
0.85
Pearson Coefficient
0.9
Number of Clusters
400
0.93
0.92
0.92
0.91
0.8
W-CLID predictor
800
0.94
0.93
0.93
0.93
CLID predictor
1000
0.94
0.94
0.94
0.94
rotation prediction
2000
0.94
0.93
0.93
0.92
4000
0.93
0.93
0.92
0.88
0.6
8000
0.93
0.92
0.88
0.79
0.5
1.0
5.0
20.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Number of Neighbors
# Cluster # Dataset (%)
The heat map of Pearson Coefficient between the Top-1 accuracy and W-CLID predictor. Right: Pearson coefficient vs. the ratio between the number of clusters and the dataset size when using one neighbor. The result is stable with a reasonably large number of clusters, e.g., the squarc-root the dataset size (Green Dashed Line).
Figure 2: Robustness Analysis for ImageNet. Left:
Table Kendall Ranking Coefficient results on Out-of-Domain Tasks when ImageNet labels are not available. The CLID predictor has the best predictive performance for transfer lasks. Full results in Table including results for the few-shot experiments.
CLID
Ceontrast
AR
MI
Pretext
foods
0.75
0.17
-0.35
0.01
0.51
flowers
0.81
0.25
-0.35
0.14
0.46
pets
0.71
0.23
-0.25
-0.08
0.54
caltech101
0.56
0.11
-0.18
-0.05
0.67
stl10
0.56
0.02
-0.34
-0.21
0.41
aircraft
0.67
0.21
-0.27
0.16
0.57
cars
0.81
0.32
-0.28
0.16
0.48
Avg
0.70
0.19
-0.29
0.02
0.52
domain. We report the Kendall correlation score, which is consistent with the existing benchmarks out-of-domain generalization (Vedantam et al. 2021
We first investigate the scenario where the ImageNet labels are not accessible (Table 2 This is an extension of our previous scenario, and is also a realistic assumption especially when the models are pretrained on web-scale data without clear annotations. We find that our CL.ID predictor the best, even beating the W-CLID prodictor. One reason because the lincar cocfficients w are obtained from regressing in the ImageNet domain, which might increase overfitting and hurt transfer. We also find that -Lrontrast not good predictor, and simple pretext task like rotation prediction already has reasonable performance even in transfer settings.
In the second setting, we assume to have access to ImageNet labels (Table 3) and explore whether transfer performance prediction can be improved using this additional information. We find that the ImageNet accuracics have pretty high correlation with the downstream tasks, with an average Kendall coefficients of 0.71. To integrate ImageNet performance information to each our competing predictors, we proceed as follows. Let p
pred be the ranking of i-th checkpoint from the predictor, and rf rfmg be that from the ImageNet accuracy.

---

Published in Transactions on Machine Learning Research (11/2023)
Table 3: Kendall Ranking Coefficient results on Out-of-Domain Tasks when ImageNet labels are obtained. The results are computed with the joint rank product between the predictors and the ImageNet accuracy. While the ImageNet accuracy can predict the transfer accuracy reasonably, the proposed CLID predictor further enhances the predictive performance. Full results in Table which include results with few-shot experiments. Blue cells indicates the joint ranking outperforms using ImageNet accuracy alone.
Imagenet
CLID
W-CLID
Source Ent
Target Ent
Leontrast
Pretext
foods
0.86
0.84
0.81
0.88
0.84
0.56
0.75
flowers
0.70
0.79
0.74
0.74
0.71
0.58
0.64
pets
0.75
0.77
0.78
0.66
0.72
0.55
0.76
caltech101
0.62
0.59
0.58
0.60
0.66
0.41
0.77
stl10
0.76
0.71
0.71
0.61
0.71
0.36
0.64
aircraft
0.60
0.66
0.61
0.68
0.64
0.50
0.62
cars
0.68
0.79
0.74
0.72
0.74
0.64
0.66
Avg
0.71
0.74
0.71
0.70
0.72
0.52
0.69
We compute joint ranking by computing the rank product, which is the geometrie mean of these two ranks:
Joint Vereating img
In addition to comparing with previously presented basclines, we also add the strongest basclines from (Vedan- lam et al. 2021): we train a linear classifier and measure the negative label entropy - (Y)))). Since the entropy can be measured on either source domain or target domain, we denote each variant as "Source Ent" and "Target Ent". The underlying intuition is that, if the model has more confidence in the prediction (lower entropy), it should generalize better.
We find that the proposed CLID predictor can further enhance the ImageNet accuracies predictions, with average Kendall coefficients of 0.74 for the joint ranking. We find that "Target Ent." and "Source Ent." both have a reasonable performance, which is consistent with observation in Vedantam et al. (2021). but they underperform our predictor. Note that computing "Target Ent." and "Source Ent." requires training an extra linear layer, which increases their computational requirements.
4 Related Work
Representation Evaluation Several recent works address the question of representation evaluation self-supervised learning. [Whitney et al. (2021) propose use the learning dynamics of the downstream classifiers to measure the representation complexity. However their method still depends on extra human labels. Pretext tasks like jigsaw rotation prodiction are shown to be well correlated with the self-superviedd evaluation (Reed et al. 2021 Deng & Zheng 2021) and architecture search (Liu et al. 2020). However, they also rely on crafting ad-hoc data augmentations. Ericsson et al. (2021) argues that it is important to look at the transfer performance of the SSL models in order to judge its quality. We agree with this statement and therefore also test our method for out-of-domain generalization settings (section 3.5). Closely related to our work, a very recent paper Garrido et al. (2022) builds on theoretical insights of He Ozay (2022) to propose an evaluation of joint-embedding self-supervised methods without access to supervised labels based on the effective rank of the learned embedding matrix. While Garrido al. (2022) employs the spectral rank of the metric, akin to measuring effective dimension, paper approaches the problem from the perspective of performance across tradeoff between effective dimension and learnability.
representation
The concepts of learnability and expressiveness, while not being explicitly mentioned, can be found in existing literature. For example, many SSL strategies emphasize maximizing the mutual information between inputs and reprosentations (Arora et al. 2019: Vincent al. 2008 Higgins et al. 2017 Oord et al. 2018 Bachman et al. 2019b). These can be viewed as attempts to increase expressiveness, since high mutual information leads to correspondence between inputs and representations, and thus the visual concepts among the inputs
8

---

Published in Transactions on Machine Learning Research (11/2023)
are mapped to the representation space. Recent works also pay attention to the emerging properties of learnability in the cluster structure from SSL models through human studies (Laina et al. 2020). The emphasis of learnability can be also found in the recent attempt to design compression regularizor called Conditional Entropy Bottleneck [Lee et al. 2021p. Finally, the intuition of a trade-off between learnability and expressiveness underpins other representation analysis frameworks, such as alignment and uniformity (Wang & Isola 2020) or Maximal Coding Rate Reduction (Yu et al. 2020), which are further explained in Section 2 In this paper, we aim turning this intuition into practical self-supervised evaluation tool, especially on predicting performances in out-of-distribution transfer tasks.
and Compression Learnability has been argued to be a hallmark of the human language in order to be effortlessly transmitted through generations (Kirby et al.] 2014
Learnability,
Rafferty ct al., 2011 Beckner ct al. 2017
Zhou & Yurovsky 2021; Kampen 2004), and it is also true for 2010), categories (Griffiths el al. 2006), shapes (Portelance el al. 2021
visual concepts like color (Xu et al.
etc. In deep learning, it has been explored in the context of emergent communication (Ren et al. 2020
Guo et al. 2019: Li & Bowling 2019), language drift. (Lu et al. 2020). and neural module networks (ann iet al. 2021 but is less explored for vision representation learning, except for a human study on just two SST. methods (Taina et al., (2020). While Lee et al. (2021) also investigates a regularizor for compressive self-supervised learning, it unclear whether such notion is useful as a representation evaluation framework. Learnability has a tight connection to compression (Chaitin 2007) and prequential codclength (Dawid 1984]. which quantifies the compression levels with the online learning error. Existing work (Blier & Ollivier 2018 has uses it to support the generalization ability of the learner (e.g., deep neural nets) on the dataset (e.g..
labeled images). Here, we use this concept to quantify the learnability of the representation, in the sense that if the emerged Kmeans clustering is more learnable, then the same KNN learner could achieve a lower compression bound via prequential coding.
Intrinsic dimension can be thought of as the smallest number of variable needed to approximate the representation manifold. Applying local neighborhood information to estimate the intrinsic dimensson not a new idea, and it was shown to be more efficient than the global eigenvalue approach (Pettis et al. 1979b). Ansuini et al. (2019) apply the TwoNN estimator (Facco et al. 2017) to the non-linear reprosentation manifold of modern deep neural nets. They find that the intrinsic dimension is inversely correlated to the classification accuracy Their work is further extended to confirm that natural images lies in a low-dimension manifold (Pope et al.) 2021), and that lower ID datasets leads to better generalization. Recanatesi et al. (2019) and and Li et al. (2018) further highlight the connections between intrinsic dimension and the generalization propertics of learned representations, by comparing the models before and after supervised training. Intrinsic dimension can also be estimated locally with a Maximum Likelihood Estimator (Levina & Bickel 2004), which Ma al. (2018) propose to use as a regularizor against overfitting noisy labels. While the above work mainly focus on ID with supervised learning models, our work on SSL models presents a more muanced view about ID. It is indeed the case that lower intrinsic dimension can lead to better accuracy, especially among supervised checkpoints (see "sup RN18", Soup_N33"" "sup RN50 Figure []. However, we hypothesize that when learning representation from scratch without labels, e.g., self-supervised learning, the representation manifold still need a certain amount of complexity in order to include enough information from the dataset.
Manifold Intrinsic Dimension
5 Conclusion
We propose a unifying view to evaluate self-supervised representation learning through expressiveness and learnability. We propose to estimate expressiveness with intrinsic dimension (ID), and learnability with performance of KNN learner on the K-means clustering of the representation. We show that the proposed CLID evaluation scheme better predicts the ImageNet accuracy than other evaluation schemes. We further demonstrate that our CLID can also predict the transfer task performance.
While our evaluation scheme designed to be general, a potential limitation is that it might be only suitable to the classification tasks, since we put emphasis on the emergence of learnable cluster structure. As a result, the proposed predictor might not be as useful for other downstream tasks like object detection or
9

---

Published in Transactions on Machine Learning Research (11/2023)
segmentation.
While our results is true for population of models, there are also outliers. E.g., in Figure "deepclusterv1" seems to have higher CL and ID than "simelrv2", but its accuracy is lower.
Future works could further explore the expressiennes-learaabbil framework to other field like pretrained language models, as well as devise new SSL algorithms that directly maximize intrinsic dimension or cluster learnability.
in a more theoretical context, extend this
References
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advonces in Neural Information Processing Systems, 32:6111-6122,
2019.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A thcorctical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902, 190.0.99992,, 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in Neural Information Processing Systems, 32: 32:15535-15544 2019a.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, pp. 15509-15519, 2019b.
Clay Beckner, Janet Pierrehumbert, and Jemnifer Hay. The emergence of linguistic structure in an online ilerated learning task. Journal of Language Evolution, 2(2):160-177 2017.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference Machine Learning, pp. 531 540. PMLR, 2018.
Léonard Blier and Yann Ollivier. The description length of deep learning models. 18X2: 874007044 2018.
ar preprint
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised lcarning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. ar preprint ar Xic::2000 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXio preprint ar 2xix:2144.1299, 2021.
Gregory J Chaitin. Thinking About Godel And Turing: Essays On Complezity, 1970-2001 World scientific, 2007.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual reprosentations. arXiv preprint 20:::2000000000 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong supervise learners. Advances in Neural Information Processing Systems, 33: 22243-222 22255, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVE /CFF Conference on Computer Vision and Pattern Recogrútion, pp. 15750 15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. Xio preprint arXi::2003.0422 2020c.
10

---

Published in Transactions on Machine Learning Research (11/2023)
A Philip Dawid. Present position and potential developments: Some personal views statistical theory the prequential approach. Journal of the Royal Statistical Sociely: Series A (General), (47 1984.
Weifian Deng and Liang Zheng, Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 15069-15078, 2021.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation, pp. 5414-5423, 2021.
Elena Facco, Maria d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):1-8, 2017.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervisod reprosentations by their rank. arXiv:2210.022: 02885 [cs.LG], 2022.
Thomas L Griffiths, Brian R Christian, and Michael L Kalish. Revealing priors on category structures through iterated learning. In Proceedings of the 28th annual conference of the Cognitive Science Society, volume 199, 2006.
Jean- Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning, Advonces in Neural Information Processing Systems, 33, 2020.
Shangmin Guo, Yi Ren, Serhii Havrylov, Stella Frank, Ivan Titov, and Kenny Smith. The emergence of compositional languages for numeric concepts through iterated learning in neural agents. ar preprint arXiv:1910.0522 190.09999, 2019.
Bobby He and Mete Ozay. Exploring the gap between collapsed amp; whitened features in self-supervised learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. press/v162/he226 htmll
PMLR, 17-23 Jul 2022. URL httpps://proceedlngs mlr
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729,9738, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Jacqueline van Kampen. The learnability of syntactic categories. LOT Occosional Series, 3:245 256, 2004.
Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution of language. Current opinion in newrobiology, 28:108 114, 2014.
Tro Laina, Ruth Fong, and Andrea Vedaldi. Quantifying learnability and describability of visual concepts emerging in representation learning. Advances in Neurol Information Processing Systems, 33, 2020.
Kuang-Huci Lee, Amurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances in Neural Information Processing Systems, 34, 2021.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In
L. Saul, Y. Weiss, and L. Bottou (eds.), Aduances in Neural Information Processing Sys- tems, volume 17. MIT Press, 2005. URL https://prroceedingg www.ips.cccppper////...
.pdf
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes, arXie preprint arXiv:1804.00 08838, 2018.
Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communication. In Advances in Newral Information Processing Systems, pp. 15825-15835, 2019.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021. URL https://opeereevier
net//forummidceKayyppSSS/pp
Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. Are labels necessary for neural architecture scarch? In European Conference Computer Vision, pp. 798-813. Springer, 2020.
Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pictquin, and Aaron Courville. Countering language drift with seeded iterated learning. arXiv preprinl ar Xix:20003. 12694, 2020.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionnllty-drvven learning with noisy labels. In International Conference on Machine Learning, pp. 3355 3364. PMLR, 2018.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretoxt-invariant reprosentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018. URL http:://axixii wrg. httg//bb/1887..m 03748 cite arxiv:1807.03748.
Karl W. Pettis, Thomas A. Bailey, Anil K. Jain, and Richard C. Dubes. An intrinsie dimensionality estimator from neir-neighbor information. IEEE Transactions Pottern Analysis and Machine Intelligence, PAMI-1 15:7 1979a. doi: 10.1109/TAAMI.1977 4766873.
Karl W. Pettis, Thomas A. Bailey, Anil K. Jain, and Richard C. Dubes. An intrinsic dimensionality estimator from neir-ne information. IEEE Transactions on Pottern Analysis and Machine Intelligence, PAMI-1 (1):25 37, 1979b. doi:
197.9476777.7.77
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact learning. In International Conference Learning Representations, 2021. URL
Eva Portelance, Michael C Frank, Dan Jurafsky, Alessandro Sordoni, and Romain Laroche. The emergence 2021.
of the shape bias results from communicative efficiency. arXiv e-prints, pp.
Anna N Rafferty, Thomas I. Griffiths, and Mare Futlinger. Exploring the relationship between learnability and linguistic universals. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, pp. 49-57, 2011.
Stefano Recanatesi, Malthew Farrell, Madhu Advani, Timothy Moore, Guillaune Lajoie, and Eric Shea-Brown Dimensionality compression and expansion in deep neural networks. arXiv preprint arXiv:1906.0044 2019.
Colorado J Reed, Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. Selfaugment: Automatic augmentation policies for self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2674 22832 2021.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B Cohen, and Simon Kirby. Compositional languages emerge in neural iterated learning model. orXiv preprint or (xXi:2202.01365, 2020.
Kenny Smith, Monica Tamariz, and Simon Kirby. Linguistic structure an evolutionary trade-off between simplicity and expressivity. In Proceedinggs of the annual meeting of the cognitive science society, volume 35, 2013.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005. 102444,00...24, 2020.
Michael Tichannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual infor- mation maximization for representation learning. In International Conference on Learning Representalions 2020. URL https://openrevier. wttp://mooooumiddddooo022222
Ankit Vani, Max Schwarzer, Yuchen Lu, Feshan Dhekane, and Aaron Courville. Iterated learning for emergent systematicity in [vqa). In International Conference on Learning Representations, 2021. URL
Ramakrishna Vodantam, David Lopez-Paz, and David J Schwab. An empirical invostigation of domain generalization with empirical risk minimizers. Advances in Neural Information Processing Systems, 34, 2021.
P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning proceedings. 2008
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
William F Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, and Kyunghyun Cho. Evaluating representations by the complexity learning low-loss predictors. In Neural Compression: From Information Theory to Applications- @orkshop@ ICLR 2021, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733-3742, 2018
Jing Xu, Thomas Griffiths, and Mike Dowman. Replicating color term universals through human itcrated learning. In Proceedings of the Annual Meeling of the Cognilive Science Sociely, volume 32, 2010.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-944 94444429444 2020.
Yuchen Zhou and Dan Yurovsky, A common framework for quantifying the learnability nouns and verbs. In Proceedings of the Annual Meeting of the Cognâtive Science Society, volume 43, 2021.
13


Reviews:
[{"review_id": "1W1LH3qqnW", "reviewer": "Reviewer_EeFB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method is simple and is relatively cheap computationally. \n2. It seems that the method correlates well with within and without domain generalization.\n3. Authors do a fairly good job at evaluating their proposed method. \n4. The method is a great way to approximate model transferability to downstream tasks, and does not require any labels or training be done to obtain. \n\nWeaknesses:\n\n1. It's unclear whether the CLID metric correlates as well with other downstream tasks, such as zero-shot retrieval, semantic segmentation, relational reasoning etc. While the authors provide enough empirical evaluation to demonstrate the usefulness of the method, it's not clear whether this method is truly as generally applicable as other existing methods. \n2. CLID's main usefulness is that it's useful in situations where labels for a downstream task are not available as a model selector and as a measure of transferability. However, one could instead use a variety of known datasets/tasks with available labels and their transfer performance as an alternative metric -- something which is generally and extensively done today. While CLID is simply, and seemingly computationally expensive, I would still feel less confident using it in a production system as a predictor of downstream performance when compared to using a diverse set of tasks, and domains to evaluate my models and use that as a transferability measure. \n3. It's also not clear whether the CLID metric is purely computed on ImageNet, or computed on each of the datasets that are being compared one by one -- this is a clarity issue that should be addressed by the authors. \n4. Related to 3, the authors don't discuss whether CLID is dependent on ImageNet being the dataset to compute it on, and whether it correlates as well when the K-means cluster is computed over representations of other datasets. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose CLID, a method and metric for evaluating self supervised learning models (and potentially general neural network models), in terms of their learnability and expressivity. Those are measured using Intrinsic Dimension, a form of compressibility measure, and, Cluster Learnability, where data points encoded by a model, are clustered using K-means, and then a fitted on the training set using a KNN classifier. The resulting performance of this KNN classifier on ImageNet is how CL is obtained. \n\nThe authors conduct two empirical studies; one where they compare the CLID performance of a given model with the validation set performance of said model. This is referred to as 'in-distribution' performance. CLID is demonstrated to correlate very well with the ImageNet validation set results. The second study is 'out-of-distribution' and involves correlating the CLID performance with model performances on downstream tasks. Again, the CLID metric seems to correlate very well with the downstream task metrics, outperforming in general the other contenders chosen by the authors to appear in the table. \n\n ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The writing of the paper feels rather voluminous, and could, and ideally should be made more compact. It'll make for a paper that is easier to read and understand. \n\nOn the tables and figures where correlations are shown, the authors should explicitly state how each CLID metric was obtained (i.e. from ImageNet k-means or for per-dataset k-means etc). Table 6 is rather confusing, I am not sure what the bottom table is trying to communicate. Also, the few-shot results should ideally be in the main manuscript as they seem too important to omit, and they improve the credibility of the method introduced.  ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PDzkCnmOh5", "reviewer": "Reviewer_ZkU8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths:**\n\nThis paper is well written and I am glad to read this paper.  The topic this paper investigates is very important. I believe the researcher in SSL community would like to have a good predictor (indicator) to indicate how well the SSL methods are, without requiring the proxy evaluation tasks. The clarification of expressiveness and learnability is clear, and it is also clear that the combination of expressiveness and learnability correlates the performance of SSL methods. This paper carefully makes the claims that are well supported by the evidences. \n\n\n\n**Weaknesses:**\n\n1. This paper should provide more discussion to the closely related work Garrido et al. (2022). The work in Garrido et al. (2022) is simpler and more intuitive to evaluate the representation of SSL without performing the proxy tasks, as to me. Besides, this paper should also conduct experiments to compare Garrido et al. (2022) (e.g., considering the $\\alpha$ in (0, 1) for Pearson coefficient). This paper should also take credits to [1], which is closely related to Garrido et al. (2022).  \n\n2. It is better to illustrate the intuition how the proposed method (by Facco et al. (2017)) in estimating the ID. \n\n\n**Ref:**  \n[1] Exploring the Gap between Collapsed & Whitened Features in Self-Supervised Learning. ICML 2022\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of evaluating SSL methods without requiring the proxy evaluation tasks. It argues that the combination of expressiveness and learnability can evaluate the representation of SSL methods. In particular, it uses the Intrinsic Dimension (ID) to assess expressiveness and introduce Cluster Learnability (CL) to assess learnability. The proposed indicator CLID that combines ID and CL better correlates with in-distribution model performance than other competing recent evaluation schemes on ImageNet dataset. CLID can also serve as a predictor of the transfer performance of SSL models on several visual classification tasks. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.More discuss to related work, e.g, Garrido et al. (2022) and the missing ICML 2022 paper;\n\n2. Experimental comparison to Garrido et al. (2022).", "broader_impact_concerns": "No impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0YbSTMS2hg", "reviewer": "Reviewer_gRum", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n- Evaluating self-supervised representations efficiently is an important problem. Fine-tuning them for downstream tasks can be computationally intensive, and thus, a good proxy is needed to estimate their performance.\n- The proposed CLID metric demonstrates a high correlation with the classification accuracy, outperforming the baselines in terms of correlation coefficients.\n- The CLID metric can be extended to evaluate out-of-distribution generalization and can be transferred to other tasks from self-supervised datasets.\n\n\n### Weakness & Questions\n\n[W1] Other pre-trained models\\\nThe paper only demonstrates the results on supervised and joint-embedding types of self-supervised learning. Does the proposed metric work for other pre-training methods, such as reconstruction-based self-supervised learning (e.g., MAE) and vision-language pre-training (e.g., CLIP)? Adding those checkpoints to the plot of Figure 1 would be interesting and informative.\n\n[W2] Scale of CL vs. ID\\\nThe final CLID metric is simply a sum of two metrics. However, it is nonsensical since their scales are not calibrated. It would be great if the revised paper adds some discussion and justification that the scales of the two metrics are comparable, at least empirically, in Section 2.2. The paper partly addresses this issue through W-CLID, but the estimated weights are not discussed thoroughly.\n\n[W3] Additional related works\\\nThere is a line of work studying the quality of representation outside the self-supervised learning literature. These papers mostly focus on the generalizability of learned representations, often under the lens of the generalization bound in learning theory perspective. Intrinsic dimension was also studied from this perspective, to the best of my knowledge [1]. I suggest that the authors discuss such a line of work and provide a broader view for the evaluation of representations.\\\nOn the other hand, there is learning theory literature for self-supervised learning, such as [2]. These papers provide a natural evaluation metric for self-supervised representation, and adding a discussion with them could provide a comprehensive view of the related fields.\n\n[1] Measuring the Intrinsic Dimension of Objective Landscapes, ICLR'18.\\\n[2] A Theoretical Analysis of Contrastive Unsupervised Representation Learning, ICML'19.\n\n\n### Comments on the resubmission\n\nThe main concerns in the last round were (1) the dependency of the CLID metric on data processing order and (2) the choice of hyperparameters. To the best of my understanding, the resubmitted paper addressed these issues by (1) making slight modifications to their method (from sequential to average accuracy) and (2) adding a sensitivity analysis of hyperparameters (Sec. 3.4). Thus, I believe the concerns from the last round have been reasonably addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to design an evaluation metric for self-supervised representations. To this end, the paper suggests measuring the expressiveness and learnability, which are estimated by intrinsic dimension (ID) and cluster learnability (CL), respectively. The model with better Pareto optimality tends to exhibit higher classification accuracy. Additionally, by combining them into a single metric, the proposed CLID metric shows a strong correlation with ImageNet accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See [W1-3].", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xRoR1jBh0F", "reviewer": "Reviewer_SmAV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The research problem investigated in this work is a very relevant and interesting direction. Better tools for assessing SSL checkpoint quality without a canonical downstream supervised classification task could enable more reliable SSL evaluation in diverse domains, and provide speedier evaluations without model fine-tuning even when a downstream supervised classification task exists.\n* The proposed method is lightweight and scalable. Both ID and CL calculations only require Euclidean distance measures (or other distance measure) and rankings in the activation space.\n* Experimental results show strong correlation between the proposed metric and the typical metric of linear probe evaluation on Imagenet, and the proposed metric significantly outperforms other metrics studied.\n* The method is intuitively reasonable, and the paper is clearly written and easy to follow.\n\n**Weaknesses**\n\n* The baseline methods such as (Wang&Isola 2020) and MCR2 are reimplemented for ImageNet, but no direct comparison is made to experiments present in the original papers. Since neither paper directly investigates ImageNet, it might be good to also apply the proposed method on a smaller-scale setting such as CIFAR-10 so that direct comparison to published results is possible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates the problem of evaluating the quality of self-supervised learning (SSL) models. The typical way to evaluate such models is to measure the accuracy of linear classifiers of embedding vectors, but this requires a suitably large evaluation dataset with supervised (image, label) pairs, such as ImageNet. When there is no natural evaluation dataset or when labels are unavailable this approach is not possible, and the current work aims to develop a metric that can assess SSL model quality using only its unsupervised training data. Ideally, this metric would accurately predict the performance of linear probe classifiers without requiring supervised labels. The proposed metric has two components: Intrinsic Dimension, which is meant to measure the expressiveness of the model, and Cluster Learnability, which is meant to measure how easily concepts can be learned from the representations. ID is calculated using the TwoNN method from prior work, and CL is calculated by learning a K-means model of the representations, learning a KNN classifier from the K-means model results, and measuring the agreement between the K-means predictions and the KNN predictions on validation data. The final proposed metric is the sum of these scores after normalization. Experiments are conducted on a variety of pretrained SSL models which show the proposed metric can accurately predict ImageNet performance with linear probing, and that it can accurately predict OOD transfer performance with and without use of ImageNet labels.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could the authors provide a direct comparison between their work and existing results like Wang&Isola 2020 and MCR2 by applying their method to a scenario explored in the original papers?", "broader_impact_concerns": "The authors did not discuss Broader Impacts, but this did not impact my evaluation of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I want to thank the reviewers and authors for participating in the process.  I'm sorry for my part in how delayed the process played out this time.
Decision: Accept (poster)