Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle an interesting and pertinent problem in self-supervised learning, specifically model selection without using task-specific labels. To that end, the authors propose using the rank of the embedding matrix learned in an SSL pipeline as a surrogate metric for model selection. They provide theoretical motivation based on Cover's theorem and Shannon Entropy of the eigenspectrum to use rank as the metric of performance. Given the recent literature on dimensionality collapse, the proposal seems reasonable and is supported by extensive empirical results spanning large-scale datasets and multiple SSL frameworks. 
Overall, I think it's an interesting proposal but has some caveats, as described below.
strength_and_weaknesses: Strengths:
1. The paper is very well-written and easy to follow and understand. 
2. The method is theoretically motivated and provides an insight into why such a metric is useful to observe.
3. The authors carry out extensive experiments to support their proposal. Also, this work could allow other researchers to replicate the experiments (given they have sufficient compute budget, which is indeed a challenging ask) and thereby lead to advancing the state of SSL in vision.

Weakenss:
1. A key concern I have regarding the work is the claim about monotonic relationship between RankMe and performance. A detailed theoretical explanation in the Appendix of Stringer et al. 2019 (https://www.nature.com/articles/s41586-019-1346-5) where they describe how high rank indicates lack of smoothness of the representation manifold, thereby leading to inferior generalization. Indeed, follow up work from Ghosh et al. 2022 (https://arxiv.org/abs/2202.05808) demonstrate a non-monotonic relationship between eigenspectrum decay and linear probe performance. Furthermore, Fig. 3 right plots (corresponding to the performance for representations -- arguably the more common practice in SSL) seems to demonstrate a similar non-monotonic relationship (possibly under-represented due to the log scaling of the x-axis) between rank and performance (clear in VicReg points). Given this non-monotonic relationship, the authors would probably have to do minor tweaks to their model selection proposal and algorithm but it is worth mentioning in the paper for the reader to clearly understand this phenomenon.
2. Another concern I have corresponds to the discrepancy in the language of the claims in the introduction/abstract and the main text. The authors present their method as a surrogate for model performance, although it is only a necessary but not sufficient condition. I would suggest indicating that RankMe has a necessary but not sufficent relationship with model performance in their claims to make the scope and utility clear to the reader. 
3. The necessary but not sufficient relationship is not surprising because RankMe is agnostic to the label structure. A similar phenomenon is noted in Ghosh et al. 2022 (https://arxiv.org/abs/2202.05808), where they use eigenspectrum decay as a measure of representation quality. 
4.  A minor point, mostly related to a particular claim in the submission, is that RankMe is the first attempt at unsupervised model selection. Although this is concurrent work, but a recent NeurIPS submission aims to achieve similar goals (https://nips.cc/Conferences/2022/Schedule?showEvent=53893). Given the abstract (and the author list), I believe this work is based on the eigenspectrum decay coefficient from Stringer et al. 2019 and I would be curious to know what the authors think about the relationship between RankMe and $\alpha$. 
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear and easy to read. I think this paper makes a useful contribution in terms of reproducibility and demonstrating their proposal. If the authors tweak their claims to better align with the final results, I believe it is a strong submission. 
A quick question about Fig 3 and Fig 4:
* Why is the embedding rank always plotted in the x-axis? I thought the performance for representations would be compared to the rank of representations, instead of the embeddings. Did the authors find a better correspondence to the rank of embeddings? Or is this done owing to a different design choice (I might have missed this in the text, apologies for that).

This paper is among the early works to advancing model selection in an SSL pipeline towards a more unsupervised framework. Although there is some concurrent work that aims to solve this problem, this work is still valuable. I believe if the authors can incorporate some of the aforementioned weakensses by taking into light recent work in the field, it would be a very valuable contribution.
summary_of_the_review: I believe this paper is in the correct direction and has valuable insights that could help the community. I do feel in its current form, there are some gaps that need to be addressed, and thereby I have rated it 5. But if the authors can address some of the aforementioned concerns, I am happy to increase my rating above the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes RankMe, a method that can assess the quality of JE-SSL representations and work as a predictor of the representations’ performance on target datasets. Without requiring any labels, training, or parameters to tune, this method is simple and computationally friendly. Furthermore, this paper demonstrates that RankMe can be used for hyperparameter selection.
strength_and_weaknesses: # Strength 
- The introduced method is simple, computationally friendly, and effective.
- Extensive experiments have proved the correctness of RankMe.
- The idea of using rank as a metric to select hyperparameters is interesting. Its competitiveness with traditional oracle-based hyperparameter selection methods makes it a promising tool.

# Weaknesses
- The novelty of this work is limited. It is known that self-supervised learning suffers from dimensional collapse. The rank of the embedding is a measure of the degree of dimensional collapse, so it can reflect the downstream performance to some degree. The main result Eq 2 comes from the empirical observation, which is pretty not rigorous (but the authors have claimed things like principled guidances and theoretically motivated balabala).
- The writing, presentation, and organization of the paper are poor. There are many grammar errors, such as, in page 1, INTRODUCTION, “tuning JE-SSL methods on unlabeled datasets remain challenging” would be “tuning JE-SSL methods on unlabeled datasets remains challenging”. And There are typos, e.g., in page 3, Fig.1, “selwction” would be “selection”. The derivation of RankMe in section 3.1 is a bit difficult to follow and sometimes obscure. Please check the manuscript carefully.
- RankMe relies on multiple hypotheses. Though the authors have validated them with some studies, they should also clarify when/where/how these hypotheses apply and fail so that the followers know the applicability of RankMe.
- RankMe can only be used to tune hyperparameters but not for model/method selection? We know cross-validation is indeed a canonical method for hyperparameter tuning and model/method selection. Why/How RankMe is better than cross-validation? In the aspect of performance of applicability?

## Minor issues
- In Figure 1, the lines are emphasized by deliberately ignoring the outliers. I cannot see a strong correlation when separately inspecting the red points (corresponding to SimCLR).

clarity,_quality,_novelty_and_reproducibility: Overall, the clarity and quality are low and should be improved. 

The novelty is limited as discussed above.

The reproducibility of this work may be good.

summary_of_the_review: Given the poor writing, limited novelty, and some technical issues on hypotheses, I am learning to reject this paper at this time.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new methodology for tuning hyperparameters of SSL methods, without using labels. The method is motivated by a result in the linear regression setting wherein the train accuracy of a linear probe improves with embedding rank. Based on this, the authors propose that a robust rank measure (originally introduced in prior work) that can be used for model selection.
strength_and_weaknesses: I find the premise of the paper—selecting hyperparameters for SSL methods without labels—compelling, particularly given the rise of large vision/language models trained on massive amounts of data. However, both conceptually, and experimentally, the effectiveness of the proposed approach is not clear. 

[Conceptually] There are several leaps from the setting in which the first equation in S3.1 holds to practice: (i) regression to classification, (ii) the linear probe does not overfit, (iii) embedding rank scales monotonically across datasets and (iv) embeddings and representations are monotonically linked. It is not clear that these hold true in practice. For instance, depending on the amount of data available for the target task, overfitting of the linear probe is quite likely in practice. Additionally, for many natural transfer tasks, it is possible that the embedding rank on the source dataset does not perfectly correlate with that on the target. 

[Experimentally] Even in the ablations conducted by the authors, there are several instances where practice doesn’t line up with the authors intuitions (eg, Figures 2, 3 and 4). As a result, it is hard to assess how reliably this method will work as the source dataset is varied or we consider a larger assortment of downstream tasks. For instance, the assumptions do not hold on Stanford Cars, where rankme also fails to perform good model selection (Table S3).

The results would be a lot more compelling if the authors considered: (i) models trained on a different source dataset other than ImageNet (eg, the CLIP vision encoders from [https://github.com/mlfoundations/open_clip](https://github.com/mlfoundations/open_clip)), and (ii) more downstream tasks (eg, all the tasks from Kornblith et al. [arXiv:1805.08974]).

Another drawback of this approach (compared to the oracle of training a linear probe on ImageNet) is that it cannot be used to contrast models trained with different objectives/architectures, which is arguably the bigger consideration in practice. For instance, in Figure 3, on  every dataset, there are points corresponding to SimCLR and VICReg with the same test performance but very different rankme score.

Other comments/questions:

- What was the motivation for the four selected downstream tasks considered in the paper? Given that the average results that the authors report are entirely tied to the choice of datasets, the authors should expand their selection to include more standard tasks. The numbers reported in the paper should also be per task rather than the average.
- How is the linear probe fit and how are its hyperparameters selected?
clarity,_quality,_novelty_and_reproducibility: The writing of the paper is fairly clear and the problem being studied is an important one. My concerns are primarily about the quality of the experimental analysis and the utility of the proposed method, which I have discussed above.
summary_of_the_review: Overall, I am leaning towards rejection because the utility of the proposed approach seems to be fairly limited (hyperparameter selection for a single objective+architecture). Even in that setting, the current experimental results are not sufficiently convincing. I would be willing to increase my score if the authors expanded their analysis to include: (i) models trained on a different source dataset other than ImageNet (eg, the CLIP vision encoders from https://github.com/mlfoundations/open_clip), and (ii) more downstream tasks (eg, all the tasks from Kornblith et al. [arXiv:1805.08974]).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper argues that one can predict or compare the quality of a learned representation from JE-SSL in an unsupervised manner from the embedding matrix rank.   They connect this to theory and show empirically that the rank is predictive of downstream performance across a variety of datasets.  

The reviewers found the motivation compelling and the presented approach simple and effective.  They found the paper well written and noted that the empirical evaluation was extensive.  However, the scores were ultimately 3, 3, 6 (raised from 5 during the discussion).  The major concerns of the reviewers were that the proposed strategy seemed limited to hyperparameter selection of JE-SSL, novelty (one reviewer noted that dimensional collapse is a known issue with SSL) and somewhat incorrect claims.  One reviewer found the paper hard to follow and not well organized.  Although this work is promising, the average review score was unfortunately low and none of the reviewers seemed willing to strongly champion the paper.  Therefore, the recommendation is to reject the paper.  Hopefully the reviews provide useful feedback to improve the manuscript for a future submission. 

It seems clear that the work has potential and addresses an interesting open question in the literature (i.e. how to compare / select SSL representations in an unsupervised manner).  It seems that a clear direction forward is to explore how this can be used to compare architectures and SSL methods more generally, rather than limiting to hyperparameter selection for a single method.  
justification_for_why_not_higher_score: Low reviews, limited novelty, limited applicability.
Decision: Reject