Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims at finding a way to measure the quality of the representations learned in self-supervised learning (SSL), without having to evaluate them on downstream tasks. Based on recent experiments concerning the variance of neural representations in mammal brains, they consider the decay of the eigenspectrum of the representations covariance to evaluate those representations.  Through theoretical results, they suggest that the decay should be neither too quick nor too slow to enable downstream predictors that can be trained quickly but generalize well. Experimentally, they seem to suggest that the best representations always have a similar decay and this decay can thus be used as a metric for SSL. Furthermore, this decay is close to the one previously found in mammal brains.
strengths_and_weaknesses: **Update**: As pointed out by all reviewers (me included) I think that there are valid concerns about using the metric proposed by the authors for selecting SSL representations. For me, the main contribution of the paper is not that much about having some new metric for model selection, but rather providing experimental and theoretical insights into understanding what constitutes useful SSL representations. 

---

**Strengths**:
- **Well written**: the paper is clear and was a pleasure to read.
- **Problem well-motivated and of interest** the goal of evaluating representations efficiently is of interest to the ML community.
- **These insights could help to understand and design SSL**: even if the method is never applied as a metric for SSL, I think that it sheds light on important properties of desirable representations that could be of interest for designing or theoretically understanding SSL algorithms.
- **Connects ML and neuroscience** this paper sheds light on the recent findings in neuroscience that could be of interest to the representation learning community. These findings are the starting point of the paper (not a contribution) but disseminating these interesting findings to the ML community has its own value. 

**weaknesses**
- **Doubts about computational gains** the main selling point of the method is its computational efficiency, yet it is not clear that there are major advantages compared to linear probing. In particular, the large computing time for linear probing (26 hours for ImageNet) makes me doubt those results. My guess is that you are not prefeaturizing the representations, ie, encoding each image only once before training and then saving the features. In my experience prefeaturization + training a linear predictor takes much less than an hour on a single GPU and 16-bit precision. Please correct me if I'm wrong, but if this is the case, then the advantage of using $\alpha$ as a metric is much less clear.
- **Only useful for a given architecture** In figures 3 and 4 (a and b) performance of representations extracted by CNNs seem to always be strongly positively correlated with $\alpha$ while $ViT$ are inversely correlated. If one was to use $alpha\approx 1$ for choosing representations in ViTs you would be quite wrong (around 10 accuracy points off). This suggests that one can only use $alpha$ to compare representations learned using similar encoders.
- **Some experimental evidence is sometimes misleading**  I find figures 3 and 4 to be somewhat misleading, for example for each architecture there is no sign of an inflection point around and so one can change the value of $\alpha$ by changing the ratio of CNNs vs ViTs. Furthermore, the plots and correlations are somewhat arbitrary, eg, in figure 4b if you plotted and computed the same correlations for $alpha =0.8$ or $0.9$ you would get high correlations (maybe even higher than currently). Table 1 is also misleading if no prefeaturization is applied.

questions: -  what are the blue lines in figure 3b and 4b? are they linear regressions on every point before and after $\alpha=1$? in particular does the left ones take into account the ViT models on the left of $\alpha=1$, which show the inverse correlation?
- how are you training the linear predictor? Are you encoding the images once before training or at every epoch?

**Minor suggestions/mistakes**(no need to answer)
- please use \citet when the reference is part of the sentence
- line 46: the citation for [7] should be [8] I believe, ie, you cite the wrong article. 
- line 75: agnostic and metric -> agnostic metric
- line 84: With -> with
- equation 1: should be $f\_{\theta^*} = \dots$
- line 93: is target -> is the target
- line 96: why do you not remove the mean feature when computing the empirical covariance matrix? I.e. this is not the empirical covariance matric, right?
- line 110: "Regression : Recent" -> "Regression: recent"
- line 114 : upto -> up to
- line 227: "all the models exhibit \alpha,$ is unclear
- line 270: Figure Fig. -> Fig.
- line 273: Furthermore, there is $\alpha$ -> Furthermore, \alpha
- figure 5: why are there missing values (in white)?



limitations: The main limitations of the proposed method are that it seems to be: (1) highly dependent on the architecture; (2) slow despite that being the original motivation. 

The former is somewhat discussed in the paper. The latter is not sufficiently well discussed assuming that linear probing is much quicker than suggested in Table 1 (see weaknesses above).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a fairly simple task-agnostic way of assessing the quality of NN representations. The idea is to evaluate the power-law exponent (alpha) of the spectrum of the empirical representation covariances produced by the network. Connecting with some recent theoretical results (that were however derived in limited settings), the authors argue that alpha close to 1 is correlated with good convergence and generalization properties. For this reason, the authors propose the extrapolation of this exponent as a possible metric of NN performance, of representation quality, and for model selection.

AFTER REBUTTAL
The authors have been very open to discussing their results and addressed most of the concerns raised by the reviewers. While in my opinion some further work is needed in order to prove that the proposed measure can truly be impactful, the present study explores an unconventional and interesting direction for developing new metrics for evaluating representations.
strengths_and_weaknesses: The idea of looking at the spectra of empirical representation covariances seems in line with a series of recent theoretical works and seems to explore an interesting research direction. 
However, what the authors propose in practice is based on a very strong assumption on the eigenvalue distribution, which admittedly does not hold in general. Moreover, the presented numerical results do not show a clear quantitative behavior, aligned with the authors' speculations, but just a qualitative agreement. If this method is to be seriously employed for model selection, I believe a more scientific inquiry, where the authors try to disprove their assumptions, could really help shed some light on the very complicated phenomenon of representation learning.
questions: - Shouldn't the means be subtracted when the covariances are evaluated (eq. unnumbered after (1))?
- Theorem 2.2: given that the generalization bound is not tight, is this result truly informative of the best value for alpha?
- The results shown in figure 3 seem to indicate the claimed importance of alpha=1 only qualitatively, but the behavior is not at all clean. How can alpha be used for model selection or performance assessment if the power-law behavior is only an approximation, and if the extrapolated exponents can hover around 1 with alternating perfromances?
- Plots in figures 3-4 are very crowded and the descriptions are too far from their appearance. 
- line 232-233: there seems to be a typo in the correlations, getting the same value on the two sides seems very unlikely.
- In figure 5, the "connection" between alpha and the test error is very hard to assess in this way. The plots are coarse-grained and the color map is unhelpful (wouldn't be better, for alpha, to assign a specific color to 1, so that the "ideal" region is highlighted?). Is the relationship still just qualitative, or is it exactly true that the closer one gets to 1 the better it is? If not, how can this observation become useful in practice?
- Did the authors try to explicitly enforce the power-law behavior with the desired exponent at training, even on a small model? Is it possible to actually test the thesis presented in this paper in a more scientific way?
limitations: The limitations of this work seem to reside in its very qualitative nature. In my opinion, the authors do not address nor acknowledge this issue, and the work would need a major revision in order to accommodate for it.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper aims to find a task-agnostic statistical measure for SSL models that can serve as a good indicator of downstream performance.
With the observation that the eigenvalues of the covariance matrix of the features follow a power law, this work proposes to use the decay rate of the eigenvalue spectrum $\alpha$ (i.e. $\lambda_i \propto i^{-\alpha}$).

Theoretical analyses (Sec 2) suggest that there should be a sweetspot for $\alpha$, from trading off the convergence time and the generalization error.

Empirical study (Sec 3) are performed on STL-10 and MIT-67 datasets, with VGG/ResNet/ViT model pretrained using Barlow Twins/SimCLR/BYOL objective. The results show that there's a clear correlation between $\alpha$ and the task performance (in-distribution, or transfer).

=== **Update post rebuttal** ===

My TL;DR from the paper is that good performing models necessarily have a decay coefficient $\alpha$ that lies within a certain range, which means $\alpha$ can serve as a metric for model selection. Previously I'm concerned about the practical applicability of this metric, since empirical findings show that the dependency on $\alpha$ is unclear in practice. During rebuttal, the authors have addressed most of my concerns, in particular:
- How to find the "inflection point" of $\alpha$ for different architectures & tasks: see discussion at the end of Sec 3.2 (and Appendix B.4); the idea is that $\alpha$ can be used together with linear probe to reduce the computation budge (rather than replacing linear probe altogether, which would be impossible given that  is not a sufficient condition).
- the strength of correlation: the author addressed this here: https://openreview.net/forum?id=ii9X4vtZGTZ&noteId=XTLkW4joEFR

Overall, even though I'm still unsure how useful $\alpha$ can help on model selection (which is the main goal of the study on assessing representation quality), I think the paper provides a relatively thorough study on one plausible solution for assessing representation quality.
strengths_and_weaknesses: Strength:
- The paper studies an importance question, and provides both empirical evidence and theoretical justifications.
- The paper is clearly written and easy to follow.

Weaknesses: my main concern is that it's unclear how transferrable the conclusions from the empirical results are to various different settings. It would be interesting to see whether the same trend holds; if not, then it'd be interesting to understand the conditions for which the trend holds.
- There are no results on large-scale datasets (current experiments are on STL-10 and MIT67).
- Currently strong correlations (Fig 3, Fig 5) are derived from all models, including those with poor accuracies. It's unclear whether the strong correlation still exist if we consider only relatively good performing models.
- The results in Fig 4 do not show a strong correlation; there's a weaker correlation between the model performance and the $\alpha$ value calculated from lower-layer features.

Side note: please make the theorems rigorous (at least in the appendix where there's no space constraint). For instance, please state the convergence time in terms of the error tolerance, and please avoid using $\approx$ (e.g. on line 532)
questions: Please note that I'm using the line numbers in the full version, i.e. paper with the appendix.

- Thm 2.2: $m$ is the rank, which may or may not equal to (i.e. could be smaller than) $\min(n, D)$?
- What's the takeaway of the training loss plot in Fig 2?
- Line 197: "no significant difference between the train & test set": for OOD, would the gap between $\alpha$ values estimated on the source and target domain be indicative?
- Line 225, about the values of $\alpha$ for CNNs and ViTs: are the dimensions kept the same across these architectures?
- Line 234 mentions that the correlation between $\alpha$ and the downstream performance is weaker at earlier layers. I wonder what this implies for the applicability of $\alpha$ as metric: can $\alpha$ be used for any representations, or only the representations from deeper layers? Are their other conditions that should be satisfied?
- If we only consider $\alpha$ values around 1, do we still see the strong correlation between $\alpha$ and the performance?
  - For example on Line 245, "$\alpha$ closer to 1 in the deeper layers": if we consider only $\alpha$ values calculated from the deeper layers, does the strong correlation still hold? In other words, are far-from-1 $\alpha$ values only from earlier layers of the network, which are rarely taken as features in practice?
- If we know that the desired value for $\alpha$ is 1, can we regularize the model in the pretraining phase to encourage the model to reach this value (at least for in-distribution generalization)?
- Line 265: please note that the Barlow Twins objective is applied on the features after the projection head, whereas for downstream tasks we take the features before the projection head, so there's a discrepancy.
- For transfer performance, are $\alpha$ values estimated on different downstream datasets similar? If different models are preferred by different datasets, is this reflected in the corresponding $\alpha$ values?

Minor points:
- Line 202, 227, 274: it would be better to specify the appendix section.
- Typos
    - Line 172: a missing closing bracket.
    - Line 245: "Appendix Fig. 3" links to the Fig 3 in the main paper.
    - Line 248: "Fig 3 and Fig 3".
    - Line 270: "Figure Fig 5".

limitations: The paper discusses the limitations and future directions. One limitation is potentially having inconsistent results between $\alpha$ and performance when $\alpha$ is based on features from early layers of a network.

There is no direct societal impact.
review_title: null

Meta-review: metareview: Decision: Accept

This paper proposes a task agnostic method to evaluate the representation quality of neural network, by looking at the eigenspectrum decay power factor for the feature covariance. Theoretical results and empirical results show the method can be used as a cheap alternative for understanding neural networks and performing model selection.

Reviewers commended the approach being simple yet novel, and they are happy with the overall clarity of the manuscript. The main criticisms were:
1. Whether experimental results generalize to larger dataset and network architectures.
2. The interpretation of the method: useful tool for model selection or for understanding neural networks only?

In author feedback the above questions were partially addressed. Note here the lack of big data experiments is not a major factor of my decision. The more important point is the authors’ clarification on the eigenspectrum decay being “necessary condition” for good performance, which makes the argument for model selection weaker in a theoretical sense, although the empirical results seems to be ok. I would suggest the authors to make a clear discussion on this as promised in author feedback.

As a side note, my brief read of the paper seems to tell me that the theory part considers pre-training, and I would say pre-training is a broader concept than SSL. This means the theory is not specific to SSL, and I hope the authors can clarify this point.

Also I suggest the authors to discuss He and Ozay ICML 2022 in light of this paper’s results.
https://proceedings.mlr.press/v162/he22c.html
confidence: Less certain
Decision: Accept